LLM Project Lab: 20 projects to go from first API call to production agent

• ✍️

  Commit message writer

  Level 1~2hPrompting · Deployment

  Mark done

  A CLI that reads your staged diff and proposes a clear commit message. You learn system prompts, context limits and how to trim input.

  ▸ Details

  1. Write a script that captures the staged diff as text.
  2. Send it to any LLM API or a local model with a system prompt describing your commit style.
  3. Truncate or summarize very large diffs so you stay under the context limit.
  4. Print the suggestion and let the user accept, edit or reject it.
  5. Add a git hook or alias so it runs in one command.

  Starter prompt

  Copy

  Build a small CLI in [LANGUAGE] that reads the output of the staged git diff, sends it to an LLM ([PROVIDER OR LOCAL MODEL]) with a system prompt enforcing [COMMIT CONVENTION], and prints a proposed commit message. Truncate diffs over [N] characters with a note. Let the user accept (y), edit (e) or cancel (n). Keep the API key in an environment variable. Include a README with setup steps.

• 🪵

  Stack trace explainer

  Level 1~2hPrompting

  Mark done

  Paste an error, get a plain-language explanation and three likely causes. You learn prompt structure, few-shot examples and output formatting.

  ▸ Details

  1. Create a simple CLI or web form that accepts a stack trace.
  2. Write a prompt with a fixed layout: summary, likely causes, what to check first.
  3. Add two short few-shot examples of good explanations.
  4. Compare answers with and without the examples on five real errors.
  5. Keep the best prompt version in a file under version control.

  Starter prompt

  Copy

  Create a [CLI OR SMALL WEB APP] in [LANGUAGE] that takes a pasted stack trace and asks an LLM ([PROVIDER OR LOCAL MODEL]) to return: a one-sentence summary, three likely causes ranked, and the first thing to check. Store the prompt template in a separate file with two few-shot examples I can edit. Add a flag to run without the examples so I can compare.

• 🔁

  Model comparison playground

  Level 1~3hPrompting · Local models · Evals

  Mark done

  Send the same prompt to two models side by side and log latency, tokens and output. You learn how models differ and what cost really means.

  ▸ Details

  1. Set up access to one hosted LLM API and one local model.
  2. Build a page or CLI that sends one prompt to both at once.
  3. Record latency, input and output token counts for each call.
  4. Estimate cost per call from your provider's current pricing.
  5. Save every run to a CSV file so you can compare later.

  Starter prompt

  Copy

  Build a [CLI OR SIMPLE WEB PAGE] in [LANGUAGE] that sends one prompt to two backends in parallel: [HOSTED MODEL] and [LOCAL MODEL]. Show both answers side by side with latency and token counts. Read price per token from a config file and show estimated cost. Append each run (timestamp, prompt, model, latency, tokens, cost, output) to runs.csv.

• 📝

  Meeting notes to action items

  Level 1~2hPrompting

  Mark done

  Turn raw meeting notes into decisions, owners and deadlines. You learn to write precise instructions and to handle messy, ambiguous input.

  ▸ Details

  1. Collect three or four sets of real or realistic meeting notes.
  2. Write a prompt that asks for decisions, action items, owners and dates.
  3. Tell the model what to do when an owner or date is missing.
  4. Render the result as a markdown checklist.
  5. Tweak the prompt until no action item is invented.

  Starter prompt

  Copy

  Write a script in [LANGUAGE] that reads a text file of meeting notes and asks an LLM ([PROVIDER OR LOCAL MODEL]) to extract decisions and action items with owner and due date. If an owner or date is not stated, the output must say 'unassigned' or 'no date', never guess. Output a markdown checklist. Include three sample notes files in /samples.

• 💬

  Streaming terminal chat

  Level 1~3hPrompting · Memory

  Mark done

  A chat client in your terminal that streams tokens and keeps a conversation history. You learn message roles, streaming and context window management.

  ▸ Details

  1. Build a loop that reads user input and sends the message history to the model.
  2. Stream the response token by token to the terminal.
  3. Count tokens and drop or summarize old turns when you approach the limit.
  4. Add /reset and /system commands to change the conversation.
  5. Save conversations to a local file on exit.

  Starter prompt

  Copy

  Build a terminal chat app in [LANGUAGE] that talks to [PROVIDER OR LOCAL MODEL] with streaming output. Keep the full message history, and when it exceeds [N] tokens, summarize the oldest turns into one message. Support commands: /reset, /system [TEXT], /save. Keep the code in under [N] files with clear comments.

• 🧾

  Invoice data extractor

  Level 2~4hStructured output

  Mark done

  Pull vendor, date, totals and line items from invoice text into validated JSON. You learn schemas, validation and retry on bad output.

  ▸ Details

  1. Define a schema for the fields you need, with types and required fields.
  2. Ask the model to answer only in JSON that matches the schema.
  3. Validate every response in code; on failure, retry once with the error message.
  4. Test on ten sample invoices, including a messy one.
  5. Log which fields fail most often and adjust the prompt.

  Starter prompt

  Copy

  Create a [LANGUAGE] module that takes invoice text and returns JSON matching this schema: [PASTE SCHEMA: vendor, invoice_date, currency, subtotal, tax, total, line_items]. Use [PROVIDER OR LOCAL MODEL]. Validate the response with [VALIDATION LIBRARY]; if invalid, retry once including the validation errors. Add tests with [N] sample invoices in /fixtures and report per-field accuracy.

• 🏷️

  Issue triager

  Level 2~4hStructured output · Evals

  Mark done

  Classify new issues by type, component and priority, with a confidence score. You learn classification prompts, enums and when to ask a human.

  ▸ Details

  1. Export 30 existing issues with their labels as ground truth.
  2. Ask the model to return a label from a fixed list plus a confidence value.
  3. Route low-confidence results to a 'needs human' bucket.
  4. Measure accuracy against your ground truth.
  5. Try one prompt change and compare the numbers.

  Starter prompt

  Copy

  Build a [LANGUAGE] script that reads issues from [SOURCE: CSV OR ISSUE TRACKER EXPORT] and asks an LLM to classify each into type [LIST], component [LIST] and priority [LIST], returning JSON with a confidence between 0 and 1. Anything under [THRESHOLD] goes to needs_review. Compare against the existing labels column and print an accuracy table per field.

• 🛠️

  Natural language to SQL (read-only)

  Level 2~5hTools · Structured output

  Mark done

  Ask questions about a sample database in plain English. You learn tool calling, schema context and hard safety limits.

  ▸ Details

  1. Load a small sample database and connect with a read-only user.
  2. Give the model the table schema and one tool: run a SELECT query.
  3. Reject anything that is not a single SELECT before it runs.
  4. Return the rows to the model and let it answer in plain language.
  5. Show the generated SQL to the user every time.

  Starter prompt

  Copy

  Build a [LANGUAGE] app that answers natural-language questions about a [DATABASE TYPE] database. Expose one tool to the LLM: run_select(sql). Provide the schema in the system prompt. Before executing, parse the SQL and reject anything that is not a single SELECT; connect with a read-only user and a [N]-second timeout. Show the SQL, the rows (max [N]) and the model's answer.

• 🌦️

  Multi-tool assistant

  Level 2~4hTools

  Mark done

  An assistant with three small tools (calculator, date math, a public API of your choice). You learn tool design, argument validation and the tool loop.

  ▸ Details

  1. Write three tools with short, unambiguous names and descriptions.
  2. Implement the loop: model asks for a tool, you run it, you send back the result.
  3. Validate tool arguments and return clear errors the model can recover from.
  4. Cap the loop at a fixed number of steps.
  5. Log every tool call with its arguments and result.

  Starter prompt

  Copy

  Build a [LANGUAGE] assistant using [PROVIDER OR LOCAL MODEL WITH TOOL SUPPORT] with three tools: calculate(expression), date_diff(start, end), and [YOUR TOOL](args). Implement the tool-calling loop manually, cap it at [N] iterations, validate arguments and return error messages the model can act on. Print a trace of each tool call. Add five example questions that need more than one tool.

• 📋

  PR summarizer

  Level 2~5hStructured output · Prompting

  Mark done

  Summarize a pull request diff into what changed, why it matters and what to review. You learn chunking long input and consistent structured summaries.

  ▸ Details

  1. Fetch a PR diff and split it per file.
  2. Summarize large files individually, then combine into one summary.
  3. Return a fixed structure: overview, risky changes, suggested review order.
  4. Post the summary as a comment or print it locally.
  5. Compare the summary with what a teammate would write on three PRs.

  Starter prompt

  Copy

  Create a [LANGUAGE] tool that takes a PR diff (from a file or [CODE HOST] API), splits it per file, summarizes files over [N] lines separately, then produces JSON with: overview, changes_by_area, risky_changes (with file and reason), review_order. Render it as markdown. Use [PROVIDER OR LOCAL MODEL]. Never include secrets found in the diff in the output.

• 📚

  Internal docs assistant

  Level 3~1 dayRAG

  Mark done

  Ask questions over your team's docs and get answers with citations. You learn chunking, embeddings, retrieval and grounded answers.

  ▸ Details

  1. Collect 20 to 50 markdown or text docs.
  2. Split them into chunks with titles kept as context, and embed them.
  3. Store vectors in a local vector store or a database extension.
  4. Retrieve the top chunks for a question and pass them to the model.
  5. Require citations and an 'I don't know' when the docs don't cover it.

  Starter prompt

  Copy

  Build a RAG assistant in [LANGUAGE] over the markdown files in [FOLDER]. Chunk by heading with [N]-token max, keep the file path and heading as metadata, embed with [EMBEDDING MODEL], store in [VECTOR STORE]. For each question, retrieve the top [K] chunks and ask [PROVIDER OR LOCAL MODEL] to answer only from them, citing file and heading, and to say it doesn't know otherwise. Add a /reindex command.

• 🧪

  Eval harness for your RAG

  Level 3~5hEvals · RAG

  Mark done

  A small test suite that scores your docs assistant on real questions. You learn golden sets, retrieval metrics and LLM-as-judge with care.

  ▸ Details

  1. Write 25 questions with the expected source doc and a reference answer.
  2. Measure whether the right doc appears in the retrieved chunks.
  3. Score answers with simple checks first, then an LLM judge with a strict rubric.
  4. Spot-check the judge against your own grading on ten answers.
  5. Run the suite on every prompt or chunking change and save the results.

  Starter prompt

  Copy

  Create an eval harness in [LANGUAGE] for my RAG app at [PATH OR ENTRYPOINT]. Read cases from evals.yaml (question, expected_source, reference_answer). For each case, record whether expected_source is in the top [K] retrieved chunks, and grade the answer with an LLM judge using this rubric: [RUBRIC]. Output a table and a JSON results file with a timestamp so I can compare runs.

• 🧠

  Assistant with long-term memory

  Level 3~6hMemory · RAG

  Mark done

  A personal assistant that remembers your preferences across sessions. You learn what to store, how to retrieve it and how to let users edit it.

  ▸ Details

  1. After each conversation, ask the model to extract durable facts worth keeping.
  2. Store facts with timestamps in a local database.
  3. Retrieve relevant facts at the start of each new conversation.
  4. Add commands to list, edit and delete memories.
  5. Handle conflicts: newer facts replace older ones.

  Starter prompt

  Copy

  Build a chat assistant in [LANGUAGE] with long-term memory stored in [SQLITE OR OTHER STORE]. At the end of each session, ask the LLM to extract up to [N] durable user facts as JSON. At the start of each session, retrieve the [K] most relevant facts by embedding similarity and include them in the system prompt. Add /memories, /forget [ID] and /edit [ID] commands. Newer facts on the same topic replace older ones.

• 🔎

  Codebase Q&A

  Level 3~1 dayRAG · Tools

  Mark done

  Ask 'where is X handled?' about a repository and get file-level answers. You learn code-aware chunking and hybrid keyword plus vector search.

  ▸ Details

  1. Walk a repo and chunk files by function or class where possible.
  2. Index chunks with both embeddings and a keyword index.
  3. Merge results from both searches before sending them to the model.
  4. Return answers with file paths and line ranges.
  5. Compare hybrid search with vector-only on ten questions.

  Starter prompt

  Copy

  Build a codebase Q&A tool in [LANGUAGE] for the repo at [PATH]. Chunk source files by function or class (fallback: [N] lines), store file path and line range. Index with embeddings in [VECTOR STORE] and a keyword index. Merge both result lists, send the top [K] chunks to [PROVIDER OR LOCAL MODEL], and answer with file:line citations. Add a flag to switch between hybrid and vector-only for comparison.

• 🏠

  Fully local notes search

  Level 3~5hRAG · Local models

  Mark done

  RAG over your personal notes that never leaves your laptop. You learn local embeddings, small models and the quality trade-offs they bring.

  ▸ Details

  1. Run a local model and a local embedding model on your machine.
  2. Index your notes folder and watch it for changes.
  3. Answer questions with citations to note titles.
  4. Try two model sizes and note speed versus answer quality.
  5. Confirm no network calls are made while it runs.

  Starter prompt

  Copy

  Build a fully offline notes search in [LANGUAGE] over [NOTES FOLDER] using [LOCAL MODEL RUNTIME] for both embeddings and generation. Watch the folder and reindex changed files. Answer questions with citations to note titles. Add a config option to switch between [SMALL MODEL] and [LARGER MODEL] and print generation time for each answer.

• 🚨

  Log triage agent

  Level 4~2 daysAgents · Tools

  Mark done

  An agent that reads recent error logs, groups them, checks recent deploys and drafts an incident summary. You learn multi-step planning with read-only tools.

  ▸ Details

  1. Give the agent read-only tools: search logs, list recent deploys, read a file.
  2. Let it group errors by signature and pick the top issues.
  3. Ask for a summary with evidence links for every claim.
  4. Cap steps, tokens and time per run.
  5. Review five real runs and tighten the tool descriptions.

  Starter prompt

  Copy

  Build a log triage agent in [LANGUAGE] using [PROVIDER OR LOCAL MODEL WITH TOOL SUPPORT]. Tools (all read-only): search_logs(query, since), list_deploys(since), read_file(path). The agent groups errors by signature, picks the top [N], checks whether they started after a deploy, and writes a markdown incident summary where every claim cites a log line or deploy. Limit to [N] steps and [N] tokens per run and print a full trace.

• 🔧

  Test-fixing agent in a sandbox

  Level 4~2 daysAgents · Tools · Deployment

  Mark done

  An agent that runs your test suite, reads failures and proposes a patch inside a container. You learn sandboxing, feedback loops and human approval.

  ▸ Details

  1. Run the project inside a disposable container with no secrets.
  2. Give the agent tools to run tests, read files and write a patch.
  3. Loop: run tests, read failure, edit, rerun, with a hard step limit.
  4. Output a diff for a human to approve, never an auto-merge.
  5. Record how often patches pass on a set of seeded bugs.

  Starter prompt

  Copy

  Build an agent in [LANGUAGE] that fixes failing tests in [REPO] inside a disposable [CONTAINER RUNTIME] container with no network and no secrets. Tools: run_tests(), read_file(path), write_file(path, content). Loop up to [N] times: run tests, read the failure, edit, rerun. At the end, output a unified diff and a short explanation for human review. Include [N] seeded bugs to benchmark the success rate.

• 🧭

  Research agent with a budget

  Level 4~2 daysAgents · RAG · Evals

  Mark done

  An agent that breaks a question into sub-questions, searches your document set and writes a sourced brief within a fixed budget. You learn planning and cost control.

  ▸ Details

  1. Have the agent write a short plan of sub-questions first.
  2. Answer each sub-question with retrieval over your document set.
  3. Track tokens and cost per step and stop at the budget.
  4. Write a final brief where each paragraph cites its sources.
  5. Evaluate briefs on ten questions for coverage and citation accuracy.

  Starter prompt

  Copy

  Build a research agent in [LANGUAGE] that takes a question, writes a plan of up to [N] sub-questions, answers each with retrieval over [DOCUMENT SET OR INDEX], and writes a brief with citations per paragraph. Track tokens and estimated cost per step; stop and summarize what it has when it reaches [BUDGET]. Save the plan, each step and the final brief to a JSON trace file.

• 📡

  Production LLM gateway

  Level 4~2 daysDeployment · Evals

  Mark done

  A small service in front of your LLM calls that adds tracing, caching, rate limits and fallbacks. You learn the plumbing every production app needs.

  ▸ Details

  1. Put one internal endpoint in front of all your model calls.
  2. Log every request with a trace ID, latency, tokens and cost.
  3. Add a cache for identical requests and a per-user rate limit.
  4. Fall back to a second model when the first times out or errors.
  5. Build a simple dashboard of cost and error rate per day.

  Starter prompt

  Copy

  Build an HTTP gateway service in [LANGUAGE] that proxies LLM requests to [PRIMARY PROVIDER] with fallback to [SECONDARY PROVIDER OR LOCAL MODEL]. Add: request tracing (trace ID, latency, tokens, estimated cost) to [LOG STORE], an exact-match cache with [TTL], per-API-key rate limits, and timeouts of [N] seconds. Expose /metrics with daily cost and error rate. Include a Dockerfile and tests for the fallback path.

• 🤝

  Support reply copilot

  Level 4~3 daysAgents · RAG · Evals

  Mark done

  Drafts replies to support tickets from your help docs, with a human approving every send. You learn guardrails, escalation rules and end-to-end evals.

  ▸ Details

  1. Retrieve relevant help articles and past resolved tickets for each new ticket.
  2. Draft a reply with citations and a confidence level.
  3. Escalate automatically on refunds, legal topics or angry customers.
  4. Keep a human approval step before anything is sent.
  5. Track how often agents edit drafts heavily and use that as your eval signal.

  Starter prompt

  Copy

  Build a support reply copilot in [LANGUAGE]. For each ticket from [SOURCE], retrieve the top [K] help articles and similar resolved tickets, then draft a reply with citations and a confidence score. Escalate (no draft) when the ticket mentions [ESCALATION TOPICS]. Show drafts in a simple review UI with approve, edit and reject; log the edit distance between draft and sent reply for evaluation.

Nothing matches.