Agent Builder Quest: 10 missions to master AI agent design

A chat agent that finds a meeting slot for people in different cities using three tiny tools. You learn the core loop: the model picks a tool, your code runs it, the result goes back until the model answers.

Tool schemasThe agent loopStop conditions

What you build

1. Write three pure functions: city_to_timezone(city), convert_time(time, from_tz, to_tz), working_hours_overlap(zones).
2. Describe each one as a tool with a clear name, description and typed parameters.
3. Run the loop: send messages plus tools, execute any tool call, append the result, repeat.
4. Stop on a final text answer or after a max of 6 steps, whichever comes first.
5. Print every tool call and result so you can read the trajectory.

👾 Boss level: Return tool errors as readable messages (unknown city, ambiguous name) and check that the model recovers by asking the user instead of guessing.

Build a command-line meeting-time finder agent in [LANGUAGE] using [LLM PROVIDER WITH TOOL CALLING]. Tools: city_to_timezone(city), convert_time(time, from_tz, to_tz), working_hours_overlap(zones, start_hour, end_hour). Implement the agent loop by hand: call the model with the tools, run requested tools, append results, repeat until a final answer or [MAX STEPS] steps. Log each step. Tool errors must be returned to the model as clear messages, never thrown.

One front door, four specialists: a regex helper, a cron expression builder, a SQL explainer and a git command helper. A cheap classifier routes each request to the right specialist prompt. You learn that routing beats one giant prompt.

ClassificationStructured outputFallback routes

What you build

1. Write a small router prompt that returns JSON: route, confidence, short reason.
2. Write four focused specialist prompts, each with its own examples and output format.
3. Dispatch to the chosen specialist; if confidence is low, ask one clarifying question.
4. Add an 'unsupported' route that politely declines anything off-topic.
5. Build a test file of 30 labeled requests and measure routing accuracy.

👾 Boss level: Use a smaller, cheaper model for the router and a stronger one for specialists, then compare cost and accuracy against a single big-model prompt.

Build a dev helper switchboard in [LANGUAGE] using [LLM PROVIDER]. A router call classifies each user request into one of: regex, cron, sql_explain, git, unsupported, returning JSON {route, confidence, reason}. Each route has its own specialist prompt with [N] examples. If confidence is below [THRESHOLD], ask one clarifying question. Include a labeled test set of [N] requests and a script that prints routing accuracy and a confusion table.

Give it a messy CSV and a goal like 'one row per customer, ISO dates'. A planner writes a step list; an executor runs each step with a small set of safe operations. You learn to separate thinking from doing, and when to re-plan.

Plan as dataConstrained operationsRe-planning

What you build

1. Define a whitelist of operations: rename_column, drop_column, parse_dates, trim, dedupe, split_column.
2. Have the planner output a JSON plan: ordered steps, each with an operation and arguments.
3. Validate the plan against the whitelist before running anything.
4. Execute step by step on a copy of the data, recording a before/after row count for each.
5. If a step fails, send the error and current schema back to the planner for a revised plan (max 2 re-plans).

👾 Boss level: Show the plan to the user as a readable checklist and let them remove or reorder steps before execution.

Build a planner-executor CSV cleanup tool in [LANGUAGE] using [LLM PROVIDER]. Input: a CSV file and a goal in plain language. The planner sees the column names and [N] sample rows and returns a JSON plan using only these operations: [OPERATION LIST]. Validate the plan, then execute it on a copy of the data, logging row counts per step. On failure, re-plan with the error message, at most [N] times. Write the cleaned file and a markdown report of what changed.

A writer drafts a README for a small project folder, a critic scores it against a rubric, and the writer revises until the score passes or rounds run out. You learn how to make a model check its own work without looping forever.

RubricsCritique promptsLoop budgets

What you build

1. Write a 5-point rubric: install steps, usage example, config, limits, accuracy versus the code.
2. Writer call: read the file tree and key files, draft the README.
3. Critic call: score each rubric item 0–2 with a one-line justification, as JSON.
4. Loop: revise using only the failing items, stop at a passing score or 3 rounds.
5. Save every draft and score so you can see if revisions actually improved things.

👾 Boss level: Add a deterministic check (do the commands in the README exist in the project scripts?) and feed its result to the critic.

Build a README writer with a reflection loop in [LANGUAGE] using [LLM PROVIDER]. Input: a project folder path. Step 1: a writer drafts README.md from the file tree and [KEY FILES]. Step 2: a critic scores the draft against this rubric [RUBRIC ITEMS], returning JSON with a score and reason per item. Step 3: the writer revises using only failing items. Stop when the total is at least [PASS SCORE] or after [MAX ROUNDS] rounds. Save each draft and score to an output folder.

An agent that answers 'why did we build it this way?' from your team's architecture decision records, citing the exact paragraph. A second pass verifies each citation really supports the claim. You learn that retrieval is only half the job; grounding is the other half.

ChunkingSearch as a toolCitation checksSaying 'not found'

What you build

1. Split each decision record into paragraphs with stable IDs (file + heading + index).
2. Expose search(query, k) as a tool so the agent can search several times with different wording.
3. Require answers where every sentence ends with a citation ID.
4. Run a verifier pass: for each claim plus cited paragraph, answer supported / not supported.
5. Drop or flag unsupported claims, and answer 'not in the records' when nothing relevant is found.

👾 Boss level: Detect conflicts: when two records disagree (an older decision was superseded), make the agent say so and prefer the newer one.

Build a retrieval-augmented agent in [LANGUAGE] over a folder of architecture decision records in [FORMAT]. Chunk by paragraph with stable IDs. Give the agent a search(query, k) tool backed by [SEARCH METHOD]. Every sentence of the answer must cite a chunk ID. Add a verifier call that checks each claim against its cited chunk and removes unsupported claims. If nothing relevant is found, reply that the records do not cover it. Include [N] test questions, some with no answer in the records.

A tutor that quizzes you on a topic you choose and remembers, from one session to the next, which concepts you keep missing. You learn to decide what to store, how to retrieve it, and when to let memories fade.

Memory schemaWrite and read policiesDecay

What you build

1. Define a memory record per concept: name, last seen, times right, times wrong, a short note.
2. After each answer, let the agent call update_memory with a structured change, not free text.
3. At session start, load only the weakest and most overdue concepts into context.
4. Give weaker concepts shorter review intervals and let mastered ones fade out of the rotation.
5. Add a 'show what you remember about me' command and a 'forget' command.

👾 Boss level: Separate two memory types: facts about the learner (preferences, goals) and per-concept performance, each with its own write rules.

Build a spaced-practice tutor agent in [LANGUAGE] using [LLM PROVIDER] and [STORAGE] for memory. Topic: [TOPIC]. Memory record per concept: {concept, last_seen, correct, wrong, note}. The agent quizzes the user, grades answers, and calls update_memory(concept, change) after each one. At session start, load the [N] weakest or most overdue concepts only. Add commands to view and delete memories. Persist between runs.

Point it at a mock inventory of cloud resources and it proposes cleanup actions: stop idle machines, delete old snapshots, shrink oversized disks. Safe actions run, risky ones wait for your approval. You learn risk tiers, approval gates and audit trails.

Risk tiersApproval gatesDry runsAudit log

What you build

1. Create a JSON inventory of fake resources with owner, last used date, size and tags.
2. Classify each tool by risk: read (auto), reversible change (auto with log), destructive (needs approval).
3. For destructive actions, pause and show a dry-run diff with the agent's reason; the human approves, edits or rejects.
4. Feed rejections back to the agent so it adjusts the rest of the plan.
5. Append every proposal, decision and outcome to an audit log file.

👾 Boss level: Enforce the gate in code, not in the prompt: the delete tool refuses to run without a valid approval token, and write a test that proves it.

Build a human-in-the-loop cleanup agent in [LANGUAGE] using [LLM PROVIDER]. Input: [INVENTORY FILE] describing mock cloud resources. Tools: list_resources(), stop_instance(id), delete_snapshot(id), resize_disk(id, size). Tag tools by risk tier. Destructive tools must pause, print a dry-run diff plus the agent's reason, and wait for approve / edit / reject. The approval check lives in the tool code, not the prompt. Write every step to an append-only audit log.

Describe a shipped feature and an orchestrator hands the work to specialists: release notes writer, docs updater, announcement drafter, QA checklist maker. You learn handoff contracts, shared state and how to keep specialists from stepping on each other.

Handoff contractsOrchestrationShared stateMerge step

What you build

1. Define a handoff contract: task, inputs, expected output format, and what the specialist must not change.
2. Give each specialist its own short prompt and only the tools it needs.
3. Let the orchestrator decide which specialists to call and in what order, running independent ones in parallel.
4. Collect outputs in a shared state object, then run a consistency check across them (same version, same feature name).
5. Log each handoff with its inputs and outputs so you can replay one specialist alone.

👾 Boss level: Let a specialist hand back to the orchestrator with a question when inputs are missing, instead of inventing details.

Build a multi-agent release assistant in [LANGUAGE] using [LLM PROVIDER]. Input: a feature description and [DIFF OR CHANGELOG]. An orchestrator dispatches to specialists: release_notes, docs_update, announcement, qa_checklist. Each handoff uses this contract: {task, inputs, output_format, constraints}. Specialists can return a question instead of an answer. Store results in a shared state object, run a consistency check across all outputs, and print a trace of every handoff.

Wrap a database migration reviewer agent behind a tool server so other agents (and your coding agent) can call review_migration and get a structured verdict. You learn to design an agent's public interface: inputs, outputs, limits and errors.

Interface designTool protocolsTimeoutsVersioning

What you build

1. Build the inner agent: it reads a SQL migration and flags locking risks, missing rollbacks and data loss.
2. Define one tool with a strict input schema and a JSON verdict: risk level, findings, suggested fix.
3. Serve it over a standard tool protocol (MCP or your own HTTP endpoint) with a timeout and step limit.
4. Return errors as structured results the calling agent can reason about.
5. Connect it to another agent and watch it use your reviewer as just another tool.

👾 Boss level: Version the tool contract, add a second tool (explain_finding), and keep the old version working for existing callers.

Build a SQL migration reviewer agent in [LANGUAGE] and expose it as a tool server using [TOOL PROTOCOL OR HTTP FRAMEWORK]. Tool: review_migration(sql, dialect) returning JSON {risk: low|medium|high, findings: [{line, issue, fix}]}. The inner agent has read-only tools for [SCHEMA SOURCE]. Enforce a [N]-second timeout and [N]-step limit, return structured errors, and include a small client agent that calls the tool on [N] sample migrations.

Boss mission: take one agent you built in this quest and make it production-grade. Every run is traced, a trajectory eval suite runs before every change, and regressions block the merge. You learn to prove an agent works, not just hope.

TracingTrajectory evalsRegression gatesCost tracking

What you build

1. Trace every run: each model call, tool call, arguments, result, latency and token count, tied by a run ID.
2. Build a replay command that re-runs a stored trace step by step.
3. Write 25 eval cases checking both the final answer and the path: right tools, no forbidden calls, under N steps.
4. Run each case several times and report pass rates, not single results.
5. Add a CI job that compares against the last baseline and fails on regression, plus a small dashboard of cost and latency per run.

👾 Boss level: Turn every bad production trace into a new eval case with one command, so the suite grows from real failures.

Add production observability and evals to my agent in [LANGUAGE] at [AGENT PATH]. 1) Trace every model call and tool call (inputs, outputs, latency, tokens) under a run ID, stored in [TRACE STORE]. 2) Add a replay command for a stored run. 3) Create an eval suite of [N] cases checking final output and trajectory (expected tools, forbidden tools, max steps), running each case [N] times. 4) Add a CI step that compares pass rates to a saved baseline and fails on regression. 5) Print cost and latency per run.