Building AI agents that survive production

Most agent demos work. Most agents I get called in to fix also worked, in a demo. The gap is almost never the model: it’s the tools, the context, the missing evals, and nobody being able to see what the agent did on the day it went sideways.

This is what I walk teams through before their first agent goes live. It’s framework-agnostic: the ideas hold whatever SDK or model you pick.

First question: do you actually need an agent?

An agent is a model that decides, step by step, which tool to call next, until it decides it’s done. That autonomy is the whole point and the whole cost: every decision is one you can’t fully predict, test or budget in advance.

Three shapes, from simplest to most autonomous. Single LLM call: input in, output out. Workflow: you can draw the steps on a whiteboard, so code owns the control flow and the model fills in the fuzzy parts. Agent: the path genuinely depends on what’s discovered along the way.

🎮 Agent, workflow, or single LLM call?

1 / 9 · Score: 0

Pick the simplest shape that does the job.

Single LLM callWorkflowAgent

Next →

/ 9

↺ Try again

The core loop

Strip away the frameworks and every agent has the same parts: a model that decides, tools it can use, the context it sees, a stop condition, and guardrails around all of it.

1. 📥Build contextTask, instructions, relevant knowledge, history so far.
2. ↓→
3. 🧠Model decidesFinal answer, or which tools to call.
4. ↓→
5. 🛠️Run toolsValidate, check approval, execute with a timeout.
6. ↓→
7. 📝Record & appendTrace the step, sanitize the result, compact if needed.

↺ Repeat until a final answer, or until the step or cost budget runs out

Here’s the same loop in language-agnostic pseudocode:

function run_agent(task, tools, limits):
    context = build_initial_context(task)
    steps = 0
    cost = 0

    loop:
        if steps >= limits.max_steps or cost >= limits.max_cost:
            return stop("budget_exceeded", context)

        decision = model.decide(context, tools.descriptions)
        trace.record(step=steps, input=context, output=decision)
        cost += decision.cost
        steps += 1

        if decision.type == "final_answer":
            return validate_and_return(decision.answer)

        for call in decision.tool_calls:
            if not tools.exists(call.name):
                result = error("Unknown tool: " + call.name)
            else if requires_approval(call):
                result = request_human_approval(call)   # may pause the run
            else:
                result = tools.execute(call, timeout=limits.tool_timeout)

            trace.record(step=steps, tool=call, result=result)
            context = append(context, call, sanitize(result))

        context = compact_if_needed(context)

Look at everything besides the model call: hard budgets, tracing, an approval gate, sanitizing, compaction. Those lines separate a production agent from a demo.

Designing tools

In my experience, this is where agent quality is won or lost. The model can only be as good as the actions you hand it and the descriptions you write for them.

Write tool descriptions like you’re onboarding a new hire: what it does, when to use it, when not to, what each parameter means, units and limits. “Returns up to 50 orders, newest first. Use since to page further back” saves the model from guessing.

Context management

Context is everything the model sees at decision time. Models get worse, not better, when you drown them. I think of it in four layers:

1. Stable instructions: role, rules, output format, what’s off-limits. Short and specific.
2. Task: what this run is about.
3. Relevant knowledge: retrieved documents, the customer record, policy excerpts.
4. Working history: tool calls and results so far.

Retrieval: don’t preload everything. Keep what you inject small and cited, and test retrieval on its own: if the right document isn’t retrieved, no prompt will fix the answer.

Memory across runs is powerful and risky, because wrong information persists and compounds. Store it explicitly, make it inspectable, and let humans correct or delete it.

Guardrails and human-in-the-loop

The prompt is a request; guardrails are enforcement. “Never refund more than $500” in the system prompt is a hope. A refund tool that rejects amounts over $500 is a guardrail.

Put limits in the tool code, in the credentials (least privilege, scoped to the user or tenant), and in validation of the final output. Then sort every tool by risk:

Risk	Examples	Default
Read	Looking things up	Runs freely, within rate limits and data-access rules
Reversible write	Drafts, internal notes, tags	Automate once evals show it’s reliable
Irreversible or external	Customer emails, money, deletes, permissions	Human approval, at least at launch

An approval step only works if the reviewer sees exactly what will happen (the actual email, the actual amount), why, and can approve, reject or edit. If reviewers approve everything without reading, the gate has become theatre.

Evaluation

If you take one thing from this guide: don’t ship an agent without an eval set, and don’t change a prompt or model without running it.

1. 1

   Collect real cases

   Start with 20 to 50 real examples from tickets, historical requests and logs. Include the easy, the common and the ugly.

2. 2

   Define good

   Sometimes an exact answer. More often criteria: right tools called, no forbidden action, correct amount, right policy cited.

3. 3

   Grade deterministically first

   Did it parse? Did it call issue_refund when it shouldn't? Under N steps? Fast, cheap, no drift.

4. 4

   Run before every change

   Prompt, tool description, retrieval or model change: run the full set, compare to the baseline, look at which cases moved. Run each case several times.

5. 5

   Grow it forever

   Every production failure becomes a new eval case.

For open-ended outputs you’ll likely use an LLM as a judge. Treat it as a component that needs its own validation:

• Calibrate it against humans. Have people grade a sample and check the judge agrees often enough to trust.
• Give it a specific rubric. “Rate quality 1–10” is noise; “Does it state the refund amount?” is signal.
• Watch for known biases, such as favouring longer answers or the first option. Randomize order.
• Pin the judge, or you’ll mistake a judge change for an agent change.

🧠 Tool design and evals: check yourself

Score: 0 / 6

1. 1. Which tool is easiest to test and permission?

   run_sql(query)get_customer_orders(customer_id, since)manage_data(mode, payload)

   It does one thing with typed inputs. Arbitrary queries and mode flags are hard to test and to scope.

2. 2. The agent passed an unknown customer ID. Which error helps most?

   500 Internal Server ErrorError: not foundCustomer ID not found. IDs look like CUS-12345; use search_customers to find it by email.

   It says what went wrong and what to try next, so the model can recover instead of looping.

3. 3. Where should "never refund more than $500" be enforced?

   In the system promptIn the refund tool, which rejects larger amountsIn the tool description

   Prompts are requests. Code and credentials are enforcement.

4. 4. You tweaked a tool description. What do you do before merging?

   Try two examples by handRun the full eval set and compare which cases changed against the baselineShip it and watch the dashboard

   Small edits can fix three cases and quietly break five others.

5. 5. Agents are non-deterministic. How should each eval case be run?

   Once, and trust the resultSeveral times, looking at the pass rateOnly when it failed last time

   One run tells you little. Pass rates show whether a change really helped.

6. 6. Your LLM-judge scores jumped, but you changed nothing. Likely cause?

   The agent improved on its ownThe judge model changedThe eval set is too large

   Pin the judge version, or judge changes look like agent changes.

↺ Try again

Observability

When an agent goes wrong in production, the first question is “what did it actually do?” If you can’t answer in a couple of minutes, you’re not ready to launch.

Trace every step under one run ID: input, each model call (context and decision), each tool call with arguments and results, approvals, final output, errors, plus the prompt and model versions. Traces contain customer data, so apply the same retention, access and redaction rules as the source systems.

Track cost, latency and step count per run, per step and per tool, and look at the distribution: the expensive tail is where loops hide. A budget that only lives on a dashboard doesn’t stop anything at 3 a.m.; enforce it in the loop.

Capture feedback (thumbs, rejected approvals, edited drafts, escalations) and tie it to the trace. It’s your main source of new eval cases.

Failure modes to design for

None of these are exotic. Flip each card for the defences.

🃏 The four failures I see most

Tap a card to flip it

🔁 LoopsSame call, same arguments, again and again. Defences: step and cost caps, repeat-call detection, errors that suggest a next move, a clean way to say "I can't".🔧 Tool misuseRight tool, wrong arguments, or wrong tool because descriptions overlap. Defences: strict validation, distinct descriptions, evals on tool choice.💉 Prompt injection via tool outputsAny email, page or ticket the agent reads can carry instructions. Treat tool output as untrusted, gate sensitive actions, keep the blast radius small.📉 Silent quality driftNothing crashes; it just gets worse. Pin model versions, run evals on a schedule, sample production runs, watch step-count trends.

Rolling out

I rarely recommend a big-bang launch.

1. 1

   Stage 1

   Shadow mode

   Runs on real inputs; outputs go nowhere. Compare them with what humans did.

2. 2

   Stage 2

   Assist mode

   The agent drafts; a human reviews and sends everything.

3. 3

   Stage 3

   Partial autonomy

   Low-risk actions run automatically; risky ones still need approval.

4. 4

   Stage 4

   Expand

   Widen scope only when evals and production data support it.

Keep a kill switch at every stage. For leads signing off: What can it do without a human? What’s the worst action it could take, and what stops it? How will we know it’s getting worse? Who reads the traces, and how often?

Launch checklist

✅ Before the agent goes live

0 / 34 completed

• Scope: we've confirmed, with real examples, that a single call or a workflow isn't enough.
• Scope: the agent's job, and what's out of scope, are written down.
• Scope: escalating to a human is a defined, tested outcome.
• Tools: each does one thing and has a clear description, including when not to use it.
• Tools: state-changing tools are idempotent or protected against duplicate calls.
• Tools: defaults are safe (small pages, drafts, dry-runs).
• Tools: errors say what went wrong and what to try next.
• Tools: outputs are trimmed to what the model needs.
• Context: system instructions are short and specific.
• Context: retrieval has been tested on its own.
• Context: long runs are compacted without dropping the task or constraints.
• Context: cross-run memory, if any, is inspectable and correctable.
• Guardrails: hard limits live in code and credentials, not only in the prompt.
• Guardrails: least-privilege permissions, scoped to the user or tenant.
• Guardrails: irreversible and external actions require human approval.
• Guardrails: approval requests show exactly what will happen.
• Guardrails: final outputs are validated before they're used.
• Security: all tool output is treated as untrusted.
• Security: prompt injection is tested through every input channel the agent reads.
• Security: the blast radius of a successful injection is understood and acceptable.
• Evals: an eval set built from real cases exists, including hard and adversarial ones.
• Evals: deterministic checks are used wherever possible.
• Evals: any LLM judge is calibrated against human grading.
• Evals: they run before every prompt, tool, retrieval or model change.
• Evals: they also run on a schedule to catch drift.
• Observability: every run is traced end to end under one ID, with prompt and model versions.
• Observability: cost, latency and step count are tracked per run, with alerts.
• Observability: hard step and cost caps are enforced in the loop.
• Observability: traces follow our data retention and access rules.
• Observability: user and reviewer feedback is captured and linked to traces.
• Operations: there's a kill switch and a manual fallback.
• Operations: rollout is staged (shadow, assist, partial autonomy).
• Operations: someone owns reviewing traces and failures, on a set cadence.
• Operations: production failures become new eval cases.

↺ Reset