Measuring impact without vanity metrics

Sooner or later someone asks: is this agent thing actually working? The easy answer is a dashboard of activity. More code, more PRs, more prompts. It looks great and proves nothing.

Agents make activity cheap. So any metric that counts activity will go up whether or not you’re shipping better software.

Activity vs outcomes

PRs merged sits in the top right on purpose. It’s closer to an outcome, but agents make it trivially easy to split work into more PRs. Watch it only next to rework and lead time.

Swap the vanity metric

Flip each card for what I’d measure instead.

🃏 Vanity metric, better question

Tap a card to flip it

Lines of code generatedHow long from ticket start to running in production, for comparable work? More code is often a cost, not a win.Percent of code written by AIWhat share of agent-assisted PRs merged without a major rewrite? That tells you whether the delegation actually saved time.Number of prompts or sessionsWhere do people still get stuck? Usage says people tried it, not that it helped.PRs opened per developerHow much merged work needed a follow-up fix or revert within a few weeks? Speed that comes back as rework isn't speed.Time saved, self-reported per taskDid cycle time for the team's real work change against a baseline? Individual guesses tend to be generous.Tokens or spend per developerCost per shipped change, looked at next to quality. Low spend with lots of rework is not a saving.

A measurement plan that fits on a page

1. 1

   Before

   Take a baseline

   Pull a few weeks of lead time, rework and review wait time from your tracker and repo history, before changing how the team works.

2. 2

   Week 1

   Tag the work

   Mark agent-assisted PRs with a label or template field. Without it, you can't compare anything.

3. 3

   Monthly

   Compare like with like

   Compare similar kinds of work: bug fixes to bug fixes, small features to small features. A mix shift can fake an improvement.

4. 4

   Monthly

   Ask the team

   Five minutes, three questions: where did agents help, where did they cost you time, what would you stop doing? Numbers say what, people say why.

5. 5

   Quarterly

   Decide something

   A metric nobody acts on is decoration. Each review should change a convention, a guardrail or where you use agents.

Before you move on

✅ Key takeaways

0 / 5 completed

• I don't report activity metrics like lines, prompts or percent of AI code as impact.
• We have a baseline for lead time, rework and review wait time.
• Agent-assisted work is tagged so we can compare like with like.
• We measure the team's system, never individual developers.
• Every metrics review ends with a decision.

↺ Reset