Running a developer productivity initiative in the age of AI coding tools

Your company bought AI coding tool licenses. Sooner or later someone asks “Is it working?”, and if you run engineering, that question lands on you.

Vendor dashboards, anecdotes and accepted-suggestion counts don’t answer it. What does: measurement discipline, a controlled comparison, and a monthly review people take seriously.

1. AskFour questions you could be wrong about
2. ↓→
3. Baseline2–3 months of history, before rollout
4. ↓→
5. PilotPilot vs comparison teams, 8–12 weeks
6. ↓→
7. ReviewOne-page scorecard, monthly

↺ Decide: expand, adjust or stop

Start with the question you’re really answering

“Is AI making us more productive?” is too vague to measure. Split it into questions you could be wrong about. If a metric doesn’t answer one of them, it stays off the scorecard.

🃏 The four questions

Tap a card to flip it

SpeedAre we delivering changes to users faster?QualityIs quality holding or improving while we do it?ExperienceAre developers spending less time on work they consider low value?CostIs the cost of the tools justified by the above?

Why “lines of code” and “% of code written by AI” are bad metrics

Neither number should reach your executive team as evidence of success.

🎮 Good metric or vanity metric?

1 / 9 · Score: 0

Where does each number belong on your scorecard?

Outcome or guardrailUsage onlyVanity: drop it

Next →

/ 9

↺ Try again

Choose a small, balanced metric set

Pick metrics that pull against each other, so you can’t improve one by quietly degrading another. Five to seven total; if you can’t explain a number in one sentence, cut it.

1. Flow and delivery metrics

The DORA metrics balance speed and stability. Pick one definition of cycle time and stick with it.

Metric	What it tells you
Cycle time / lead time for changes	First commit (or PR opened) to production
Deployment frequency	How often you ship
Change failure rate	Share of deploys causing an incident, rollback or hotfix
Time to restore service	How fast you recover
PR review time (diagnostic)	Whether faster authoring just moved the bottleneck to review

2. Developer experience signals

Delivery numbers tell you what happened, not where the friction is. SPACE is a good reminder that productivity has several dimensions (satisfaction, performance, activity, communication and collaboration, efficiency and flow).

3. Quality and cost guardrails

Not targets, tripwires: change failure rate, escaped defects or incidents, tool cost per active developer.

Keep it small

A 25-metric dashboard gets looked at once and then ignored. Five to seven numbers, each explainable in a sentence.

Establish a baseline before rollout

The step most organizations skip, and the one that makes everything else credible. You can’t reconstruct “before” afterward: behaviour changes, teams shift, memory lies.

✅ Baseline checklist

0 / 4 completed

• Extract two to three months of history from version control, CI/CD and incident tools
• Run the developer survey before anyone gets the tools (or before the wider rollout)
• Write down the context: team sizes, major projects, on-call load, holidays, reorgs (December isn't March)
• Look at variance, not just averages: if a metric swings widely, a small post-rollout change is probably noise

↺ Reset

Run a controlled pilot

A pilot beats a big-bang rollout because it gives you a comparison group. Without one, you can’t separate the tools from everything else going on.

Team selection

Two to four pilot teams, a similar number of comparison teams, matched on type of work, codebase age and stack, size and seniority, and release process.

Duration

At least eight to twelve weeks. The first weeks measure the learning curve, not the steady state; expect some people to be slower at first while they find their footing.

What to compare

Compare each group to its own baseline, then compare the size of the change between groups. That controls for org-wide events like a new CI system, a release freeze or a reorg.

Track core delivery metrics, PR size and review time, survey results at start and end, and lead notes every two weeks. A few teams won’t give you statistical certainty, but they will give a directional signal and a clear map of where the tools help.

The roles involved

Without clear owners, an initiative becomes a license purchase with a Slack channel.

🃏 Who does what

Tap a card to flip it

Executive sponsorVP/Director or CTO. Owns the question and the expand/adjust/stop call, keeps it out of performance reviews, presents uncomfortable results too.Platform / enablementOwns config, access, security review, metrics pipeline, scorecard, guides, training. If it's nobody's actual job, it stalls within a quarter.Team leadsFind good use cases, give feedback every two weeks, flag review overload or opt-outs, explain the numbers in context.DevelopersThe implicit fourth role. They need to know what's measured, why, and that it won't be used to rank them.

Monthly review cadence

Once a month, one hour, same people: sponsor, enablement lead, pilot and comparison team leads.

1. 1

   15 min

   Scorecard walkthrough

   What moved, versus baseline and versus the comparison group.

2. 2

   20 min

   Team lead context

   What's behind the numbers? What surprised you?

3. 3

   15 min

   Friction and wins

   What blocks adoption? Which use cases clearly work?

4. 4

   10 min

   Decisions

   One to three actions with owners. Review them at the start of next month, or this turns into a status update.

One-page scorecard template

One page, outcomes separated from usage, and a comment on every line. A number with no explanation invites the wrong interpretation.

Area	Metric	Baseline	Last month	This month	Trend	Comment
Delivery	Median cycle time (PR opened to production)
Delivery	Deployments per week
Delivery	Median PR review time
Quality	Change failure rate
Quality	Production incidents
Experience	Survey: ease of getting changes merged (1–5)
Experience	Survey: time lost to toil (1–5)
Usage	Weekly active users of AI tools / licensed seats
Cost	Tool cost per active developer

✅ Before you send this month's scorecard

0 / 7 completed

• Every row has a baseline value
• Every row has a comment, not just a number
• Usage is kept separate from outcomes
• Pilot results are shown next to the comparison group
• "What we learned this month" (two or three bullets) is filled in
• Decisions have owners
• Risks and open questions are listed

↺ Reset

When “we feel faster” but the numbers don’t move

It will probably happen: positive surveys, flat cycle time. Work through it in this order.

1. 1

   Check where the time goes

   Split cycle time into coding, waiting for review, in review, waiting for deploy. Time saved writing code often gets absorbed by review queues, flaky CI or manual releases. The tools work; fix the constraint.

2. 2

   Check what the saved time is spent on

   More tests, tech debt, exploring alternatives: real value delivery metrics won't show. Ask leads, and say so in the scorecard.

3. 3

   Check the feeling itself

   People can feel faster without being faster; time spent prompting, reading and correcting is easy to underestimate. You need both perception and measurement.

4. 4

   Check your measurement

   Weekly fixed deploys mean deployment frequency can't move. A few very long PRs skew the mean: use the median or look at the distribution.

5. 5

   Report it honestly

   "Devs say it helps, delivery hasn't moved yet, review looks like the bottleneck, here's our plan" beats a cherry-picked chart.

Pitfalls to avoid

Flip each card for the way out.

🃏 Pitfalls

Tap a card to flip it

Goodhart's lawTarget deploy frequency and people split deploys. Use metrics to learn and spot problems, never as targets tied to rewards.Surveillance fearsIn writing: team level or above only, never in performance reviews, here's what we collect and who sees it. Break it once and surveys go dishonest.Team vs teamA 15-year-old monolith and a greenfield service aren't comparable. Compare each team to its own baseline; cross-team only for matched pilot pairs.Declaring victory earlyMonth one is novelty or learning curve. Commit to the full pilot duration before concluding.Ignoring downstream effectsMore code faster can mean more review, bugs and maintenance. Watch PR size, review time and change failure rate.

🧠 Quick check

Score: 0 / 4

1. 1. Pilot teams' cycle time improved. What proves the tools did it?

   The vendor dashboardThey improved more than matched comparison teams over the same periodDevelopers say they feel faster

2. 2. Where does "% of code written by AI" belong?

   Headline success metricA separate Usage sectionIndividual performance reviews

3. 3. Surveys are positive but cycle time is flat. First move?

   Declare the tools a failureBreak cycle time into stages to find the bottleneckSet a cycle time target

   Saved authoring time is often absorbed by review or release.

4. 4. How long should the pilot run before you conclude?

   Two weeksEight to twelve weeks at leastUntil the first good month

↺ Try again

What to do this week

✅ This week

0 / 5 completed

• Write down the four questions (speed, quality, experience, cost) and share them with your leadership team
• Pull the last three months of delivery data, even roughly, as a baseline to refine
• Draft an 8–12 question developer survey and send it before any further rollout
• Name the owners: sponsor, enablement lead, pilot and comparison team leads
• Publish a short measurement charter: what you measure, at what level, and what the data will never be used for

↺ Reset