How to measure the impact of AI agents on your team

Engineering leaders hear the same thing over and over: "We're faster with AI." The CFO asks the next question: how much faster, and what does it cost? Usually there's nothing to answer with.

Why gut feeling doesn't work

In 2025, METR researchers ran a controlled experiment with experienced open-source developers. Participants expected AI to speed them up, and afterwards they believed it had, by about 20%. The measurement showed the opposite: tasks done with AI took longer. (Study overview)

This doesn't mean AI is useless. It means a team's self-assessment isn't reliable here, and you need to measure with data.

Why counting lines of code is pointless

In October 2026, Harvard economists published a working paper based on data from 700+ companies. After AI agents were adopted, code volume grew by about 30%, commits by 20% and pull requests by 23%. The number of closed tasks and epics didn't change in any statistically meaningful way, and getting through review took 49% longer. (Overview)

More code is not a result. The bottleneck has moved to review and rework.

What to measure

If you're evaluating any AI tool on your team, look at four groups of metrics. A CFO can follow all of them, and you can collect them without heavy analytics.

Onboarding. How long it takes from a new developer's first day to their first accepted MR. For the "before" number, ask the team lead about the last two or three new hires.

Review returns. The share of MRs sent back for rework at least once. Take a sample from the month before the rollout and from the first month after.

Rework after AI. How many hours a week go into fixing code the agent wrote. For "before," a team estimate will do. After the rollout, part of this can be counted automatically.

Senior developers' time. How many hours a week go into answering the same questions about the project. According to Atlassian, searching for information is now the top reason developers lose time.

How to run the measurement

1. Before you start, pick 3–4 metrics and record the "before" values. Without them, all you'll have at the end is impressions again.
2. Agree on the date when you'll make the decision, and book that meeting right away.
3. During the evaluation, don't change the team or the type of tasks. Otherwise the comparison stops meaning anything.
4. At the end, compare the values and convert the difference into hours and dollars at your team's rate.

How we do it

AI Council shows some of this data on its own, from agent activity: which knowledge and rules the agent got in each session, how often it checks the rules, and what state branches are in at review. In our measurements, the savings in time and tokens are 30–45% on a single task in a clean test, and 40–60% on a group of tasks when a team and several agents work together.

You can get a rough sense of the numbers in the ROI calculator.