How to Measure Whether AI Actually Saved Time
A short AI answer is not the same thing as a faster workflow.
To find out whether AI saved time, compare two complete versions of the same task:
- a person does it with the ordinary tools they normally use;
- a person does it with the AI tool available.
Use the same input, the same quality standard, and the same definition of “finished.” Count setup, searching, waiting, checking, corrections, and retries. Stop the clock only when the final result passes the acceptance test.
Conventional versus AI-assisted
The conventional workflow is not “a person working with no tools.” It is the normal process without the AI tool being tested. A developer may still use an editor, documentation, internet search, or Stack Overflow.
The AI-assisted workflow uses the same process with the AI tool added. The person remains responsible for checking and accepting the result. AI-assisted does not mean autonomous.
A real measured example
A controlled experiment by Peng and colleagues tested GitHub Copilot with professional programmers. The research paper is available on arXiv.
The task was concrete: build a JavaScript HTTP server. Each participant received the same kind of private GitHub Classroom repository, a task description, and starter code. The repository included 12 fixed tests. The task was finished when the first submitted version passed all 12 tests.
The conventional group did not have Copilot. They could use ordinary resources such as web search and Stack Overflow.
The AI-assisted group had Copilot available after a short introduction. Copilot suggested code in the developer’s editor. The developer could accept, reject, change, test, and debug the suggestions.
The researchers measured the whole job—from repository creation to the first commit passing all 12 tests—not the time until Copilot produced its first suggestion.
Among the participants who completed the task:
- without Copilot: 160.89 minutes on average;
- with Copilot: 71.17 minutes on average;
- reported difference: 55.8% faster with Copilot;
- 95% confidence interval: 21% to 89%;
- statistical result: p = 0.0017.
The study included 95 accepted professional programmers, with 45 assigned to the Copilot group and 50 to the control group. Thirty-five completed the task and survey. It ran in 2022, before Copilot became generally available.
This is evidence that Copilot helped with this particular standardized coding task under those conditions. It is not a guarantee for every task, worker, or current AI product.
Why another study reports a different kind of number
A separate field study of 5,179 customer-support agents examined a generative-AI conversational assistant. The National Bureau of Economic Research paper reported 14% more issues resolved per hour on average, and 34% more for novice and low-skilled agents.
That study measured output per hour, not the average minutes needed for one ticket. Do not rewrite “14% more issues per hour” as “each ticket took 14% less time.” They are different measurements.
How to test your own workflow
Choose a low-risk, repeatable task using public, synthetic, or disposable data. Do not use confidential, personal, client, regulated, or identifying information.
Write down:
- The task: what the person must produce.
- The input: exactly what data goes into the process.
- The tools: normal tools in the conventional version and the named AI tool in the assisted version.
- The finish line: the test that decides whether the result is acceptable.
- The clock: when timing starts and stops.
Then run both versions.
For each version, record:
- setup and preparation;
- active work;
- waiting;
- checking or testing;
- corrections and rework;
- recovery from a failed result;
- total time;
- whether the final result passed.
The first AI draft is not the finish line. If it takes 10 seconds to produce but 20 minutes to check and repair, the AI did not save 20 minutes. It shifted the work into supervision.
The decision rule
The AI-assisted workflow is a useful improvement only when it reaches the same quality standard in less total time—or produces meaningfully more acceptable work in the same time—without creating unacceptable errors or review burden.
Report the result narrowly:
“In this task, with this input, tool, worker group, and acceptance test, the AI-assisted workflow reached the accepted result faster.”
Do not turn one task into a claim about an entire profession.
Sources and limits
- Peng et al., “The Impact of AI on Developer Productivity: Evidence from GitHub Copilot”
- Brynjolfsson, Li, and Raymond, “Generative AI at Work”
The studies used different tasks and different measurements. The Copilot study measured completion time for one coding assignment. The customer-support study measured issues resolved per hour. Neither establishes a universal AI time-saving rate.