Research at Okta examined the impact of using GitHub Copilot in software engineering and found a gap between individual gains and overall throughput.
Despite increased engineer motivation and perceived skills, as well as reduced working hours, GitHub Copilot adoption revealed no statistically significant improvement in engineering output metrics. We call this disconnect the efficiency-throughput gap.
The Organizational Ohm’s Law (OOL) provides a practical and holistic framework for measuring engineering productivity, integrating subjective surveys with objective metrics to assess AI’s impact as a function of individual potential and organizational resistance.
Achieving organizational productivity gains requires more than deploying an AI-powered tool. It demands a deliberate effort to learn, adapt, and reinvent teams, processes, and infrastructure to align with AI-augmented developer capabilities.
Beyond the Hype: The Efficiency-Throughput Gap with GitHub Copilot, by Fei Liu and Todd McKinnon.
The usual caveat here: The research looked at tool usage for a time period in 2024. Given the many advancements since that is of course a very long time ago indeed.
It does highlight an important property often seen in systems thinking and constraints management: Speeding up one part of a larger system does not automatically make the whole more performant. Optimizing the whole does not usually mean optimizing the individual parts.