Exploring Prosperity examines how technology, markets, money, and human ingenuity shape our future. Evidence, not cheerleading.
The evidence on AI productivity is surprisingly contradictory.
Artificial intelligence made experienced software developers 19% slower at completing their work1. Yet those same developers believed it had made them 20% faster. In another study, customer service agents using AI increased their productivity by 15%, with the largest gains among less experienced workers2. And in an experiment involving 640 Kenyan entrepreneurs, access to AI produced no significant improvement in average business performance3.
What explains such different results?
On September 15, I launched an AI research agent to monitor empirical studies and company experiences examining AI’s impact on productivity. I wanted to understand how AI affects the speed and quality of work, employment, entrepreneurship, and the economics of businesses of different sizes.
The agent has assembled 82 records: 29 analytical research entries and 53 company accounts. The research includes experiments, observational studies, and surveys. Company accounts offer evidence of how AI is being deployed, although their claims are generally harder to verify.
In early October, I interviewed Erica Groshen (HERE), former commissioner of the Bureau of Labor Statistics. She described more than 1,000 studies examining AI’s effects on labor, with widely varying findings. My smaller collection, which includes earlier research, is beginning to reveal why drawing broad conclusions is so difficult.
Three factors appear to explain some of the divergence: what researchers measure, the suitability of AI for the task, and the judgment of the person using it.
First, studies measure different outcomes. Some examine speed, others accuracy, quality, sales, or profits. In a randomized experiment by researchers at Stanford and MIT, accountants using AI to classify financial transactions improved their accuracy by 17.5 percentage points, but total completion time did not decline significantly. They produced more accurate work without finishing faster. Counting hours saved won’t tell us how much additional economic value better work creates.
Second, AI’s usefulness depends on the task. In an experiment involving Boston Consulting Group consultants, AI improved both speed and quality on assignments within its capabilities. But performance deteriorated when consultants relied on it for problems beyond those capabilities.
Third, the user’s judgment influences the outcome, although the relationship is complicated. Novice customer service agents gained the most, apparently benefiting from knowledge embedded in the AI system. In the Kenyan experiment, initially weaker businesses performed worse with AI access, while stronger businesses showed signs of improvement.
The developer experiment offers another perspective. Conducted with early-2025 AI tools on experienced programmers’ own repositories, it found that AI assistance increased completion time. The researchers suspect that the overhead of using AI may be particularly costly when developers already know their codebases exceptionally well.
Access to powerful technology alone did not ensure better performance. Perhaps AI helps most when it supplies knowledge the user lacks and the user can recognize which recommendations deserve trust. The studies suggest this possibility without establishing a general rule.
I spent 27 years on Wall Street, working alongside exceptionally intelligent people making consequential decisions. Yet even in that environment, we were constrained by how much information we could gather, how effectively we could analyze it, and how clearly we could communicate our conclusions. I contributed my share of lengthy, confusing explanations, emails, and PowerPoints.
My experience with AI makes me optimistic about its economic implications. It can process extraordinary quantities of information, identify relationships that might otherwise escape notice, and communicate findings with remarkable clarity and consistency. Of course, its polished explanations can also conceal errors.
Better information and analysis could reduce costly mistakes, uncover opportunities, and improve the allocation of capital. Faster decisions could accelerate subsequent decisions, allowing improvements to compound throughout organizations and supply chains. These benefits may take years to appear in conventional economic statistics.
I see some of this in my own work. AI allows me to investigate questions more thoroughly, test hypotheses, and reorganize data. Increasingly, I can work with entire ideas rather than labor over individual sentences.
But the developers in the opening experiment also believed AI was helping them. How do I know my own experience is different?
I don’t claim that AI saves me time. I spend considerable effort challenging its conclusions, checking evidence, and correcting mistakes. I believe the finished work expresses my ideas far more clearly and effectively than I could previously manage, although that judgment is subjective, too. The quality depends heavily on the effort I put into it.
Even my research agent illustrates the need for scrutiny. Its initial inventory mixed research studies with company accounts, requiring a closer audit to distinguish the evidence from the anecdotes.
My collection remains limited, the technology is changing rapidly, and many experiments examine narrow tasks over short periods. The evidence is useful, but the broader economic consequences remain uncertain.
This uncertainty worries me because the research offers plenty of ammunition for both AI optimists and skeptics. Each can find credible studies supporting what they already believe. Policymakers face the same temptation, with potentially greater consequences. Policies based on premature conclusions about AI’s effects on jobs, businesses, or economic growth could produce unintended consequences, whether those conclusions are overly optimistic or pessimistic.
My own experience suggests that AI will likely substantially improve the quality and speed of decisions throughout the economy. The studies suggest that those benefits will be uneven. Understanding AI’s effects requires measuring the value of better work, identifying the tasks where AI helps, and learning how people develop the judgment to use it effectively.
How broadly people develop that judgment may determine much of AI’s economic impact.
You can find the full inventory of studies HERE.
Thank you very much to our sponsors:
Foundation: When it comes to protecting your Bitcoin, you shouldn’t have to simply trust the company that made your hardware wallet.
Foundation believes in verifiability over reputation. Its firmware and apps are open source, its builds are reproducible, and both Passport Core and Passport Prime have undergone independent hardware security audits. The point isn’t simply that Foundation says its products are secure. It’s that you can verify that for yourself.
Foundation is also expanding what users can control:
Passport Prime goes beyond Bitcoin. It can manage passwords and two-factor authentication codes, serve as a FIDO2 security key, and provide encrypted storage—all on a device you own and control. Foundation also plans to enable app sideloading, opening Prime to applications from independent developers.
Users can now add their own entropy. The latest Envoy and KeyOS releases allow users to introduce their own randomness when creating a seed, reducing reliance on the device itself to generate it.
Order a new Passport Prime and receive a free bumper case using Code PROSPERITY at → foundation.xyz
River Financial: If not cold storage for your Bitcoin, I’d highly recommend River for buying and holding- River is engineered to protect your Bitcoin over the long run. Use this link for discounts: https://river.com/signup?r=T7GGAF7G
The METR study.