Stream the Latest Episode
Listen and watch now on YouTube, Spotify and Apple. See the episode transcript at the top of this page, and timestamps for the episode at the bottom.
Brought to You By
Statsig — The unified platform for flags, analytics, experiments, and more.
Graphite — The AI developer productivity platform.
—
In This Episode
There’s no shortage of bold claims about AI and developer productivity, but how do you separate signal from noise?
In this episode of The Pragmatic Engineer, I’m joined by Laura Tacho, CTO at DX, to cut through the hype and share how well (or not) AI tools are actually working inside engineering orgs. Laura shares insights from DX’s research across 180+ companies, including surprising findings about where developers save the most time, why devs don’t use AI at all, and what kinds of rollouts lead to meaningful impact.
We also discuss:
The problem with oversimplified AI headlines and how to think more critically about them
An overview of the DX AI Measurement framework
Learnings from Booking.com’s AI tool rollout
Common reasons developers aren’t using AI tools
Why using AI tools sometimes decreases developer satisfaction
Surprising results from DX’s 180+ company study
How AI-generated documentation differs from human-written docs
Why measuring developer experience before rolling out AI is essential
Why Laura thinks roadmaps are on their way out
And much more!
Interesting ideas from the conversation
Here are three interesting observations that came in this conversation:
Idea #1: Code is a liability!
Laura (at 16:04): “One of my more controversial opinions is that source code is a liability. It sounds controversial — but then when people think about it, they realize, yeah, it actually is a liability!
We're in a world where it is trivially easy to produce a tremendous amount of source code. And so what does that actually mean for productivity and business impact when what could have been written in one line is now written in five lines?
Do we really want to measure AI impact in terms of lines of code generated? I certainly don't.
We did not include acceptance rate in our framework for good reason. I think it does give insight into whether the tools are fit for purpose. But when we're looking at broadly measuring business impact and the impact on developer experience and the impact on the business, acceptance rate is just such a tiny part of the story.”
Idea #2: Roadmaps are on their way out
Gergely (at 55:45) “As an end user of a customer of a company that has invested heavily in AI made that be Google, Microsoft, a startup. What should you be seeing?
Laura: “As an end user, what I expect is faster time to market. And that's really on the other side, on the building side, what we're trying to emphasize and what a lot of our conversations have focused on with other executives and engineering leaders. We're really trying to reduce the amount of time to market. This has a lot of implications.
Another one of my unconventional opinions, I think roadmaps are on their way out in the age of AI. Companies that are going to win with AI are not ones that think about things in roadmap sequential form, but think about it more as experiment portfolios. And so rapid experimentation and trying to figure out what delights your customers is going to help companies win.
The companies that will win are the ones that focus or that already have muscles to do experimentation, A/B tests, trying to figure out, you know, how to delight their customers.
As an end user, what I don't want is thrashing as an end user. And I could see that happening. It's just like, we're just gonna, cause now there are fewer reasons to say no to things. And there's probably good reason that some of those things weren't built.”
Idea #3: AI time savings are smaller than most people assume
Laura (at 27:34): “Typing speed has never been the bottleneck in development. Now we have all this code generated faster than we can type. That's great. But it still takes me time to review that code!
It's not that we're saving time because we don't have to type. A lot of that time, we're just reallocating to reviewing or other parts of code authoring that's not typing. (…)
The industry average [of spending time on coding] is like 25%. There was a study at AWS that an average AWS engineer only spends 20 % of their time coding. And so when we apply AI to the coding tasks, we're only working with 20 % of that time to begin with. And when we save 10 % of that time, that actually doesn't amount to, ‘we can, you know, ship 10 new product lines overnight!’ ”
The Pragmatic Engineer deepdives relevant for this episode
Timestamps
(00:00) Intro
(01:23) Laura’s take on AI overhyped headlines
(10:46) Common questions Laura gets about AI implementation
(11:49) How to measure AI’s impact
(15:12) Why acceptance rate and lines of code are not sufficient measures of productivity
(18:03) The Booking.com case study
(20:37) Why some employees are not using AI
(24:20) What developers are actually saving time on
(29:14) What happens with the time savings
(31:10) The surprising results from the DORA report on AI in engineering
(33:44) A hypothesis around AI and flow state and the importance of talking to developers
(35:59) What’s working in AI architecture
(42:22) Learnings from WorkHuman’s adoption of Copilot
(47:00) Consumption-based pricing, and the difficulty of allocating resources to AI
(52:01) What DX Core 4 measures
(55:32) The best outcomes of implementing AI
(58:56) Why highly regulated industries are having the best results with AI rollout
(1:00:30) Indeed’s structured AI rollout
(1:04:22) Why migrations might be a good use case for AI (and a tip for doing it!)
(1:07:30) Advice for engineering leads looking to get better at AI tooling and implementation
(1:08:49) Rapid fire round
References
Where to find Laura Tacho:
• LinkedIn: https://www.linkedin.com/in/lauratacho/
• Website: https://lauratacho.com/
• Laura’s course (Measuring Engineering Performance and AI Impact) https://lauratacho.com/developer-productivity-metrics-course
Mentions during the episode:
• AI Measurement Framework: https://getdx.com/research/measuring-ai-code-assistants-and-agents/
• Are Coders' Jobs At Risk? AI's Impact On The Future Of Programming: https://www.forbes.com/sites/sylvainduranton/2024/04/15/are-coders-jobs-at-risk-ais-impact-on-the-future-of-programming/
• Guide to AI-Assisted Engineering: https://getdx.com/guide/ai-assisted-e
• DORA's Impact of Generative AI in Software Development report: https://dora.dev/publications/#impact-of-generative-ai-in-software-development
• Booking Case Study: https://getdx.com/customers/booking-uses-dx-to-measure-impact-of-genai/
• AI coding assistants wave goodbye to junior developers: https://www.cio.com/article/3509174/ai-coding-assistants-wave-goodbye-to-junior-developers.html
• OpenAI Just Released a Coding Tool to ‘Help’ Programmers (Replace Their Jobs, Probably): https://gizmodo.com/openai-just-released-a-coding-tool-to-help-programmers-replace-their-jobs-probably-2000603414
• Cursor: https://cursor.com/
• Measuring Software Engineering Productivity: https://newsletter.pragmaticengineer.com/p/engineering-productivity
• DORA, SPACE, and DevEx: Which framework should you use?: https://getdx.com/guide/dora-space-devex/
• Abi Noda on LinkedIn: https://www.linkedin.com/in/abinoda/
• How AI is changing software engineering at Shopify with Farhan Thawar: https://newsletter.pragmaticengineer.com/p/how-ai-is-changing-software-engineering
• How Linux is built with Greg Kroah-Hartman: https://newsletter.pragmaticengineer.com/p/how-linux-is-built-with-greg-kroah
• Amazon says developers spend a surprisingly small amount of time per day coding: https://www.businessinsider.com/amazon-developers-spend-only-one-hour-coding-daily-aws-ai-2024-12
• Impact of Generative AI in Software Development: https://dora.dev/research/ai/gen-ai-report/
• Vercel Clerk: https://vercel.com/marketplace/clerk
• WorkHuman: https://www.workhuman.com/
• Workhuman increases ROI from AI assistants 21% with DX: https://getdx.com/customers/workhuman-increases-roi-from-ai-assistants-21-with-dx/
• Visual Studio: https://visualstudio.microsoft.com/
• Jesse Adametz on LinkedIn: linkedin.com/in/jesseadametz
• DX Core 4: https://getdx.com/core-4-reporting/?
• Indeed: https://www.indeed.com/
• Accelerating code migrations with AI: https://research.google/blog/accelerating-code-migrations-with-ai/
• Granola: https://www.granola.ai/
• Write Useful Books: A modern approach to designing and refining recommendable nonfiction: https://www.amazon.com/Write-Useful-Books-recommendable-nonfiction-ebook/dp/B0983HFQX7
• Unsavory Truth: How Food Companies Skew the Science of What We Eat: https://www.amazon.com/dp/1541697111/?
—
Production and marketing by Pen Name.
Share this post