For today’s issue, we’ve dug into your 900+ responses to look for trends in AI tool usage among software engineers and engineering leaders. This article surfaces insights that are less about specific tools, and more about the effect these tools have on tech professionals. We cover:

1. Costs. Unsurprisingly, companies pay for most tool usage, and those responsible for budgets are increasingly nervous that AI-related costs are headed only one way: up.

2. Usage limits. Around 30% of respondents say they have hit limits. Switching tools, upgrading plans, or moving over to API pricing are common responses.

3. Impact on “Builders.” Folks who make larger code changes and do “quality-of-life” work are builders, and they’re also dealing with more AI slop. Some also grapple with a loss of professional identity.

4. AI tools speed up “Shippers.” Engineers who focus more on getting things done are the most positive about AI tools. But they also add tech debt faster and might build the wrong things.

5. “Coasters:” learning faster while generating AI slop. Less adept engineers can uplevel faster with AI, but they generate a lot of “AI slop” while doing so, which frustrates builders.

6. Changing software engineer & engineering manager (EM) roles. Engineers have to orchestrate and context switch more often, while engineering managers can be more hands on. It’s interesting to see the engineer and manager roles becoming more similar.

7. Other impacts on the craft. We’re going from “how” to build to “what” to build, solo devs are seeing improved results, workloads are increasing with AI tools, and more.

We previously published a detailed summary of the survey which focused on AI tooling for software engineers, covering the most-used AI tools, trends, AI agent usage, company size and usage, and tools that engineers love.

1. Costs

Concern about the cost of AI tools is a trend throughout the survey, with around 15% of respondents mentioning it in some way.

Tech companies foot the bill for the majority of spending on AI tools. More respondents say their employers pay for AI coding tools than those who say they pay themselves, and predictably, employers fund more expensive packages than what individuals buy personally.

Companies commonly pay for “max” plans with the likes of Claude Code, Cursor, and Codex (around $100-200/month per engineer), although some companies’ budgets only stretch to $20/month per engineer – around the price point of GitHub Copilot, and the cheapest Claude or ChatGPT subscriptions.

The most-mentioned AI tool spending patterns:

• When companies pay: ~$200/month plans. Many have enterprise subscriptions, sometimes with subsidies and vendor lock-in. Some companies allow usage-based coverage on top of monthly plans.

• When personally paying for tools: ~$20/month or free tiers. This can stack up across different tools. Around 5% of respondents have separate work and personal subscriptions, and free tier usage is widespread for personal use.

For now, companies seem to be in the experimentation phase with AI tools, and several respondents say that they believe their companies have unsustainable AI-tooling budgets. This is likely because businesses are figuring out the best way of leveraging the tools, and the message to engineers at such places is to not worry about price and usage while that unfolds. A CTO at a small, US-based company shares:

“Right now, we’re not sweating the costs because we’re trying to evolve best practices for the tools, but that has resulted in some devs really blowing through budget, so we may start instituting caps on spending.”

Breaking the budget

At small and mid-sized companies, leadership teams seem more comfortable about going over budget, than engineers running out of budget. There are more accounts from C-level folks and founders about racking up large bills than there are from engineers. A CPTO (Chief Product and Technology Officer) at a mid-sized company:

“I ran up several monthly bills of $600 with Cursor. We have the dev team subscribed to ~$100/month plans. We’re now in the process of moving the rest of the team to Claude Code, as we can get more resources for around $100/month in cost.”

Top spenders can be allocated higher budgets. A number of tech businesses have separate, larger budgets for their heaviest AI users. A senior C++ engineer working in the video game industry says:

“I’ve become my team’s AI champion. In theory, my limits are higher than normal, but I keep myself limited to what others can use, so I can show them useful things they can do.”

UK and EU companies worry more about budgets than US-based ones. Most responses which mention finance teams pushing back against spending even $30-50/month per engineer on AI tools are based in the UK and EU. One amusing example is a 10-person, seed-stage startup where the CEO questioned why they were paying as much as £25/month per engineer for one of the cheapest AI tools around.

In general, it feels like European companies want to see clear value-add in order to justify an increase in tooling spend, whereas US companies are more comfortable with investing first and measuring impact later. At present, the impact of these tools is hard to quantify.

A niche approach is that of AI teams educating devs to use cheaper models. Some European companies go as far as offering education to new joiners on using cheaper models. From an AI Enablement Lead at a 1,000+ person, digital transformation company:

“Within our organisation, we’ve had incidents where our Claude users have overshot their limits. We’re now attempting to educate devs in knowing the difference between different models (knowing when to use Claude Sonnet versus Claude Opus).”

Cost trajectory worries

The cost trajectory of AI tools is generally considered unsustainable in our survey. Devs using the tools heavily tend to hit usage limits, and their employers then have to pay more. At places with API-based pricing, usage is increasing. Those in leadership positions who are responsible for budgets are generally concerned about the direction of costs.

Subsidies are keeping things at bay – for now. A common enough pattern in our survey is of heavily-subsidized enterprise plans that come with vendor lock-in. Several responses raise concerns about what will happen when the subsidies run dry. Experienced engineering leaders recall that cloud providers also played the same game of subsidizing for a few years, then raising prices when a customer was fully “locked in.”

The AI hype cycle is dampening awkward conversations about budgets at some places. A principal engineer at a Fintech tells us:

“The AI hype has created a special, generous budget for AI tools, and there’s no effective budget – yet!”

But some finance teams are getting grumpy. A CTO at a sports-tech company says:

“It’s hard to keep our CFO supportive about investing in these tools because the productivity benefits have proven difficult to conclusively prove. The point that resonated the most was the loss of value when people hit daily limits: having to stop work immediately! Surprisingly, our CFO is still pushing back, despite having experience of getting a lot of value through their own AI usage with their spreadsheets.”

Most survey respondents think the price of AI tools will have to rise. If that happens, it would cause problems at several companies – particularly those in Europe:

“I cannot see how the spend on AI tools is fiscally sustainable in its current form; Max 100 with Claude Code is $100 a month. A single small task powered by Kimi K2.5 using OpenCode is $5, mostly in input cost. If we assume that the third party inference providers are doing so at a sustainable price, the much more expensive Opus model cannot be sustainable, never mind profitable at these plan costs.” — Founder at a seed-stage company, Europe.

“From the economic perspective, at some point, these companies will need more funding or profit, I’m curious how much it costs them to have a proper agent, and still become profitable. It feels slow when you run out of credits when working on repetitive tasks.” — Principal Software Engineer at a seed-stage company, Europe

2. Usage limits

Another major trend in our survey results is the topic of usage limits:

• Hitting limits: ~30% of respondents. Running out of tokens or hitting reset limits is frustrating and disruptive, especially when you’re working on a task or are in a flow state. The majority of respondents who complain about hitting limits are on cheaper plans (typically $20/month.) But this issue is also mentioned about higher subscription levels.

• Under the limit: ~20% of respondents. Avoiding usage caps generally correlates with being on more expensive plans with higher limits, or in roles with enough non-coding work for it not to matter, or when devs do enough work “manually” for AI usage to not be an issue.

Why users of AI tools hit limits

Common reasons cited in the survey:

Being a new AI user or a power user. These are two distinct groups, but an engineering manager at a mid-sized company in Canada says that each one similarly blows through token limits for different reasons:

“We’re mindful of trying to manage costs by setting AI spend limits across the org. We have two subsets of users at odds with each other:

1. Individuals who are still learning and blow through their credits at an inordinate rate, forcing us to keep limits low.

2. Power users who hit the limit through regular use and apply pressure to raise the limit.

It’s a tough balance.”

Using Opus for all work. A few engineers mention being careful about how they use Opus because it previously ate up their token budgets. Here’s a software engineer at a mid-sized company in Europe:

“I made the mistake of using Opus in the past and burning through budgets quickly. Now, my routine is to start in ‘plan’ mode with Opus. I then paste the acceptance criteria and description of the issue and let the plan mode figure it out. I then switch to Composer or Sonnet and have the agent take over from there.”

Mistakes that eat up tokens are easy to make. These include starting on work or a problem from the wrong end, using AI directly for a task rather than opting for a simple script, trying some new tool or technique that ends up consuming tokens (OpenClaw and Ralph Loops are cited), and others.

What happens when the limit is hit?

Hitting the limit with an AI tool is inconvenient and happens to many developers, who take a variety of next steps:

Switch the model or tool. Around a quarter of respondents who hit limits mentioned switching. From a software engineer at working at Atlassian:

“In my company, for Cursor and Windsurf we have monthly limits. Our internal coding tool (called codelassian) also has daily prompt and hourly token limits. When I hit a limit in one tool, I switch to the other.”

Upgrade to a pricier plan. When it’s an option, this is a no-brainer at most places, especially as the alternative would be devs twiddling their thumbs waiting for the limit to be reset. A senior engineering manager at a mid-sized company says:

“In my team, we are regularly hitting session limits with Claude. We upgraded some teammates to the Max 20x plan – and on this plan we have not been hitting limits, so far.”

Adopt API-based pricing. This is the easiest way to keep working without abandoning a task you’re knees-deep in. A senior engineer at a large company says:

“The company provides both the Claude and Copilot corporate offerings. When the limits are reached, I tend to use API keys that my teammates give me.”

3. Impact on “builders”

We identified three different types of professional in the survey:

• Builders: those who care about quality, good architecture, following good coding practices, and who talk about the craft of software engineering, etc.

• Shippers: those who primarily focus on outcomes for a product, features, testing, and experimenting with users. A fair number of leaders, managers, and engineers who were more hands-off with coding before AI tools are in this category, as are product engineers.

• Coasters: engineers who are not considered particularly good or great engineers, but they get the work done. They often do this without much taste or concern for quality, and seem to be mostly coasting along and doing what they’re told.

The overall consensus in our survey results is that AI will amplify and multiply tendencies and patterns that existed before, and the impact of the tools varies accordingly among users. Let’s start with the impact we’ve observed upon builders in the responses:

Builders say they get value from AI tools in the following areas:

Larger code changes. Builders generally find AI helpful for work like:

• Refactoring

• Migrations

• Improving test coverage

• Carrying out large codebase changes

All these are changes that are laborious, but not very challenging technically. They also require experience in knowing what you want to do and how to do it.

Accomplishing “quality of life” tasks. Builders mention that with AI tools, they get to fix and improve things like nagging bugs that otherwise wouldn’t be “worth it” in time invested, but the barrier to entry is lower with AI.

A good example of this is in last week’s podcast with David Heinemeier Hansson (DHH), the creator of Ruby on Rails, in which he revealed how one of their engineers optimized P1 – the fastest 1% of web requests:

“One of our most agent accelerated people asked: “What about P1? What about the floor? Can we fix the floor?” He found that the floor [of request speed] was 4 milliseconds.

Well, 4 milliseconds can add up if you have a bunch of fast requests. So, he just said: “We’re going to optimize P1. The fastest 1% of requests, we’re going to make them even faster.” He took it from 4 milliseconds to less than half a millisecond. He did this P1 project over a couple of days, like a side project.

He had an intuition that there was something here. He let agents run with it. The work ended up being 12 pull requests, and about 2,500 lines of code changed.

This is exactly why the explosion of the pie suddenly lets us look at problems we would never have contemplated looking at before.”

Typing is no longer a bottleneck. Some builders report falling even more in love with coding with the help of AI and agents, since physically typing out code is no longer a bottleneck for them. They enjoy being able to prompt. From one “builder”:

“For someone who loves to build – but also values code quality, performance, reliability, and security – I ship a lot more quality code faster, if for no other reason than because the AI can read and write 100x faster than me. I get to stay at the conceptual level of shipping a product, and I can dive into debugging with the agent as needed. But if the agent has a good handle on the situation I can give it as much of the tedious parts as I wish.” – Staff Engineer, at a large tech company, US

The negative sides of AI tools, as experienced by builders:

• More AI slop. Builders seem to be the most overwhelmed and derailed by reviewing a lot more AI-generated code. They can get frustrated with low-quality code shipped by colleagues which could be categorized as “AI slop.”

• More debugging. AI-generated code introduces bugs and issues, and builders tend to spend the most time debugging and fixing those issues.

• Identity loss. Some builder-types report a sense of identity loss and even some grief. Much of this relates to no longer doing hands-on coding because they cannot justify it, since AI agents generate pretty decent code faster than someone can type it.

4. AI tools speed up “Shippers”

The “shipper” archetype thrives when they get things to production quickly. This group is by far the most enthusiastic about AI tools in survey responses. They are also the ones who praise – or hype up – the tools because of their personal experiences of shipping much faster with them.

The biggest upsides mentioned by shippers:

Read more

Listen and watch now on YouTube, Spotify, and Apple. See the episode transcript at the top of this page, and timestamps for the episode at the bottom.

Brought to You by

• Statsig – ⁠ The unified platform for flags, analytics, experiments, and more. Stop switching between different tools, and have them all in one place.

• WorkOS – The infrastructure B2B and AI-native companies use to sell to enterprise. It covers everything enterprise security requires: SSO, SCIM, RBAC, Audit Logs, AI governance, and more. Engineering teams ship it in days. Trusted by 2,000+ fast-growing companies, including OpenAI, Anthropic, Cursor, and Vercel. WorkOS.com

• Sonar – The makers of SonarQube, the industry standard for automated code review. Sonar helps reduce outages, improve security, and lower risks associated with AI and agentic coding. See how SonarQube Advanced Security is empowering the Agent Centric Development Cycle (AC/DC) with new capabilities.

In this episode

David Heinemeier Hansson (DHH) is the creator of Ruby on Rails and Omarchy, co-founder and CTO of 37signals (maker of Basecamp and HEY), and the author of several books including the best-seller, Remote: Office Not Required, co-written with Jason Fried.

Six months ago, in an episode of the Lex Fridman podcast, David shared how he doesn’t use AI tools to write code: he types out all his code. But things have changed a lot since then.

In this episode, we discuss his approach to building software, how it’s changed in the last six months, and why he now takes an agent-first approach, and how he barely writes any code by hand. We go into how he uses AI agents: which alter how he builds and explores ideas, but also how his standards of quality and craft remain the same.

We also discuss how 37signals thinks about product development, from the role of designers to the importance of aesthetics and taste. David gets into how he sees beauty and functionality as closely linked, and why strong opinions about design lead to better software.

Finally, we look into the uneven impact of AI which amplifies senior engineers while creating challenges for junior developers, and what this may mean for the role of the software engineer.

Key observations from DHH

Here are 12 of my most interesting takeaways from talking with DHH:

1. His philosophy on AI has not changed, but the available tools very much have. Autocomplete-style coding assistants were genuinely annoying for experienced developers six months ago. Things changed with the shift from tab-completion to agent harnesses, plus the emergence of powerful models like Opus 4.5 – when agents started producing code which DHH does want to merge with little to no alteration.

2. Beautiful code and products aren’t matters of vanity; they’re signals of correctness. Dipping into philosophy, DHH says: “When something is beautiful, it’s likely to be correct.” He argues that Steve Jobs wanted the inside of a computer to be beautiful because people who care about circuit board layout are also those who sweat on the details of the UI.

3. DHH’s development workflow, today: he runs tmux to have two models running, and neovim in the center. Specifics:

• One fast LLM running (typically Gemini 2.5) in one split terminal

• A slow but more powerful model in another terminal (usually Opus)

• NeoVim for reviewing diffs via Lazygit

4. Ruby on Rails seems to be enjoying a Renaissance thanks to AI. Rails is one of the most token-efficient ways of building web apps and is well-suited for agent workflows. Testing is part of the framework, which helps agents write tests and validate their own outputs. It also produces code that humans can read and verify, which matters when reviewing agent output at speed.

5. A big win from using AI agents is tackling stuff that you wouldn’t have before. A senior engineer at 37signals ran a “P1 optimization” project to improve the fastest 1% of requests. They optimized the P1 from being at 4 milliseconds, to under half a millisecond. This is the sort of work that wouldn’t have been considered previously!

6. Running several AI agents feels less like “project management” and more like “wearing a mech suit.” Being a project manager of agents did not appeal to DHH, but now that he’s building with several agents, he feels like he’s in control of the work which is being hyper-accelerated.

7. Senior engineers benefit from AI a lot more than juniors. At 37signals, senior engineers gain more from AI tools as they can validate whether an agent’s output is production-ready. DHH also notes that Amazon reached the same conclusion, and no longer lets junior programmers ship agent-generated code to production without review.

8. 37signals has one designer for every two engineers. The company has around 20 software engineers and 10 designers. Designers do far more than design; they’re also product managers and “implementers” rolled into one. On top of making things look good, they figure out what should be built, how it should work, and often build the first version. DHH compares design at 37signals to jewelry design: “you should know the properties of gold. You should know how it bends.”

9. AI agents could turn 37signals’ “designer model” into the industry standard. AI tools now empower designers to implement more of their vision directly, and DHH suspects the rest of the industry is converging toward what 37signals has always done: working with small teams, where designers are also builders.

10. Command Line Interfaces (CLI) feel like the ultimate AI interface, which validates the Unix philosophy of the 1970s. DHH is building CLIs for all 37signals products because they let agents chain tools together. “GitHub also has a CLI, and Sentry as well,” he says. “You can tie all these things together so an agent can check errors, write a fix, post a PR, and report back to basecamp.”

11. The demise of the two-month product development cycle described in the book ‘Shape Up: Stop Running in Circles and Ship Work that Matters’. The 2019 title by Ryan Signer covered how 37signals worked at the time, and DHH reveals that this methodology now needs rewriting because AI acceleration has made that timeline feel slow.

12. Eight hours of sleep is non-negotiable – even during an AI gold rush! DHH believes the dopamine loop of shipping with agents is intoxicating and can lead to higher risk of burnout. So, he sleeps eight hours and doesn’t use an alarm.

The Pragmatic Engineer deepdives relevant for this episode

• Are AI agents actually slowing us down?

• How Claude Code is built

• The future of software engineering with AI: six predictions

• The AI Engineering Stack

• Mitchell Hashimoto’s new way of writing code

• How Linux is built with Greg Kroah-Hartman

Timestamps

(00:00) Intro

(02:11) Omarchy and Ruby on Rails

(08:25) 37signals overview

(10:12) Launching HEY

(18:38) Building HEY

(22:47) Designers at 37signals

(28:08) The craft of design

(31:52) Why DHH now embraces AI workflows

(39:45) The AI inflection point

(44:23) DHH’s agent-first workflow

(55:09) AI’s impact on junior developers

(1:03:08) Developer experience with AI

(1:16:43) What does AI mean for developers?

(1:23:33) 37signals teams and hiring

(1:38:20) Work-life balance with AI

(1:41:41) Why DHH keeps building

(1:45:24) Closing

References

Where to find DHH:

• X: https://x.com/dhh

• LinkedIn: https://www.linkedin.com/in/david-heinemeier-hansson-374b18221

• Website: https://dhh.dk

• Newsletter: https://world.hey.com/dhh

• Podcast: https://37signals.com/podcast

Mentions during the episode:

• Omarchy: https://omarchy.org

• Linux: https://www.linux.org

• Ubuntu: https://ubuntu.com

• Arch Linux: https://archlinux.org

• Hyprland: https://hypr.land

• Ruby on Rails: https://rubyonrails.org

• Basecamp: https://basecamp.com

• Fizzy: https://www.fizzy.do

• Jason Fried on X: https://x.com/jasonfried

• HEY: https://www.hey.com

• Shape Up: Stop Running in Circles and Ship Work that Matters: https://basecamp.com/shapeup

• Zoltán Hosszú applying to 37signals: https://zoltan.co/37signals

• Daring Fireball: https://daringfireball.net

• Smalltalk: https://en.wikipedia.org/wiki/Smalltalk

• DHH: Future of Programming, AI, Ruby on Rails, Productivity & Parenting | Lex Fridman Podcast #474:

• Homer’s typing Bird:

• Real-world engineering challenges: building Cursor: https://newsletter.pragmaticengineer.com/p/cursor

• Building a best-selling game with a tiny team – with Jonas Tyroller: https://newsletter.pragmaticengineer.com/p/thronefall

• Andrej Karpathy on X: https://x.com/karpathy

• Reflexive AI usage is now a baseline expectation at Shopify:

• Claude Code: https://code.claude.com

• OpenCode: https://opencode.ai

• MacBook Neo: https://www.apple.com/macbook-neo/

• tmux: https://github.com/tmux/tmux/wiki

• Kimi K2.5: https://kimik2ai.com/k2.5

• Agent first, agent native: https://basecamp.com/agents

• Sentry: https://sentry.io

• Moore’s law: https://en.wikipedia.org/wiki/Moore%27s_law

• The Bitter Lesson: http://www.incompleteideas.net/IncIdeas/BitterLesson.html

• Scaling Uber with Thuan Pham (Uber’s first CTO): https://newsletter.pragmaticengineer.com/p/scaling-uber-with-thuan-pham-ubers

• Waymo: https://waymo.com

• Elon Musk: “There will not be a steering wheel” in 20 years: https://www.axios.com/2017/12/15/elon-musk-there-will-not-be-a-steering-wheel-in-20-years-1513304216

• Leopold Aschenbrenner — 2027 AGI, China/US super-intelligence race, & the return of history:

Dwarkesh Podcast

Leopold Aschenbrenner — 2027 AGI, China/US super-intelligence race, & the return of history

Listen now

2 years ago · 89 likes · 16 comments · Dwarkesh Patel

• Terminator 2 Things & Ideas we would have never thought:

• Commodore 64: https://en.wikipedia.org/wiki/Commodore_64

• PlayStation: https://www.playstation.com

• Jevons paradox: https://en.wikipedia.org/wiki/Jevons_paradox

• OpenClaw: https://openclaw.ai

• The creator of Clawd: “I ship code I don’t read”: https://newsletter.pragmaticengineer.com/p/the-creator-of-clawd-i-ship-code

• John Carmack on X: https://x.com/ID_AA_Carmack

• TDD, AI agents and coding with Kent Beck: https://newsletter.pragmaticengineer.com/p/tdd-ai-agents-and-coding-with-kent

• Extreme Programming Explained: Embrace Change: https://www.amazon.com/Extreme-Programming-Explained-Embrace-Change/dp/0321278658

• Smalltalk Best Practice Patterns: https://www.amazon.com/Smalltalk-Best-Practice-Patterns-Kent/dp/013476904X

• From IDEs to AI Agents with Steve Yegge: https://newsletter.pragmaticengineer.com/p/from-ides-to-ai-agents-with-steve

—

Production and marketing by Pen Name.

Martin and Kent go back decades, and Martin jokes that his career is “mostly about writing down Kent Beck’s ideas.” They first collaborated in the 1990s, and each has published influential books – ‘Extreme Programming Explained’ and ‘Test-Driven Development’ by Kent, and ‘Refactoring’ and ‘Patterns of Enterprise Application Architecture’ by Martin.

At the Pragmatic Summit, they each shared a wealth of hard-earned learnings and decades-worth of perspective, along with a healthy dose of skepticism. Needless to say, the conversation did not disappoint, and this article summarizes what we discussed in their own words. You can also check out the full recording.

We cover:

1. Technology shifts similar to AI. The arrival of the microprocessor, introduction of object-oriented languages, the Internet, and agile software development principles were all major changes – but one big difference was that it took time for these technologies to be adopted. Not so with AI.

2. Agile and AI similarities. With Agile, company incentives were often misaligned, “snake oil” vendors were everywhere, and a “mid pack” of developers who resisted the change saw their career prospects hit. These trends look likely to repeat with AI.

3. What’s happening inside companies. There’s some confusion – and even panic – at large companies, while AI tools don’t work nearly as well on large and complex codebases as on greenfield projects. Also, a “re-soloing” of software development is inbound.

4. Avoiding burnout with AI agents. Set and maintain boundaries, and pay attention. Martin suggests to catch when you start producing “negative value”: that’s when to take a break.

5. Unhealthy performance metrics. Companies are starting to measure things like frequency of pull requests – when they should be looking to quantify outcomes and results.

6. Lower quality on purpose? It seems every business is optimizing for speed with AI, but quality can get dropped. Also: building features is more obvious with AI, than investing in “futures.”

7. Test-Driven Development (TDD): tests no longer optional? Kent pioneered TDD, and today it’s more relevant than ever for working with AI.

8. Thriving in an AI-native industry. Focus on working with agents to express your craft, try to get more enjoyment in understanding your domain, and take on more ambitious work.

Before we start, a programming note: this week, there will be no The Pulse on Thursday — I’ll be attending AI Engineer Europe in London on Thursday and Friday, including doing a fireside chat, and hosting one with Linear CTO Tuomas Artman.

1. Technology shifts similar to AI

Do you recall a tech change as similarly promising and unpredictable as AI?

Martin: “Nothing has hit with the magnitude of AI. This is a whole size different from anything we’ve faced before. On a smaller scale, we were very much involved in the growth of object oriented languages, which scared a lot of people. It didn’t scare us so much because we were part of it.

Looking back, the internet had a huge impact on us all, and of course, Agile software development, too. Agile had a very big impact on a lot of organizations: you could tell by how hard they resisted it. We had to persuade people of the importance of these technological changes; yes, even the internet! It may sound surprising but there were people who didn’t think it was important.

The thing about AI is that today there is no argument about how important it is.”

Kent: “The other analogy I have is the introduction of the microprocessor. Before that, computers were big boxes; you couldn’t move them around. If you wanted another computer, you’d mortgage your house for it. Having a computer was a big deal.

I was a kid in Silicon Valley with my dad as a programmer when the Intel 4004 hit the market [in 1971]. We went: “Wait a minute, that chip is a computer? Oh my goodness!” The possibilities of computing suddenly expanded thanks to it. If you could figure out how to write software on this chip and figure out how to design hardware around this thing, you could suddenly do things you hadn’t even imagined.

And so I think part of AI is this expansion of imagination. I’m writing projects that are ridiculously ambitious: I’m working on a persistent Smalltalk. I’m writing library-quality code for Rust.”

Balancing skepticism and curiosity

What was the feeling in the industry during those revolutions, and the differences between professionals who thrived back then and those who didn’t?

Martin: “There was a mix of people chasing the hype and those saying, “this new thing is nothing special.” I think you’ve always got to have that balance of skepticism and curiosity, and to be selective about it. I mean, I have been completely skeptical about some big changes: Blockchain was one I was extremely skeptical about.

My skepticism is well-rooted because I’ve seen so much “snake oil” over the years. In fact, my skepticism has to be absolute and total, which means I have to be skeptical about my skepticism! To be that skeptical also requires curiosity: you’ve got to be curious enough to say “how do I probe in order to detect signs of something useful?”

You also need to be aware that your early interactions may not actually be a true signal. When I started playing around with AI, it was with GitHub Copilot a year and a half ago. I was pretty unimpressed; it would give you something wonderful, but most of the time it gave you such garbage that you would just delete it right away. If that had been my only impression of AI, I would’ve immediately flipped the “bozo bit” on it, like I did with blockchain.”

Kent: “Here’s the thing, the capabilities of AI can change week to week. I’ll try something with Gemini one week and it fails miserably. Then Claude Code works pretty well, and then it doesn’t. And then I try Gemini for the same thing and it works, when it hadn’t worked last week!

People want an answer, but the answer’s always changing. In this environment, you can’t possibly have the answer. That’s the bad news, but the good news is that nobody else has the answer either. So, you’re just as smart as everybody else because we’re all equally ignorant.”

2. Agile and AI similarities

In 2001, the ‘Agile Manifesto’ came out, of which you were both co-authors. I think many companies are expecting the same thing with AI as Agile promised: better, faster, cheaper software. But how did Agile adoption really play out?

Kent: “It turns out people don’t want faster, cheaper, better! Inside some companies, the incentives are misaligned with actually achieving that. And so as geeks trying to achieve these improvements and saying: “it’s 40% better, 12% cheaper and less fattening,” people will punish you if that doesn’t align with their incentives inside organizations.

In the ideal organization, everybody would care about the same things, but that’s just not the way it works! So, if AI is coming along to promise the same things, we’re going to see the same reaction as before.”

Martin: “An obvious difference is the sheer magnitude and speed there is with AI. Also, I think there will be a big difference between people who use it well and people who use it badly. The trick is figuring out how to use it well and putting the effort in to learn. There will be a big distinction between those two groups.

But I suspect there will still be some similarities with Agile. The core notions behind Agile and extreme programming are solid and good, but a huge snake-oil industry appeared around it – the “Agile industrial complex”, as I refer to it. This is also happening with AI right now, and it’s often hard to see the difference between snake oil and the real stuff.”

AI as an amplifier

Kent: “AI is an amplifier. If you’re young and learning quickly, AI can amplify your learning. I personally think this is the golden age of the junior programmer. I get people coming to me all the time saying things like “my son started his second year in CS and wants to go into something more commercial like art history.” And I’d say, “this is like if you’re a carpenter and they just introduced the circular saw and you think, ‘oh, well, carpentry is over. Anybody can build a house now.’ Well, no! Now, you have more powerful tools. You have less of the crummy work to do.

I think that young people are going to learn faster, and experienced folks who are working effectively are going to work quicker and more effectively.”

Developers stuck in the middle

Kent: My concern is that there’s a “middle” of people who got into programming as a way to make money. If we look back at the Dotcom crash, there was a “mid pack” of such people who ended up going into real estate, more or less. But today, I don’t know where that “middle” will go, and it’s also much bigger now than 25 years ago.”

Martin: “But that middle has also been “flushed out” to some degree by retrenchment in the software industry at the end of the zero interest rate period. So, that’s an interesting difference because we’ve had these things occurring at once: the AI boom, and the economic headwinds of the past 2-3 years.

This is an interesting mix that wasn’t present in the ‘90s with the Dotcom Boom. Back then, it was pretty much all a solid boom.”

Return of “let’s get rid of programmers!”

Kent: “Another interesting confluence of factors is the periodic, “we can get rid of all the programmers, woo-hoo” trend, which started with Cobol in the 1970s. With Cobol, business analysts were supposedly going to be able to write the programs, and the logic was that we wouldn’t need programmers anymore. That comes back repeatedly.

Agile, however, was definitely not a “let’s get rid of programmers” trend. With Agile, we wanted programmers to be more effective in their jobs. And since we started it, and were programmers, we were able to push that agenda pretty effectively.

However, today the “get rid of programmers” trend is repeating. As programmers, it behooves us to think about why they keep wanting to get rid of us. Some of that’s about us as programmers, and some of it not. Still, we should think about why people periodically want to axe us. In the end, this trend amps up the fear factor that everybody’s experiencing.”

“Re-soloing” of programming

Kent: “A big trend is the “re-soloing” [reduced in-person collaboration] of programming.

A big part of extreme programming (XP) was creating a safe social environment for basically antisocial people. On an XP team, people are talking to each other for hours a day, and are happy to do so because it’s set up to be a positive experience.

Now, I see programmers saying, “I’ve got six agents, so really I’m managing a team.” No, you’re not: you’re using six tools at once, which is fine, but it’s very different from having a conversation with somebody who sees things slightly differently, or has a different energy level from you on the day.

We used to have programmers in individual offices with doors, and you’d shut the door and slide the pizza underneath. That was easy to manage, but then along came this messy, social, complicated, chaotic process of software development, which just happened to produce really good results.

But now, instead of 50 people on my team, I can have five and they don’t have to talk to each other, and each can have 10 agents. Is that the same? No, it’s not.”

More effective two-pizza teams & the future of pairing

Martin: “Are we seeing two-pizza teams [of 5-10 people] becoming one-pizza teams because agents don’t eat pizza, or do we see two-pizza teams staying and becoming much more effective and capable? My bet is on more effective two-pizza teams.

We’re beginning to see some interesting feedback in terms of pair programming. With pair programming, is it one human and the genie (AI) programming, or is it two humans and one genie? If it’s two of us, perhaps we can control the genie a bit better, and we also have interaction.

I’ll be very interested in reports of people trying to control genies in pairs, possibly even beyond pairs. There’s also the whole ‘mob programming’ thing, and how that will go with genies. I don’t necessarily think that one person and many genies is the right answer.”

Kent: “My experience of pairing with two humans, plus one or more genies, has been very positive. And the fact the AI is slow is really nice. Every time models come out and are faster, I’m like, “Oh, there’s less time to talk.” When the AI goes away for three minutes, we can talk about our philosophy of naming, or how we express conditionals, or about what we should be doing next. But if it pops back in 15 seconds, you don’t have time for that conversation.”

4. Avoiding burnout with AI agents

Do you find yourself getting close to burnout, especially when spinning up multiple threads? Do you have strategies for managing the mental impact?

Read more

Today, we cover:

1. Founders back coding with AI: Mark Zuckerberg & Garry Tan. The Meta chief is shipping diffs after 20 years, while Garry Tan at Y Combinator is knee-deep in coding, 15 years later. Founders with technical backgrounds being hands-on with AI agents could be a good thing – especially when the “honeymoon” period ends.

2. A bad week for Claude Code and GitHub. Claude Code’s source code was leaked when a sourcemap file was accidentally uploaded, and revealed that the tool uses anti-distillation to deal with competitors, and also some potential future features such as an always-on background agent. Also: DMCA copyright strikes from Anthropic raise a big question: can a codebase that is fully AI-generated be covered by copyright?

3. Industry pulse. Meta sets targets for AI-generated code, GitHub’s 6 years of reliability issues, massive job losses at Oracle, GitHub Copilot rollouts and then rolls back ads, RAM prices fall (for now), and more.

1. Founders back coding with AI: Mark Zuckerberg & Garry Tan

Two interesting stories of AI tools encouraging busy founders to start writing code again with AI agents.

Mark Zuckerberg back to landing diffs, 20 years later

Read more

Listen and watch now on YouTube, Spotify, and Apple. See the episode transcript at the top of this page, and timestamps for the episode at the bottom.

Brought to You by

• Statsig – ⁠ The unified platform for flags, analytics, experiments, and more.Stop switching between different tools, and have them all in one place.

• WorkOS – Everything you need to make your app enterprise ready. WorkOS gives you APIs to ship enterprise features in days: features like authentication, SSO, SCIM, RBAC, audit logs. Visit WorkOS.com

• Sonar – The makers of SonarQube, the industry standard for automated code review. Sonar helps reduce outages, improve security, and lower risks associated with AI and agentic coding. See how SonarQube Advanced Security is empowering the Agent Centric Development Cycle (AC/DC) with new capabilities like malicious package detection to provide the same rigorous guardrails for AI agents as you would for a human developer.

In this episode

Thuan Pham was Uber’s first and longest-serving CTO, and today he’s the CTO of Faire, a B2B wholesale platform. Back when Thuan joined Uber, it had around 40 engineers and 30,000 rides per day, and the system crashed multiple times a week. Over seven years, he helped rebuild the system, move it from a monolith to microservices, and scaled the engineering organization behind it. I had the privilege of working with Thuan for four of those seven years. Later, the very first issue of The Pragmatic Engineer newsletter was a deepdive into Uber’s Program and Platform split. This episode of the podcast contains a nice “full circle” moment, where Thuan shares even more details about why Uber chose to embrace that structure.

We discuss what it takes to operate and build in that kind of environment. Thuan explains how he divided his time at Uber into three “tours of duty,” from stabilizing a fragile system, to re-architecting it, and scaling the org.

We go deep into the platform-and-program split, the Helix app rewrite, and what it took to launch Uber in China in just five months (the original estimate was 18 months). We also cover Uber’s in-house tools and explain why they were necessary to support rapid growth.

Finally, we discuss his role today as CTO of Faire, how the company is using AI, and how he sees AI changing software engineering.

Key observation from Thuan

14 takeaways from Thuan that I find the most interesting:

1. Your professional reputation is a compounding asset that pays off unpredictably. Bill Gurley recruited Thuan to Uber based on knowing him from a startup a decade earlier. Similarly, when Thuan needed to hire for critical infrastructure teams at Uber, he reached out to engineers at VMware whom he’d previously worked with, and they followed him to the ridesharing app because they trusted him.

2. The program/platform split came before microservices. The concept of cross-functional “program” teams and dedicated “platform” teams became necessary because an org split across backend, frontend and mobile engineers slowed down in execution speed when Uber grew to around 100 engineers. Every feature required negotiating bandwidth across the mobile, backend, and dispatch teams. Thuan, Travis Kalanick, and Jeff Holden literally used color-coded sticky notes with people’s names to reorganize into self-sufficient teams. We cover more about this split in the deepdive, The Platform and Program split at Uber.

3. Microservices at Uber were more about surviving hypergrowth than anything else. Uber needed to decompose its massive monolith called “API.” To do so, a simple rule was applied: anything new needed to be built outside of the monolith so that no team blocked another. Teams started to build microservices, but decomposing the monolith took a good two years. Fun fact: in 2026, Uber has somewhat fewer microservices (around 4,500) than back in 2016 (around 5,000).

4. When retiring a monolith, sometimes it gets even bigger before shrinking. After Uber decided to pull services out of the massive monolith, it still kept growing because the business kept adding features! There was an ugly middle phase before the monolith started to shrink. Keep this in mind if you look into decomposing a monolith.

5. Expect multiple rewrites during hypergrowth. The right architecture depends on how fast a product and company are growing. At Uber, repeated rewrites were common because each one “bought” another window of survival for the company. Thuan’s recommendation is to understand that a rewrite simply means a company is outrunning its existing architecture: this is not necessarily a bad thing!

6. Controversial launch advice: start with the hardest launch first. When Uber rolled out in China, Travis insisted on starting with Chengdu, the largest launch city. Looking back, it was scary but also helpful, as launching in the “hardest” city first gave the team confidence and made subsequent city launches much easier.

7. Travis Kalanick spent 30+ hours interviewing Thuan. This took place over two weeks, as a series of one-on-ones. The sessions became a simulation of working together: disagreeing, aligning, and working things out. I’ve yet to hear of such an intense – and technical! – recruitment process by another CEO.

8. Uber is the only major company that had a “Senior 1” and “Senior 2” level – and Thuan is unapologetic. Thuan introduced the Senior 1 (L5A) and Senior 2 (L5B) levels because the jump from senior (L5) to Staff (L6) became very big, and larger than between previous levels. One problem this split level created was that Uber’s L5B was akin to Google’s and Facebook’s L6/E6. Thuan resisted the title inflation of just renaming L5B to ‘Staff’.

9. Name your services clearly; you don’t work at a “Mickey Mouse shop.” As Uber grew more complex, whimsical service names (like “Mustafa”) made navigating systems more tricky, and onboarding for new joiners more painful. Thuan sent a company-wide email which called for professional-sounding naming conventions, and reminded everyone that Uber was not a “Mickey Mouse shop.” The email didn’t fully solve the issue, but did force the growing org to take itself more seriously.

10. Great engineering talent is global, so bring the opportunity to developers. During Thuan’s time, Uber opened nine engineering offices worldwide in order to access world-class talent. For example, the relatively small Denmark office built and operated core parts of Uber’s infrastructure, such as the trip datastore, Schemaless.

11. What’s the most important part of a CTO’s job? Thuan thinks that it’s to build a high-talent-density team, and to “see around the corner” 18–24 months in advance. As he puts it: “your team handles the six-month problems, while you figure out what the organization needs to look like two years from now.”

12. The hardest use case of AI in software engineering is building new features on legacy codebases. At Faire, Thuan’s team uses “swarm coding” (orchestrated AI agents working in parallel) and some engineers there have doubled their output in three months. But generating greenfield code is easy; the real challenge is dealing with millions of lines of code and building features on top with all those existing dependencies.

13. AI raises the floor, but doesn’t change what makes engineers great. AI enables people who can’t code to produce decent apps, but great engineers are still finding ways to leverage AI tools and accelerate even more. The differentiators remain the same as before AI: curiosity, fearlessness, and a willingness to innovate and learn new things.

14. Thuan’s career advice: think of it in phases. Each segment of your career has different priorities, which Thuan sees like this:

• First 5–10 years: seek maximum learning and push yourself hard.

• Mid-career as a senior/staff engineer: seek roles to make an outsized impact in, perhaps at a smaller company.

• In leadership roles: teach and coach others, and bring them along with you.

The Pragmatic Engineer deepdives relevant for this episode

• How Uber uses AI for development: inside look

• The Platform and Program split at Uber

• How Uber is measuring engineering productivity

• Inside Uber’s move to the cloud

• Uber’s crazy YOLO app rewrite, from the front seat

• How Uber built its observability platform

• Developer experience at Uber with Gautam Korlam

• Uber’s engineering level change

Timestamps

(00:00) Intro

(05:32) Getting into tech

(16:09) The dot-com bust

(20:42) VMware

(26:29) Getting hired by Travis at Uber

(33:22) Early days at Uber and scaling challenges

(40:57) Uber’s China launch

(47:12) The platform and program split

(50:26) From monolith to microservices

(53:38) Internal tools at Uber

(57:05) Helix: Uber’s mobile app rewrite

(59:55) Thuan’s email about naming

(1:02:03) Org structure changes under

(1:06:34) Thuan’s work philosophy

(1:12:23) The “three tours of duty” at Uber

(1:15:37) Why Thuan left Uber

(1:17:34) Coupang and Nubank

(1:21:59) Faire

(1:25:31) How Faire uses AI

(1:28:24) AI’s impact on software engineering

(1:31:09) The role of the CTO

(1:35:13) Career advice

References

Where to find Thuan Pham:

• LinkedIn: https://www.linkedin.com/in/thuanqpham

Mentions during the episode:

• HP Labs: https://www.hp.com/hk-en/shop/tech-takes/post/what-is-hp-labs

• Silicon Graphics: https://en.wikipedia.org/wiki/Silicon_Graphics

• Miro: https://miro.com

• VMware: https://www.vmware.com

• Bill Gurley on LinkedIn: https://www.linkedin.com/in/billgurley

• Travis Kalanick on X: https://x.com/travisk

• DiDi: https://web.didiglobal.com

• The Platform and Program Split at Uber: A Milestone Special: https://newsletter.pragmaticengineer.com/p/the-platform-and-program-split-at

• Rewriting Uber Engineering: The Opportunities Microservices Provide: https://www.uber.com/blog/building-tincup-microservice-implementation

• Up: Portable Microservices Ready for the Cloud: https://www.uber.com/blog/up-portable-microservices-ready-for-the-cloud

• How Uber Built its Observability Platform: https://newsletter.pragmaticengineer.com/p/how-uber-built-its-observability-platform

• The Uber Engineering Tech Stack, Part I: The Foundation: https://www.uber.com/blog/tech-stack-part-one-foundation

• How Ringpop from Uber Engineering Helps Distribute Your Application: https://www.uber.com/blog/ringpop-open-source-nodejs-library

• PostgreSQL: https://en.wikipedia.org/wiki/PostgreSQL

• MySQL: https://www.mysql.com

• Uber’s Crazy YOLO App Rewrite, From the Front Seat: https://blog.pragmaticengineer.com/uber-app-rewrite-yolo

• Hypergrowth startups: Uber and CloudKitchens with Charles-Axel Dein: https://newsletter.pragmaticengineer.com/p/high-growth-startups-uber-and-cloudkitchens

• Coupang: https://www.aboutcoupang.com

• Nubank: https://international.nubank.com.br

• Max Rhodes on LinkedIn: https://www.linkedin.com/in/max-rhodes

• Sequoia: https://sequoiacap.com

• Wyan Gretzky’s quote: https://www.brainyquote.com/quotes/wayne_gretzky_383282

—

Production and marketing by Pen Name.

And now, inference engineering is becoming more widespread, too, as open LLM models grow more capable. This is because with closed models, inference engineering is done only by the AI engineers who build the model, whose number might add up to a few thousand globally. In contrast, with the open models which tech companies are adopting, it’s possible to tweak them to perform better at inference. For example, Cursor built its new Composer 2.0 model on top of the open Kimi 2.5 model, and successfully used plenty of inference engineering approaches to make it even faster.

So, based on this industry-wide prevalence and the related need for superior technical performance, it’s worth understanding a bit about what inference engineering actually is, and some interesting approaches worth knowing about, as a software engineer.

For some answers, I turned to Philip Kiely, a software engineer who has been working for four years at the inference startup, Baseten. With his hard-earned experience, Philip has written an excellent, in-depth book about precisely this topic, “Inference Engineering.”

In today’s issue, we cover:

1. Setting the stage: why is inference so important? More capable, widespread, open models are driving demand for inference engineering.

2. What is inference? As the phase that comes after training a model, the inference layer introduces new engineering challenges like batching, caching, and quantization.

3. When is inference engineering needed? Investing in this area is typically worth it when your product and usage scales up, and there are product requirements which a current, off-the-shelf solution lacks.

4. What hardware does inference use? Datacenter GPUs are the most common, while on-premises, air-gapped GPUs are also employed.

5. What software does inference use? Commonly-used software includes NVIDIA’s CUDA and Dynamo, as well as hardware-agnostic projects like PyTorch, vLLM, and others, which are growing in popularity.

6. What infrastructure does inference need? Autoscaling is a baseline requirement. Kubernetes is a popular choice for autoscaling inside a cluster, while multi-cloud inference might be necessary for high-scale use cases.

7. Five approaches to make inference faster. Quantization (reducing the numerical precision of a model’s weights), speculative decoding (taking advantage of spare compute to generate “draft tokens”), caching, parallelism (tensor parallelism and expert parallelism) and disaggregation (separating the prefill and decode phases to run on separate workers, not the same GPU).

This deepdive uses a few abbreviations and concepts that are everyday lingo for inference engineers, but maybe are not for those less versed in the domain:

• CUDA: Compute Unified Device Architecture. NVIDIA’s proprietary API to program NVIDIA GPUs for high-performance computing, including LLM-related use cases.

• TTFT: time to first token. Think of this as the “time to process the prompt.” This metric determines the perceived responsiveness of models and GenAI systems.

• TPS: tokens per second. Akin to a model’s “typing speed.”

• ITL: intertoken latency. The time between generating one token and the next.

• KV cache: key-value cache. The cached results of the attention algorithm, reused between requests to speed up inference. We cover more on KV cache in the Scaling ChatGPT deepdive.

• Prefill / decode: the two phases of inference. Prefill is when the model takes the full input and processes tokens, outputting the KV cache. Decode is the phase in which the model generates one token at a time.

• MoE: Mixture of Experts. An architecture that enables models to be pretrained with far less compute. More details on this approach.

Below is an introduction to inference adapted from Philip’s book, “Inference Engineering,” which is free to download as an e-book. Physical copies are currently sold out, but Philip is printing more as fast as possible.

My usual disclaimer: as with all my recommendations, I was not paid to mention this book, and no links in this article are affiliates. See my ethics statement for more.

With that, it’s over to Philip:

1. Setting the stage: why is inference so important?

Inference is the most valuable category in the AI industry, but inference engineering, on the other hand, is still in its infancy. In their work, inference engineers work across the stack from CUDA to Kubernetes in pursuit of faster, less expensive, and more reliable serving of generative AI models in production.

When ChatGPT launched in late 2022, there were perhaps a few hundred inference engineers in the world, and they didn’t call themselves that. These specialists mostly worked at frontier labs like OpenAI, Midjourney, and Anthropic, or at big tech companies like Google and NVIDIA.

Back then, it looked like this might be the way of the AI industry: that training generative AI models would be so hard and expensive that only a handful of companies would develop closed models and thereby require inference engineering for production serving. In that alternate future, the rest of the world would be mere consumers of AI via APIs, renting intelligence a token at a time.

Three years later, it turns out that training generative AI models is indeed both hard and expensive – but it’s not so hard and expensive to be limited to a handful of players. Instead, a proliferation of open models – more than two million and counting on Hugging Face (the “GitHub for AI”) – means that today every engineer can now deploy their own intelligence to power AI products.

Research labs around the world, from OpenAI and NVIDIA Nemotron in America, to Mistral AI and Black Forest Labs in Europe, to Alibaba Qwen, DeepSeek AI, Z AI, and Moonshot AI in China, regularly release open models of all modalities.

Despite closed models getting smarter and cheaper, the movement into open models is accelerating, which differ by the availability of their weights:

• Closed model: A proprietary model whose weights are unavailable to the public, like GPT-5 and Claude Sonnet.

• Open model: A model whose weights are publicly available, like Llama or DeepSeek, and which is usually released under the MIT license, or a similar permissive license (some models restrict commercial use, so always double-check license terms).

Before December 2024, there was a meaningful gap in intelligence between closed and open models, but when DeepSeek V3 and R1 were released, that gap disappeared. Note from Gergely: we previously covered how DeepSeek’s release rocked the AI industry.

Today, new closed models are matched by open models within months if not weeks, and occasionally, open models like Kimi K2 Thinking even exceed closed models’ capabilities for brief periods.

Despite the fact that open models are constantly chasing closed models on benchmarks, they nonetheless change the equation for AI product builders. And as both types improve, closed and open models cross capability thresholds.

In 2022, it was impossible to build the kinds of AI-native products that define the industry today. But over time, closed models got smarter and new categories like customer service voice agents and AI-powered IDEs became possible. The early models were slow, expensive, and unreliable, but the capabilities were there and AI engineers began building companies around them.

As open models crossed the same capability thresholds, these folks began using them to replace closed models. Many also began fine-tuning open models to cross capability thresholds faster, and even exceed closed model quality in their specific product and domain.

Switching to open models means the opportunity to use inference engineering to make the models powering AI products better in new ways:

• Latency: Closed model APIs are built for throughput, but open models can be optimized for real-time applications.

• Availability: While APIs for GPT and Claude are stuck at two nines of uptime, it’s possible to achieve four nines or better with dedicated deployments of open models.

• Cost: Open models are often at least 80 percent less expensive at scale.

So, whereas three years ago it looked like inference engineering was a niche field, the fact is that today, every company aiming to build truly differentiated and competitive AI products needs an inference strategy.

AI-native startups like Cursor, Clay, Gamma, and Mercor are redefining hypergrowth by building products that rely on open and in-house models. Leading digital native companies like Notion and Superhuman succeed by deeply integrating AI capabilities into their category-defining products.

Elsewhere, a new generation of blended research and engineering teams – World Labs, Writer, Mirage, and dozens more – are building businesses by training and productizing their own foundation models.

Adoption is even strong in enterprise and regulated industries, which historically were slow to adopt new technologies. Companies like OpenEvidence, Abridge, and Ambience are making generative AI ubiquitous in healthcare, while at the world’s largest companies, AI initiatives are moving past the pilot stage into massive user adoption. Market-wide demand for inference means that everyone from developers to executives has the opportunity to learn inference engineering and use it to advance their career and business.

I’ve been incredibly fortunate to have a front-row seat in the fastest-moving market in history over the last four years at Baseten, where we power mission-critical inference for the best AI products, including every company listed in the previous paragraphs.

The good news is that you are early. There are still relatively few professionals working on inference, and newcomers can become experts quickly. Also, the potential and impact of inference is becoming ever clearer, but the domain is still in its infancy. That means there are enormous opportunities to solve novel, interesting, and deeply technical problems at all levels of the stack.

2. What is inference?

Inference is the second phase of a generative AI model’s lifecycle:

• Training: The process of learning model weights from data.

• Inference: Serving generative AI models in production.

During the past decade’s machine learning (ML) boom, hundreds of thousands of data scientists and ML engineers became familiar with the full lifecycle of training and inference for ML models.

Inference for classic ML models is relatively straightforward. In the early days of Baseten, we ran inference for models built with tools like XGBoost on lightweight CPUs with a simple software stack.

In contrast, inference for generative AI models is complex. You can’t simply take model weights, get some GPUs, and expect inference to be fast and reliable enough for large-scale production use. Doing inference well requires three layers:

• Runtime: Optimizing the performance of a single model on a single GPU-backed instance.

• Infrastructure: Scaling across clusters, regions, and clouds without creating silos, while maintaining excellent uptime.

• Tooling: Providing engineers working on inference with the right level of abstraction to balance control with productivity.

These three layers must work together to create a system that can handle mission-critical inference at scale.

The runtime layer is responsible for ensuring an individual model running on a GPU (or across several GPUs in a single instance) runs as performantly and efficiently as possible. This layer depends on a sophisticated software stack, from CUDA, to PyTorch, to inference engines like vLLM, SGLang, and TensorRT-LLM. Low-level optimization is important, with kernels like FlashAttention delivering significant performance gains.

The runtime layer relies on a number of model performance techniques that apply new research to the challenges of inference on generative AI models:

• Batching: Run incoming requests in parallel, weaving them together on a token-by-token basis to increase throughput.

• Caching: Reuse the KV cache – the cached results of the attention algorithm – between requests that share prefixes.

• Quantization: Lower the precision of select pieces of the model to access more compute and reduce memory burden.

• Speculation: Generate and validate draft tokens to produce more than one token per forward pass during decode.

• Parallelism: Efficiently leverage more than one GPU to accelerate large models without introducing new bottlenecks.

• Disaggregation: Separate the two phases of LLM inference, prefill and decode, onto independently scaling workers.

These model performance techniques are used for all modalities and not just LLMs, such as vision language models, embedding models, automatic speech recognition, speech synthesis, image generation, and video generation, which extend the capabilities of AI systems and require their own inference optimizations. But these runtime optimizations are not enough: no matter how performant a single instance of a model server is, it will eventually receive more traffic than it can handle. This is not a CUDA problem or a PyTorch problem, it’s a systems problem that needs to be solved at the infrastructure layer.

The nature of infrastructure problems changes at each level of scale. At first, the problems are around autoscaling: knowing when to add and remove replicas, and figuring out how to do so quickly.

Past a certain scale – generally a few hundred GPUs – infrastructure problems are defined by capacity. To get access to enough GPUs, inference engineers begin spreading workloads across multiple regions and cloud providers. This quickly leads to silos, where models in one cluster may be starved for resources while other clusters have unused capacity. The final level of scale in infrastructure is a global system that treats all available resources as a single unified pool of compute.

Thoughtful multi-cloud infrastructure also improves reliability, protecting against downtime in any individual region or cloud provider. And for global applications, running inference near to end users improves end-to-end latency.

Once these runtime and infrastructure capabilities are built, they need to be presented at the appropriate level of abstraction. Inference providers like Baseten and internal teams building inference need to consider what tooling and developer experience to provide as the critical third layer in a complete inference platform.

Of course, developer experience is subjective. For inference, one extreme is the black box: give a platform model weights, and get back an API. At the other extreme is providing only basic constructs for compute, network, disk, and so forth.

The right developer experience is somewhere in the middle, where inference engineers have enough control to run mission-critical inference confidently, and enough abstraction to work productively.

This article – which is an excerpt of Inference Engineering – presents an overview of the technologies and techniques that power inference across all three layers of runtime, infrastructure, and tooling.

3. When is inference engineering needed?

Inference engineering adds speed and scale to AI products by optimizing production serving of generative models. Optimization means identifying the best solution from a range of options.

Before optimizing model performance and building robust infrastructure, you need to know what “best” means for your product; many performance improvements come from making tradeoffs in latency, throughput, and quality. In practice, optimization is often about finding the right balance, rather than maximizing a single factor.

For example, NFL players are big, fast, and strong. But they’re not as big as sumo wrestlers, fast as Olympic sprinters, or strong as champion powerlifters. Their bodies and skills are optimized to fulfill the specific demands of their position over the course of a full season.

Similarly, your inference system must be optimized to fulfill the specific demands of your model, product, and traffic. The more constraints you can introduce, the better the outcomes that can be achieved.

You should know:

• Model requirements: Which model(s) do you need to run inference on?

• Application interface: How will inputs be delivered to the model, and how is the output expected to be formatted?

• Latency budget: How fast does your product need to respond to a user action, end-to-end?

• Unit economics: How much sense does it make to spend on a per-request, per-user, or per-month basis?

• Usage patterns: How many concurrent users are you serving, and is there any pattern to their usage (e.g., more activity during business hours)?

Early in building an AI product, the answers to these questions may not be clear. At this point, it’s often better to use off-the-shelf APIs whenever possible, rather than investing in dedicated inference. But as a product scales, the requirements become clear and inference engineering becomes a worthwhile pursuit.

4. What hardware does inference use?

Inference engineering relies on accelerators: powerful hardware designed to load terabytes of data and perform trillions of operations per second.

The most common type of accelerator for inference is the GPU, and the market leader in GPUs for inference is NVIDIA. My book focuses on inference engineering for NVIDIA GPUs in the datacenter, and also covers other vendors of datacenter accelerators and local inference.

Across vendors, there are three types of GPUs on the market:

• Datacenter GPUs: Racked servers with interconnected high-performance GPUs. Example: NVIDIA B200.

• Workstation GPUs: Individual desktop GPUs for professional workflows. Example: NVIDIA RTX Pro 6000.

• Personal computing GPUs: Individual desktop GPUs for everyday use. Example: NVIDIA GeForce RTX 5090.

Inference at scale uses datacenter GPUs mounted on racks: refrigerator-sized chassis with standardized power, networking, and cooling.

Datacenter GPUs like the NVIDIA B200 offer the highest individual performance, and more importantly, include high-bandwidth GPU-to-GPU interconnects, are installed in highly standardized configurations, and are available by the millions in datacenters worldwide.

I doubt there is a B200 GPU running under your desk, but if there is, please send me a picture! Instead, inference on datacenter GPUs runs in one of three modes:

• Cloud: GPUs are rented in someone else’s datacenter, usually hyperscalers like AWS and GCP, or neoclouds like CoreWeave and Nebius.

• On-premises: GPUs are purchased and installed in a datacenter that you control directly.

• Air-gapped: GPUs are installed on-prem and you need to physically access the GPUs to run inference.

Most inference engineers use cloud GPUs. Large enterprises and governments run on-prem and air-gapped deployments, but cloud-based GPUs offer the flexibility and access that fast-growing AI products need to scale.

Even with constraints, navigating the hardware landscape is complex. From variations between cloud providers, to NVIDIA’s personal naming conventions, there are many nuances in selecting the right accelerator.

5. What software does inference use?

NVIDIA’s market dominance in the inference space is in no small part due to the robust, mature software ecosystem around its hardware. Hardware iteration cycles are slow. Best-in-class hardware companies like Apple and NVIDIA release new architectures and generations at most annually, with two-year release cycles being more common. But software iteration is fast. Often, to run a newly released open model on day zero, you need to install a nightly build or other pre-release version of each software dependency just to get support for the new model.

Software’s fast iteration cycle and lower barrier to entry dramatically expands the landscape of inference engineering. There are countless companies building software at various levels of the inference stack, in contrast to hardware, which centers on NVIDIA and a few competitors.

For inference engineers, these are some of the key software players:

• NVIDIA: Invests heavily in its own sometimes-proprietary software ecosystem, from CUDA up to Dynamo.

• Hugging Face: Maintains a model registry for all open models plus transformers (models built on the transformer architecture), and diffusers (models built on the diffusion-based generative ones).

• The Linux Foundation: Maintains hardware-agnostic projects like PyTorch and vLLM.

• LMSYS Org: Develops essential tools for inference and evaluation, most notably SGLang.

There are thousands more companies, universities, and research institutions making essential open-source contributions to inference. Over time, technologies have been built at increasing levels of abstraction:

• CUDA: Direct communication to the GPU for explicit control over computations and memory.

• Deep learning frameworks: Abstractions over CUDA for training, exporting, and running neural networks in Python.

• Inference engines: Highly configurable PyTorch-backed inference for common architectures.

• NVIDIA Dynamo: Sits on top of inference engines to power large-scale deployments.

Most inference engineering today happens at the higher levels of abstraction, configuring and deploying inference engines and orchestrating inference across multiple GPUs. No matter which level of the stack you work at, it’s essential to have a strong mental model for the adjacent levels of abstraction to guide your work.

6. What infrastructure does inference need?

When you scale production traffic, your assumptions are rigorously tested. Everything from input and output sequence lengths, to traffic patterns, to what topic a user decides to chat about; they all impact your observed performance in production. And maintaining secure, robust infrastructure is an entirely different skillset from optimizing model inference on the GPU.

No matter how fast and efficiently a single instance can serve a model, the service will be overwhelmed if traffic gets high enough. It’s an infrastructure problem, not with PyTorch or CUDA, and it requires a different mindset and different technologies.

Scaling in production introduces new complexities about where and how to get GPUs, balance traffic across them, and prevent downtime. The goal of autoscaling is to ensure you always have enough resources to serve all incoming requests, while maintaining latency SLAs and without wasting money on idle GPUs.

Autoscaling systems use Kubernetes, an open-source container orchestration system, along with a cluster-level system for provisioning and deallocating compute. Kubernetes can run one or more replicas of a model container, each on its own instance. An instance includes the GPUs and other hardware resources that the container requires.

Unless your traffic is unusually consistent, there probably isn’t a specific number of replicas that perfectly matches your needs.

Autoscaling is the practice of dynamically adjusting the number of replicas allocated to a given model within a cluster. There are two ways to make autoscaling decisions:

• Utilization: Scale up or down based on GPU utilization signals like memory usage or compute usage.

• Traffic: Scale up and down based on the number of requests being processed in the system.

Utilization and traffic don’t always match. For example, in LLM prefill, a few requests with hundreds of thousands of uncached input tokens could cause much higher utilization than many small requests with high cache hit rates.

Traffic-based scaling decisions can be made proactively, while utilization is a lagging indicator. Use both in combination to keep system resources matched with demand.

When designing a traffic-based autoscaling system, you want to configure five factors:

• Min replicas: What is the minimum number of replicas that stay running, regardless of traffic?

• Max replicas: What is the maximum number of replicas you can allocate when traffic is high?

• Autoscaling window: How long is the sliding timeframe used to measure traffic and make autoscaling decisions?

• Scale down delay: For how long after a scale-down is suggested do you wait, in case there’s another traffic spike?

• Concurrency target: How many requests can each replica handle at once?

The exact configuration determines how well the autoscaling system achieves its goals of maintaining latency SLAs without wasting resources. For example, increasing the scale-down delay prevents premature scaledowns for spikey traffic, but could result in unnecessary spend after traffic has properly cooled down.

Autoscaling within a single cluster works up to a certain point, but high-volume deployments serving a global user base need thousands of GPUs distributed around the world.

It’s straightforward to build multi-cloud inference as a collection of siloed compute across different cloud providers. But in these setups, there’s no way to use inter-cloud compute fluidly, and moving workloads across clouds is a tedious, error-prone process.

True multi-cloud inference requires building a multi-region, multi-provider bin packing tool, which treats distinct pools of compute as fungible with each other. Like Kubernetes within a single cluster, multi-cloud capacity management must take a global view, enabling global scheduling.

Running true multi-cloud inference unlocks:

• Capacity: Pool capacity from multiple providers for greater, more flexible GPU access.

• Redundancy: Split inference across providers for resiliency against outages.

• Latency: Run inference close to end users to reduce network latency overhead.

• Compliance: Run inference in compliance with data sovereignty and other regulatory requirements.

Scaling from one cluster in one cloud to many clusters in many clouds requires a new coordination layer. A multi-cloud architecture contains:

• Control plane: Handles model deployment and global scaling decisions, receives real-time event streams.

• Workload planes: Handles direct inference traffic and in-cluster scaling decisions, reports utilization and demand.

This separation of responsibilities ensures that individual workload planes can serve traffic independently. If something happens to the control plane or any given workload plane, other workloads should be unaffected.

7. Five approaches to make inference faster

One of the coolest things about working in inference engineering is that, unlike many industries where new academic research takes years or decades to be adopted, techniques from new papers are live in production within months or even weeks.

But there is a gap to bridge between research and production, and some of the most visible inference engineering work of all comes from doing so.

Real-world traffic defies constraints. But with volume, you can adapt systems over time to match the changing nature of usage. Tuning the parameters of inference engines, speculation algorithms, and model servers isn’t a one-time task. Instead, either through iterative deployments or dynamic runtime adjustments, you can continuously improve the performance of an inference system.

Finding the right combination of techniques and configurations takes patient experimentation. I remember an internal hackathon during which one of Baseten’s inference engineers worked on an autocomplete model for code, and ended up trying 77 different configurations via a handwritten script before finding a non-obvious solution that doubled TPS (tokens per second) for a customer’s model.

Sometimes, techniques are symbiotic or incompatible, which makes inference optimization even more complex. For example, quantizing the KV cache alleviates a bottleneck in disaggregation, but increasing batch sizing reduces the compute available for speculation. An inference engineer’s challenge is always to create a balanced set of optimizations that delivers more than the sum of its parts.

Let’s look into the key categories of applied research for inference acceleration: quantization, speculation, caching, parallelism, and disaggregation.

Approach #1: Quantization

Quantization means reducing the numerical precision of a model’s weights. It improves latency (both TTFT [time to first token] and TPS, increases system throughput, and opens up headroom for other optimizations like disaggregation, speculation, and prefix caching to be even more effective. But when it goes wrong, quantization can materially reduce a model’s output quality.

Models are trained with weights, activations, and other components represented in a certain native number format. Usually, this is BF16 or FP16, although 8-bit and 4-bit native precisions are becoming more popular for training.

Post-training quantization works by changing those model weights and other values from their native number format, to a lower-precision format. Cutting precision in half improves performance in both phases of inference:

• Prefill: Compute-bound prefill now runs on lower-precision Tensor Cores with twice the FLOPS.

• Decode: Memory-bound decode now loads half as much data per value, effectively doubling memory bandwidth.

Working with quantized data introduces overheads, so it’s not linearly twice as fast to go from 16 to 8 bits. In practice, quantization down a single level of precision generally offers 30%-50% better performance for LLMs. The catch with quantization is that it runs the risk of reducing a model’s output quality, and has the potential to introduce precision errors throughout the calculations that power inference.

Precision errors compound over time. Consider what happens when you square and cube different precisions of Pi:

Most of the work in quantization is in preventing precision errors and minimizing their impact on the final model output.

Sixteen-bit, 8-bit, and 4-bit precisions are the primary formats for inference. Number formats contain:

• Precision: The number of bits used to express a single value in the format. For example, FP16 uses 16 bits.

• Type: Whether these bits are interpreted to represent an integer (non decimal) or a floating-point number (decimal).

• Scale factor: A multiplier used to map values from a low-precision format back to the higher-precision format.

Combined, these attributes determine the two factors behind how well a number format represents the values used in inference:

• Dynamic range: The difference between the lowest and highest value that can be represented in the format.

• Granularity: The number of parameters or other values that are quantized along a single scale factor.

Dynamic range is essential to low-precision inference without quality loss. Sixteen bits can represent 65,536 distinct values, while 8 bits can only represent 256 different values. The dynamic range is the distribution of these values – the difference between the smallest and largest available value.

Dynamic range explains why floating-point formats are better than integer formats for inference. Floating-point formats have three properties:

• Sign: A single bit that represents whether a number is positive or negative.

• Exponent: A set of bits that, taken together, represent an exponent factor.

• Mantissa: A set of bits that together represent the base value multiplied by two to the exponent.

An FP8 number in an E4M3 data format means it has a 4-bit exponent and a 3-bit mantissa, with the remaining bit for the sign. Integer formats only have sign and value bits.

The exponent in floating-point numbers gives it a higher dynamic range, meaning it can better express very large and very small numbers. This is important because outlier values are significant in inference, and floating-point number formats better represent outliers after quantization.

Within floating-point formats, there are multiple options at each precision, like FP4, MXFP4, and NVFP4. These formats differ in granularity, or in the number of values quantized by a single scale factor.

Quantization can be applied at three levels:

• Tensor level: Calculate a single scale factor for the entire QKV tensor.

• Channel level: Calculate a different scale factor for each feature vector within the tensor.

• Block level: Within each feature vector, divide the vector into blocks of N values and calculate a scale factor for each block.

More granular quantization has a lower chance of smoothing over outliers, which preserves quality. However, more granularity also introduces extra overhead for storing and applying scale factors.

The components of a model have varying sensitivities to quantization. Reducing the precision of more sensitive components runs a higher risk of quality degradation. From the least to most sensitive components:

1. Weights: the linear layers are least sensitive to quantization.

2. Activations: The intermediate output of activation functions are only somewhat sensitive to quantization. They are rarely quantized as they are such a tiny fraction of the model’s weights.

3. KV cache: The cached values from the attention calculation are moderately sensitive to quantization.

4. Attention: The attention layers of a model are highly sensitive to quantization, especially equations like softmax.

Within each component, you can get more selective about quantization.

Even in linear layers and activations – generally the least sensitive to quantization due to their size – early and late layers, like the input and output layers of the neural network, may be left in their original precision as these layers are more sensitive.

While quantizing weights and activations helps performance, KV cache quantization gives an additional boost to techniques like prefix caching and disaggregation. The KV cache is a valuable resource and quantizing it allows inference engines to store more of it in memory and read it more quickly.

However, the KV cache for each token is used by each subsequent token. This means precision errors introduced by quantization can compound from token to token. Compounding errors are exactly why attention layers are the riskiest to quantize: not only is attention very sensitive to dynamic range, but each attention calculation relies on the results of each previous attention calculation. Therefore, over a sequence of thousands of tokens, errors accumulate quickly.

All but the most aggressive quantization schemes run functions like softmax in their original precision.

A moderate approach to low-precision inference uses a format like FP8 with high dynamic range – if possible, a microscaling format like MXFP8 – to carefully quantize select linear layers, activations, and often KV cache values. Even with these high dynamic range formats, components of the attention layer are rarely quantized.

Approach #2: Speculative decoding

The decode phase of LLM inference is an autoregressive process in which tokens are generated one at a time. The bottleneck on decode is memory bandwidth, with compute sitting idle at low-to-moderate batch sizes as weights are read from memory.

Speculative decoding takes advantage of that spare compute to try and generate multiple tokens per forward pass through the target model. If an inference engine could generate two, three, or even more tokens for each round-trip of weights through memory, it would generate far more tokens per second. Note, speculative decoding only improves TPS / ITL (inter-token latency), not TTFT (time to first token.)

There are multiple algorithms for speculative decoding and they share a common mechanism:

1. The speculator generates one or more draft tokens.

2. The target model – or the underlying model that you’re trying to accelerate – performs validation on these tokens to check if they match what the model would generate.

3. The target model accepts any valid draft tokens and generates an additional token itself, completing the forward pass.

This generates N+1 tokens per forward pass, or iteration through the decode loop, where N is the number of accepted draft tokens.

Generating draft tokens is not free, it takes both compute and memory. However, it is much faster for a target model to validate a draft token than to generate an original token. If you imagine a sudoku puzzle, solving it is hard, but checking if the solution is correct is very easy. For the target model, generating a token is like solving a sudoku, while validating a draft token is like checking a finished sudoku.

The performance uplift from any speculative decoding strategy depends on three factors:

1. Draft token cost: Time taken to generate a draft token.

2. Draft sequence length: The number of draft tokens generated per forward pass.

3. Token acceptance rate: The percentage of draft tokens accepted by the target model.

Token acceptance rate is high early in the draft sequence, but draft tokens get less reliable deeper in the sequence.

Aim for short, high-percentage sequences because while generating and validating tokens is inexpensive relative to generating tokens in the original model, it still comes with meaningful overhead. Additionally, once a single draft token is rejected as wrong, all subsequent tokens in the sequence are also rejected.

Working with speculation is interesting because so many factors affect token acceptance rate. The big one is the temperature parameter – higher temperatures yield token distributions that are harder to predict, reducing the effectiveness of speculative decoding. But even factors as simple as subject matter can make a difference on acceptance rate if the draft model or additional decoder head used for speculation is better versed in, say, math than history.

Another limitation on speculative decoding is that it’s most useful at low batch sizes where there are spare compute cycles. At higher batch sizes, speculative decoding must be dynamically disabled as compute is too saturated to afford verification.

Each speculation algorithm navigates these tradeoffs differently, and careful implementation of the right algorithm for the situation can lead to major improvements in TPS.

Approach #3: Caching

During prefill, the inference engine builds a KV cache (a store of keys and values for each token) on the input sequence. It then updates the KV cache for each token during decode. As inference is autoregressive, the value for each new token depends on the value of every previous token in the sequence.

Every inference engine uses KV caching by default on a request-by-request basis. Without KV caching, LLM inference would be unbearably slow since each previous value in the entire sequence would need to be recalculated for each subsequent token.

However, engineers can get more utility from the KV cache by reusing it between requests rather than solely within each inference sequence.

Consider the following two prompts, each with four tokens on most tokenizers:

By default, the inference engine has to run prefill on all four tokens of each prompt. But the first tokens of each prompt – “Weather in” – form a shared prefix between the pair.

With prefix caching, you can reuse the KV cache from the first request to improve TTFT on the second request by skipping prefill on the first two tokens and reading in the existing KV cache instead.

When you see pay-per-token APIs charge less for “cache hit” input tokens than “cache miss” tokens, this is why: reusing cached tokens takes very little compute power and time. As an inference engineer, you can apply the same principle to reduce latency, improve throughput and therefore save money on your own deployments.

Saving two tokens won’t make a big impact on TTFT, but prefix caching can skip prefill on thousands of tokens in certain domains:

• Complex system prompts: Agents, customer-facing chatbots, RAG scaffolds, and tool calls often feature long, complex system prompts on every call.

• Code completion: Code completion, code generation, and other coding functions require passing the same thousands of lines of code as shared context.

• Documents and retrieval: Document summarization, question answering, and retrieval all add repeated context ahead of user prompts.

• Multi-turn conversations: Ordinary conversations repeat back every message in a chat template, increasing the savings from prefix caching with every turn.

Prefix caching works from the start of the input sequence until the first non-repeated token. The fourth token in the weather example, a question mark, is shared between the two input sequences. However, the prefix ends at the first non-repeated token, so the fourth token isn’t read from cache.

Since prefixes end at the first unique token, your context engineering determines TTFT savings. Consider a different approach to the same prompt:

Here, there is no savings from prefix caching, as the very first token differs between the two sequences, even though every subsequent token is the same. To take advantage of prefix caching, ensure that novel tokens are as late in your prompt as possible.

Approach #4: Parallelism

Tensor Parallelism (TP) should be your default strategy for multi-GPU model inference. It supports dense models like Llama 405B, and the MoE (mixture of experts) models that currently dominate the open model landscape.

TP works by splitting apart each layer of the model (as opposed to Pipeline Parallelism, which keeps layers intact) and distributing the layer fragments across the allocated GPUs. For each layer, the expense of reading from weights’ memory and executing matrix multiplication is shared across the GPUs.

However, the results of each layer need to be communicated in an all-reduce fashion (across all eight GPUs) into a single output before the next layer can be computed. In nodes with high-bandwidth intra-node NVLink and NVSwitch, this communication overhead is minimized.

Increasing Tensor Parallelism improves TPS on a per-user basis, assuming the model is large enough and the sequences are long enough that the communication overhead doesn’t outweigh the faster forward pass – which is the case for most frontier models.

Expert Parallelism (EP) neatly divides experts across GPUs, so that in a model with 128 experts served in EP8 across eight GPUs, each GPU hosts 16 full experts.

EP improves total system throughput, making inference more scalable and less expensive. With individual experts processing tokens separately, each token takes just as long, but the system as a whole can handle more simultaneous tokens.

Many deployments use a mix of TP and EP to achieve both benefits.

EP requires less inter-GPU communication than Tensor Parallelism. The Expert Router, which determines which experts each token activates, is replicated onto each GPU as it is a relatively small component of the model. Inter-GPU communication is necessary for passing tokens from expert to expert, but unlike TP, it is not required to collect the results of each layer.

Thanks to this lower communication overhead, EP scales well to multi-node deployments and systems with limited interconnect bandwidth.

Approach #5: Disaggregation

Disaggregation combines three important ideas in inference engineering:

1. Prefill is a compute-bound process that determines the time to first token (TTFT), while decode is a memory-bound process that determines TPS.

2. Specialization improves performance in everything from kernel selection to inference engine parameter tuning.

3. You can effectively parallelize model serving over multiple GPUs, or even multiple nodes, if you can avoid bottlenecks from lower-bandwidth interconnects.

When prefill and decode run on the same node under heavy traffic, they have a higher chance of interfering with one another. Ideally, prefill uses more compute resources, while decode uses more memory, and the two can co-exist efficiently. However, with larger batches and more compute-intensive optimizations, prefill and decode start competing for resources.

Disaggregation, or disaggregated serving, is the idea of separating prefill and decode into separate engines on separate GPUs or nodes.

Disaggregation turns LLM inference into a three-step process:

1. The prefill engine takes the input sequence and generates a KV cache while computing the first token.

2. The prefill engine sends the KV cache over the hardware interconnect to the decode engine.

3. The decode engine computes all subsequent tokens.

In conditional disaggregation, the request is first sent to the decode engine, which checks if the input sequence is already cached, or is short enough to handle locally:

1. If so, the decode engine handles prefill locally, skipping disaggregation.

2. If not, the decode engine transfers the request to the prefill engine for disaggregated serving.

Conditional disaggregation is better for real-world traffic.

Another benefit of disaggregation is that with separate prefill and decode engines, you can optimize each engine individually and the system as a whole. For example, the compute-bound prefill engine requires a lower TP than the memory-bound decode engine.

Takeaways

This is Gergely again. Thanks to Philip for this deepdive into inference engineering, which is around 10% of the contents of his new book, ”Inference Engineering.” If you’d like to go deeper into this topic, you can download the full book for free:

Get the full e-book, for free

This title will also be available in physical, printed form: sign up to the waitlist to be notified when it’s available.

It’s encouraging that inference engineering is no longer a “monopoly” belonging to a few leading AI labs. Top AI model makers like OpenAI and Anthropic control all aspects of their AI models – from training to inference – so there’s no inference engineering to be done with them.

However, thanks to increasingly capable open models, engineering teams have the opportunity to tweak how they use models, and this is where the theory and practice of inference engineering becomes invaluable.

Even so, the discipline of inference engineering still seems to only make sense for a subset of tech companies. To justify investment in inference engineering, you need to be spending big money on inference from vendors. This is the point at which it can make sense to invest time and money to see if you can set up your own inference stack on top of open models, and swap out some existing usage.

I wonder if inference engineering is the AI version of the “build vs buy” dilemma. For software-as-a-service (SaaS), the question for every company is whether to build it in-house, or buy from a vendor. For example, should you build a project management software (it’s possible!), or just buy an existing one? And what about feature flagging, not to mention observability?

Experienced engineers all understand the pros and cons of building it yourself (time and maintenance, which is a constant drag.) Tuning and operating your own LLM stack is a much newer field, and inference engineering is at the heart of building better inference stacks than what comes “out of the box” with open models.

Picking up the basics of inference engineering feels like a valuable skill – and it’s also new and interesting. If you become well-versed in inference engineering, you could create optionality for your own team and company in LLM usage. Running your own inference stack on top of an open model gives control of what you’re running and of pricing. Inference engineering helps create options for achieving better performance from an open model by using the approaches covered in the extract above from Philip’s book.

Today, we cover:

1. Does GitHub still merit “top git platform for AI-native development” status? Availability has dropped to one nine (~90% – !!), partly due to not being able to handle increased traffic from AI coding agents. There’s also no CEO and an apparent lack of direction.

2. Should a tool auto-add itself as a contributor to PRs? Claude Code and GitHub Copilot auto-add themselves to commits, which is effectively free advertising. Codex and OpenCode purposely do not.

3. Microsoft promises Windows will not be “Microslop.” After years of forced Copilot integrations, Start menu ads, and mandatory Microsoft accounts, the Windows team is promising to undo the self-inflicted damage done to the OS. It’s better late than never, but why did Microsoft allow the “Microslop” perception to stick around so long?

4. Industry pulse. Massive LLM supply chain attack via LiteLLM, backlash after Cursor forgets to mention that Composer 2 is based on an open source model, what happens when you stop reviewing AI code, OpenAI kills Sora, and more.

1. Does GitHub still merit “top git platform for AI-native development” status?

We’re used to highly reliable systems which target four-nines of availability (99.99%, meaning about 52 minutes of downtime per year), and for it to be embarrassing to barely hit three nines (around 9 hours of downtime per year.) And yet, in the past month, GitHub’s reliability is down to one nine!

Here’s data from the third-party, “missing GitHub status page”, which was built after GitHub stopped updating its own status page due to terrible availability. Recently, things have looked poor:

Read more

After I left Uber, we stayed in touch, and a few months ago I heard he was exploring the next opportunity, and found out from him that Sam’s job search looked nothing like most people’s: he didn’t apply for a single role. Instead, there were reachouts from former colleagues desperate to hire them.

When we talked for this article, Sam had three warm leads which wanted to interview him ASAP. One startup was not even hiring, but the founder was ready to create a new position just for him.

I posted a message on LinkedIn about Sam:

“I hate the term “10x engineer” but this engineer is a role model for what a standout engineer is - in fact, some of my writing of standout engineers reference my interactions with folks like them (e.g. my article on the product-minded engineer, this one: https://lnkd.in/et7nWBgW)

And still, from the outside, this engineer is nearly completely invisible.

No social media footprint. The LinkedIn profile lists his companies worked at, and nothing else: no technologies, no projects, nothing. Their GitHub is empty for the last 5 years, and has perhaps a dozen commits throughout the last 10 years.”

This is Sam’s GitHub contributions for the last several years: absolutely nothing.

One of the most upvoted comments on my post was by cloud technologist Olivier Frolovs, who requested an article on Sam for others to learn how he operates. Now, Sam has generously agreed to an interview and has asked to remain anonymous, hence the nom de plume (pseudonym).

He doesn’t seek public attention and has a strong professional reputation. During our chat, he offered pointers for engineers looking to excel, and also as proof that an empty GitHub profile and zero social media presence don’t mean you can’t be a truly standout developer.

I also interviewed one of his former managers from Uber for that perspective. Today, we cover:

1. Getting things done. High-level task breakdowns, combined with communicating delays as tradeoffs to stakeholders.

2. Setting boundaries. Saying “no,” prioritizing family or work – and being clear about it, and treating prioritization as a daily practice.

3. Office politics. Participate selectively and cautiously, build relationships with influential colleagues, pre-sell ideas, direct communication.

4. Negotiation and conflict. Approach engineers before their managers, build bottom-up consensus, and start with relationship-building.

5. Promotions, keeping up-to-date, and finding the next job. A personal take on The Big Tech promotion processes, keeping up with the industry, and relying on referrals more.

6. Becoming a manager. Ownership and independence separate “good” engineers from “great” ones.

7. Feedback from Sam’s ex-manager. A former manager reveals what made Sam stand out to them – and shares some potential growth opportunities.

Background

In this article, my questions are in italic and Sam’s voice is in normal text.

Sam, how did you get into tech?

I was always intrigued by computers, software, and – as it came around – the internet. My dad had a black-and-white screen laptop for work with Windows 3.1 and some games. We got our first personal computer with Windows 95 and I remember it vividly.

I started developing websites for my primary school and the company my mom worked at when I was 12. I got paid a small sum, so that was my first-ever paid programming project! Around then, I taught myself Visual Basic 6 and started building 2D and 3D mini-games. I found the website DirectXVB (which is still live today), emailed the website’s owner with issues I ran into, and they helped me with pointers. Later, I taught myself PHP and built more dynamic websites.

From the age of 14, I stopped all coding – I just got tired of it! – and focused on my studies, and getting into college. I chose a non-computer science major for university, but picked up coding on the side, and I rediscovered that spark, so, in the first year of my Master’s, I decided to switch and do a Computer Science Bachelor’s. It was during college that I started to build apps and websites, and it’s when I got truly hooked on software development.

From agency, to large company, then Uber

I joined an agency as their first hire, building apps for local companies. It was a small team, we learned with trial-and-error, and getting done at 2-3 o’clock in the morning was common enough. I stayed 18 months and learned a lot about ownership, the importance of an eye for detail, and collaborating with others.

My favorite part of the job was the few times I worked directly with a designer: our agency employed freelance designers who were not involved in most of the projects because the company was trying to save money by having them work less, and not be involved in planning and rollouts. But during the implementation phase, I’d find myself talking with the designer and bouncing implementation and design ideas around.

I then joined a small startup where we built our own product. A highlight was having two designers on the team fulltime, whom I could work with and learn from. Engineering also felt like a level up: everyone cared about software quality and UX details.

Our startup got acquired by a larger company and most of us moved to the Bay Area. We stayed together as a team and were told we would maintain a “startup culture.” The founders tried their best to stay true to their word, but they couldn’t shield us from the reality of working in a corporation.

I learned a lot about corporate processes, and it was more interesting than I’d expected. As I was getting closer to the senior engineer level, I had to understand how internal politics worked, how to “massage” peer teams to help support our proposals, and how to talk with engineering leaders like senior managers and directors. Our company was also hugely focused on the annual company event: it was eye-opening for me to see just how much effort went into preparation. It consisted of several rehearsals and dedicated engineering work to showcase our stuff in a way that was near-flawless on the day.

After a few years, I felt ready for a change and joined Uber. I took a “title cut,” something akin to the “seniority rollercoaster.” At Uber, I worked in a new area and got promoted several times. After Uber, I worked at another Big Tech, and now – very recently – I’ve begun at a startup.

1. Getting things done

Feedback at Uber about you during performance calibrations was that you’re excellent at getting things done. What’s your process?

When I started out as a junior dev, I pulled long hours so I could deliver on time – regardless of how much effort it took. I don’t know what it was, but I always felt that failing to deliver on time was never an option.

I still vividly remember one project where I worked incredibly hard but still failed to deliver with the quality I expected from myself. As embarrassing as it is, I was so exhausted that I almost started crying on the spot. One of my coworkers comforted me and told me:

“Man, you’re crying about the wrong thing. No one died, no one got hurt, and no one will even care that we’re a few days late, save for the project manager. But even he’s used to everything being late. Go home, have some sleep, come back tomorrow and take it easy.”

They were right, of course. Still, I’m pretty sure this inner pressure to be unsatisfied with “good enough” explains a lot about how I work.

Having a high-level breakdown of the work, and communicating to stakeholders has been my “secret”, later in my career. After a few years as a dev, my estimation skills got better and I had to pull fewer late nights. I also found a hack that greatly helped was doing a high-level breakdown as early as possible, in all cases. As soon as I understand what the work is, I break it all down, ideally on a whiteboard or paper.

Importance of communication

You were also seen as a strong communicator, whether it was with engineers, engineering managers, or product managers. How do you get your point across?

Communicating delays as “tradeoffs” works extremely well. As soon as I start a project which I’m the lead on, I establish communication channels with key stakeholders: product managers, my engineering leadership, and business stakeholders, via email or Slack, I keep them in the loop at least weekly, and about anything that could be a roadblock.

In my experience, delays are not an issue as long as they are communicated upfront with an explanation and potential alternatives. When we hit a roadblock that slows down our work, I would never communicate that we’re “behind”. I would offer alternatives like:

• We can still ship on time, but we’d need to cut X and Y features for this release

• If we are not comfortable cutting X and Y features, then we will need to push out the target date by 2 weeks. If we are comfortable, we can push it out by 1 week

The trick, I’ve found, is to make it clear to stakeholders that we have a choice: choose more features to ship, or choose a lower-priority feature to drop.

I learned most of my hacks from people who are good at getting things done, and they have a few attributes:

• Task breakdown: early in my career, there was a senior engineer who was methodical about breaking down tasks and making estimates, even for seemingly trivial projects – and it worked!

• Communication tools: I observed the few really organized product managers, engineering managers, and tech lead, and made their communication styles into a “package” that worked for me; things like email updates, facilitating kickoff meetings, launch announcements (including how to communicate a failed/sunset project as a successful launch), and more.

Being good at communication means you have a solid foundation, and then develop a feel for how to best utilize the tools you have. There’s no “one-size-fits-all” approach: people react better or worse to different things. Try to get to know folks around you and put yourself in their shoes.

Doing great work

What does “standout” work look like to you?

I think about the quality of my work similarly to the quality of work I do at home. I have moved houses and renovated them several times and I greatly care about the quality of that work. And I’ve seen plenty of contractors come to my place, perform their work, and then leave without actually caring about the quality. They just want to “get s*** done” and be out of there. I never understood how someone can keep doing their job without feeling a lot of love for it!

I need to get energy from everything I do, not just in my job. Whether it’s playing games with my kids, helping my wife with her website, or building a new website feature for the company I work for: I approach it all with the same attitude.

Equally, if I no longer get energy from the work I do, then I basically stop enjoying it and this can be a nudge to start to look for something else. If it continues for a long time, this urge can become more persistent, and that’s the point when I have switched companies or teams. I can go on for some time without getting energy from my work, but it drains me. I try to catch myself before it gets too bad, and I’ve managed to do so, up to now. This is why I quit my last job without having anything lined up: I stopped getting energy from it for many months and talked with my management chain about it, but they were unable and unwilling to change anything. I needed a change, so it was me who made it.

Stepping outside of domain expertise

You frequently went outside of your domain, working with engineering teams on other platforms and contributing to codebases you’re not expert in. You seemed to have a great relationship with most engineers, in contrast to some devs. How did you do this?

I am pretty curious and prefer to talk directly with engineers. So, when I’d work on a project with engineers on a different stack, I would ask them to explain their high-level architecture approaches, and roll up my sleeves to make small code changes in a stack I was unfamiliar with.

Once you understand the high-level structure of a different codebase, and you also know how to make a few small changes, suddenly, it’s so much easier to figure things out on your own!

An approach that consistently worked for me is approaching problems from the customer’s perspective, and being genuinely curious. For example, I might ping an engineer working on a different system and ask:

“I noticed a customer has this problem, and to fix it, we probably need to touch the system you own. I don’t know much about this system: can you explain how it works, and what we could perhaps do to solve this issue that causes frustration for the customer?”

By making it clear that my goal is to solve a customer problem, I’m not coming across as just digging around for nothing. And by making it clear that I’d like to learn from them, it avoids being seen as someone trying to confirm what they are doing, which could come across as arrogant – especially when the other engineer is the expert on their own system. I’ve found fellow engineers are happy to explain their understanding and decisions.

2. Setting boundaries

At Uber, I recall you were very good at setting boundaries and saying “no.” How do you do that?

Honestly, I find it tough to say no – but I learned that it’s worse when I don’t. I found that saying ‘yes’ to everything usually results in an unmanageable, unbalanced pile of work. Prioritizing is key: I always remind myself to focus on what matters most. For me, the “most important” thing for any given topic could be:

• A shipping deadline that needs to be hit and is non-negotiable

• Family needs

• An urgent task that needs to be done on the same day

Family is very important to me. When I worked at Uber, I had a reasonably long commute to the office. I blocked out my calendar so I could leave on time in order to be home for dinner with my family. This did not mean I stopped work immediately; I would sometimes work during my commute and, when necessary, I logged back on to continue working after my kids were in bed.

When we had important deadline agreements at work, I made an agreement with my partner that I stayed longer in the office because I knew it was important to put in extra effort and deliver standout work then.

It goes back to prioritizing and focusing on the most important thing. Looking back, I’d say most of the time, the most important thing for me was family, and that work overrode this every now and then.

My approach to prioritizing keeps changing, though. Demands at home keep changing and expectations at work also change; after Uber, other jobs increasingly focused on async and remote work. This meant more flexibility to accommodate family time – but work could spill over into evening hours if I did not finish everything.

If I can give one piece of advice, it’s to understand what is important for you. Know your number one, number two, number three priorities: and arrange your workday so you do your top priorities. Don’t compromise on the most important one!

3. Office politics

At work, how plugged in were you to office politics?

I was aware of politics and tried to build relationships with “influential” people. I try to stay away from “cocky” types, and to find what I want to achieve through different folks.

The importance of politics is something I really started to understand when working at Uber. Initially, I was ignorant, but the more experience I got in Big Tech, the more it became obvious. It took a while before I was able to participate in it. I never liked it; I tend to be direct and transparent, but that does not work in every situation.

Did you take part in it to get stuff done?

Yes, sometimes by being direct and transparent, and communicating the right amount of information, you can get a lot done. Occasionally, it required me to “massage” an idea on multiple people before going to the person who called the shots.

What is your view of engineers who are seen as“political”?

It’s part of the game and sometimes it’s useful to have a good relationship with those people, as you can use that for your own benefit, as well. I personally would never invest much time in understanding and practising politics, as I prefer to focus on building and product.

4. Negotiation & conflict

Negotiating with teams

You were perceived as being good with other teams, and at removing roadblocks for your own. How did you approach this?

Read more

Today, we cover:

1. Is the FDE role becoming less desirable? Job postings for Forward Deployed Engineers (FDEs) have surged, but many professionals don’t want the role because it’s more like solutions…

Read more

Listen and watch now on YouTube, Spotify, and Apple. See the episode transcript at the top of this page, and timestamps for the episode at the bottom.

Brought to You by

• Statsig – ⁠ The unified platform for flags, analytics, experiments, and more. Stop switching between different tools, and have them all in one place.

• Sonar – The makers of SonarQube, the industry standard for automated code review. Sonar helps reduce outages, improve security, and lower risks associated with AI and agentic coding. See how Sonar is empowering the Agent Centric Development Cycle with new products and capabilities that strengthen the guide, verify, and solve phases of development.

• WorkOS – Everything you need to make your app enterprise ready. Skip the rebuild for enterprise features. Keep shipping. Visit WorkOS.com.

In this episode

How did a tiny team of 30 engineers build the world-famous messaging app more than a decade ago, and what can dev teams learn from that feat today? Jean Lee was engineer #19 at WhatsApp, joining when the company was still small, with almost no formal processes. She helped it scale to hundreds of millions of users, went through the $19B acquisition by Facebook, and later worked at Meta.

In this episode of Pragmatic Engineer, I talk with Jean about what it was like building WhatsApp. When Facebook bought WhatsApp in 2014, only around 30 engineers supported hundreds of millions of users across eight platforms.

We discuss how the founders kept things simple, saying “no” to most feature requests for years. Jean explains why WhatsApp chose Erlang for the backend, why the team avoided cross-platform abstractions, and how charging users $1 per year paid everyone’s salaries, while keeping growth intentionally slow.

Jean also shares what the Facebook acquisition was like on the inside, how she dealt with sudden personal wealth, and what it was like transitioning from an IC to a manager at Facebook – including the reality of calibration meetings and performance reviews.

We also discuss how AI enables smaller engineering teams, and why WhatsApp’s experience suggests ownership and trust might matter more than tools.

Key observation from Jean

Ten takeaways from Jean that I find the most interesting:

1. WhatsApp built a billion-dollar business with a tiny team, and no AI tools. WhatsApp served 450 million users with only 30 engineers, long before AI tools existed. Jean says: “I wonder if being able to move fast is independent from AI. When you’re small, you’re just more efficient.”

2. WhatsApp had no code reviews after in-place. WhatsApp cofounder, Brian Acton, reviewed the very first pull request of each new hire, and after that, there were no more code reviews. Jean recounts how Brian reviewed her debut PR in extreme detail. This first (and only!) review set the bar high, and she wrote code to that standard from then on.

3. WhatsApp had close to zero formal processes. WhatsApp had no Scrum, no Agile, no TDD (test driven development), and no formal code reviews beyond the first commit. In contrast, Skype had 1,000 engineers and mandatory Scrum training, but WhatsApp still outcompeted it and won. Jean’s response to hearing of all the formal processes Skype used in order to execute faster: “I’m surprised to hear they thought they were shipping faster because of it.” Perhaps process is often a substitute for trust, not quality?”

4. WhatsApp’s office had a countdown display showing days since the last outage. When an outage happened, no emails were sent around, and no meetings were called. The number simply reset to zero. Avoiding outages was on everyone’s mind as a result. This is an example of how visible metrics can create accountability without bureaucracy.

5. WhatsApp delayed video calling for years, until it was extremely polished. Contrary to the “launch early, then iterate” mantra, WhatsApp held features like video calling back. They also tested features extensively with family members before releasing anything publicly, as part of their refusal to launch something of less than top-notch quality.

6. Saying “no” to features was a competitive advantage. WhatsApp’s CEO, Jan Koum, rejected 99% of feature requests from the team. While competitors shipped dozens of shiny, new features, WhatsApp ruthlessly prioritized reliability and simplicity. Jan repeatedly told the team what the mission was. “I want a grandma living in the countryside to be able to use our app”, he said.

7. WhatsApp’s team was older and more experienced than most startups at the time. In 2014 when Facebook acquired WhatsApp, only four out of the 30 engineers were less than 30 years old. Perhaps part of the reason for WhatsApp’s stunning success was having an unusually experienced team from the start.

8. AI won’t replace the human touch in engineering management. Jean sees areas such as OKR management, documentation, and performance data gathering as domains in which AI can take on most of the work. But she believes that understanding and unblocking engineers is best done person-to-person, not by AI.

9. Posting about your work on Meta’s “internal Facebook” site affects career growth there. Jean noted that engineers at the social media giant who regularly posted about their launches and learnings enjoyed a sizable advantage in performance calibration reviews.

10. Jean’s advice to new grads: invest in the fundamentals. “Tools come and go, languages come and go, but foundations don’t go anywhere,” she says.

The Pragmatic Engineer deepdives relevant for this episode

• How Meta built Threads

• How Big Tech runs tech projects and the curious absence of Scrum

• Performance calibrations at tech companies

• Software engineers leading projects

Timestamps

(00:00) Intro

(01:39) Early years in tech

(06:18) Becoming engineer #19 at WhatsApp

(13:53) WhatsApp’s tech stack

(18:09) WhatsApp’s unique ways of working

(25:27) Countdown displays and outages

(27:07) Why WhatsApp won

(28:53) The Facebook acquisition

(33:13) Life after acquisition

(39:27) Working at Facebook in London

(44:07) Transitioning to management

(47:27) Performance reviews as a manager

(53:29) After Facebook

(58:53) AI’s impact on engineering

(1:02:34) Jean’s advice to new grads and startups

(1:06:45) Empowering employees

(1:08:17) Book recommendations

References

Where to find Jean Lee:

• Substack: https://exaltitude.substack.com

• LinkedIn: https://www.linkedin.com/in/jeanklee

• YouTube: https://www.youtube.com/@exaltitude

• Website: https://www.exaltitude.io

Mentions during the episode:

• WhatsApp: https://www.whatsapp.com

• KakaoTalk: https://en.wikipedia.org/wiki/KakaoTalk

• Jan Koum: https://en.wikipedia.org/wiki/Jan_Koum

• Brian Acton on LinkedIn: https://www.linkedin.com/in/brianacton

• Yahoo: https://www.yahoo.com

• Sequoia: https://sequoiacap.com

• Cocktail Flow: https://cocktailflow.com

• KaiOS: https://en.wikipedia.org/wiki/KaiOS

• Erlang: https://www.erlang.org

• Ericsson: https://www.ericsson.com

• Erlang Factory 2014 - That’s ‘Billion’ with a ‘B’: Scaling to the Next Level at WhatsApp:

• WeChat: https://www.wechat.com

• Skype: https://en.wikipedia.org/wiki/Skype

• What is Scrum?: https://www.scrum.org/resources/what-scrum-module

• Mark Zuckerberg: https://en.wikipedia.org/wiki/Mark_Zuckerberg

• Wealthfront: https://www.wealthfront.com

• A Random Walk Down Wall Street: The Best Investment Guide That Money Can Buy: https://www.amazon.com/Random-Walk-Down-Wall-Street/dp/1324051132

• Surrounded by Idiots: The Four Types of Human Behavior and How to Effectively Communicate with Each in Business: https://www.amazon.com/Surrounded-Idiots-Revised-Expanded-Effectively/dp/1250420458

• Performance Calibrations at Tech Companies: Part 1: https://newsletter.pragmaticengineer.com/p/performance-calibrations

• Performance Calibrations at Tech Companies: Part 2: https://newsletter.pragmaticengineer.com/p/performance-calibrations-part-2

• Anthropic: https://www.anthropic.com

• What Color Is Your Parachute? for College: Pave Your Path from Major to Meaningful Work: https://www.amazon.com/What-Color-Your-Parachute-College/dp/1984857568

—

Production and marketing by Pen Name.

Last week, we took an inside look into how Uber is adopting AI, internally. The rideshare giant has built close to a dozen internal systems to deal with code generated by AI agents. However, when quantifying the impact of AI, the focus was on how much output has increased, and how devs who use more AI also generate more pull requests; these are the “power user” devs who generate 52% more PRs than devs who use AI less. There was no mention of product quality – at all!

And there are signs that product quality is dropping overall. Today, we dig into this under-discussed topic, covering:

1. Anthropic: degraded flagship website. An annoying UX issue irritated paying Claude customers – and no one at Anthropic noticed. The company moves very fast, generates 80%+ of production code with Claude, but quality and user experience seem to be taking a backseat.

2. Amazon: AI-agent reliance triggers SEVs. Amazon’s retail org has a leap in outages caused by its own AI agents. Now, senior sign off is needed for junior engineers’ AI-assisted changes.

3. Big Tech: “use AI or you’re unproductive.” Companies like Meta and Uber are tracking AI token usage in performance reviews, putting pressure on engineers to use it heavily — irrespective of the tools’ quality impact.

4. OpenCode: more time spent cleaning up. Dax Raad, OpenCode’s creator, warns that AI agents are lowering the bar for what ships, discouraging refactoring, and don’t speed teams up.

5. 5. Startups: founders see LLMs slowing down long-term velocity. Sentry’s CTO and others observe that while AI removes the barrier to getting started, it also produces bloated, hard-to-maintain code that slows long-term development.

6. Research: AI agents underperform claims. Some studies show AI coding tools produce short-lived velocity gains followed by significant tech debt increases.

7. How do we solve it? Engineers with strong architectural sense become more critical than ever, proposed solutions include formal validation methods, and perhaps reviving some old school QA ideas.

1. Anthropic: degraded flagship website

This article’s genesis was last week, when I’d finally had enough of a persistent UX bug on Claude’s flagship website: the prompt I typed in regularly got lost. Below is a video of me typing “How can I…” – and “losing” the first two words when the page loaded:

It’s pretty straightforward:

1. The page starts to render and the textbox is displayed

2. The user starts to type their prompt, but the page has not finished loading subscription data

3. The subscription information loads around a second later

4. The textbox is reset and the typing is lost

This is a pretty basic bug you might expect in a prototype, except that this is the landing page of Claude.ai, and it’s a bug that impacts every paying customer – easily millions – every day. Even worse, the bug happens every time you visit the site.

Somehow, nobody at Anthropic tested the site to catch a plainly obvious bug which impacted 100% of paying customers. At the same time, no company uses AI coding tools more than Anthropic: around 80% of the company’s code is now generated by Claude Code, so we can assume a good part of the website is also created that way.

My complaint about Anthropic’s website being broken went a bit viral and got the attention of the developer team:

To their credit, three days later the bug was gone. There’s no longer a “double load” of the textbox: it takes a bit longer to load but only does so once.

Still, it makes me wonder how much longer this issue would’ve continued had nobody complained. Also, how many more bugs are present on the Claude website that nobody highlighted on social media? How many more features could be shipped in a state that is subpar for production-grade software with millions of paying customers?

Anthropic seems to be prioritizing moving very fast over doing so with high quality. There is no denying that the company is moving at incredible speed and running laps around competitors. A good example is how they built Claude Cowork in just 10 days. Claude Cowork handled work with Microsoft Word and Excel documents surprisingly well, to the point that it set off a “code red” inside Microsoft’s Office division, I understand.

Microsoft responded as fast as possible, but it still took 2-3 months to launch their (cloned) response, called Copilot Cowork earlier this month, with full access still to follow soon.

In the case of Anthropic, moving fast with okay quality seems to make good business sense: they build a better product than what already exists, so no matter if it’s a bit rough around the edges; they can fix quality issues post-launch and still be months ahead of the competition.

2. Amazon: reliance on AI agents causes SEVs

Anthropic can afford to move fast while it’s growing at an extremely high rate and expanding its market share rapidly. At the same time, established players like Amazon have extreme focus on reliability: AWS has become the top cloud provider not least by being extremely reliable (as well as aggressive on pricing).

Well, reliability at the online retailer seems to be getting worse, too, and the company’s AI agent, Kiro, could be causing SEVs (Amazon’s phrase for “outage”), according to The Financial Times (emphasis mine):

Amazon’s ecommerce business has summoned a large group of engineers to a meeting on Tuesday for a “deep dive” into a spate of outages, including incidents tied to the use of AI coding tools.

The online retail giant said there had been a “trend of incidents” in recent months, characterised by a “high blast radius” and “Gen-AI assisted changes” among other factors, according to a briefing note for the meeting seen by the FT.

Under “contributing factors” the note included “novel GenAI usage for which best practices and safeguards are not yet fully established”.

“Folks, as you likely know, the availability of the site and related infrastructure has not been good recently,” Dave Treadwell, a senior vice-president at the group, told employees in an email, also seen by the FT. (...)

He asked staff to attend the meeting, which is normally optional.

Junior and mid-level engineers require more senior engineers to sign off any AI-assisted changes, Treadwell added in the briefing note.”

This meeting was the regular “This Week in Stores Tech” operational one, but what was new was the note telling staff to attend this “optional” meeting, and the mandate for senior engineers to sign off code changes from juniors. The outages may have been caused by less experienced engineers over-trusting GenAI’s output. Also, there were incidents caused by AI changes, said the FT:

“Separately, the company’s cloud computing arm — Amazon Web Services — has suffered at least two incidents linked to the use of AI coding assistants, which the company has been actively rolling out to its staff.

AWS suffered a 13-hour interruption to a cost calculator used by customers in mid-December after engineers allowed the group’s Kiro AI coding tool to make certain changes, and the AI tool opted to “delete and recreate the environment”, the FT previously reported.”

Again, a tool causing an outage is not its own fault: it’s on the engineer who lets the tool run wild. If I delete two lines of code, then push it to production, and the server crashes, the fault is not with the text editor or the Git client, but with me who made the change. Similarly, if you prompt an AI agent to do something, and the AI agent goes off and does its stuff which causes an outage, then responsibility lies with the engineer who didn’t set up guardrails for the agent.

However, there is the issue that AI agents can wreak havoc in ways devs don’t quite understand or expect, until learning the hard way. This was what took down a lesser-used AWS service, according to the report:

“Amazon Web Services experienced a 13-hour interruption to one system used by its customers in mid-December [2025] after engineers allowed its Kiro AI coding tool to make certain changes, according to four people familiar with the matter.

The people said the agentic tool, which can take autonomous actions on behalf of users, determined that the best course of action was to “delete and recreate the environment”.

It sounds like an engineer gave overly broad permissions to the coding agent which then used its scope to delete a service. As mentioned, the engineer is responsible, but there is also a learning curve with these AI agents to consider: it’s not like this type of outage happened in the past. Plus, companies like Amazon are heavily incentivizing using AI agents for as much work as possible, which naturally leads to overuse.

3. Big Tech: “use AI or you’re unproductive”

Something happens at places that measure devs’ AI usage: pressure builds for all devs to use more AI or else be seen unproductive and at risk of poor performance reviews, potentially leading to a PIP or worse.

Meta is taking token usage into account during perf reviews. A current engineering manager at the social media giant told me that the token usage of each engineer is now a data point — one of many! — for performance calibrations. By itself, it is not a positive or negative signal, but someone perceived as having low impact and with low token usage is now seen as a blatant low performer. For high performers with outstanding impact, very high token usage is seen as a good thing as it conveys to the manager group that they’re personally invested in AI and are improving their workflow – as proved by results.

We previously covered how performance calibrations work at places like Meta.

Big Tech CEOs are starting to see AI “power user devs” as superior to their coworkers. Uber is a good example: the Dev Platform team started to analyze the output of engineers by whether or not they’re in the “power user” category, meaning they use AI agents at least 20 days per month. They found more PR output by engineers who are power users. So far, that’s useful data, but it’s just one piece of information, and doesn’t reveal the quality of the PRs, the impact of the engineer, or any other business outcome.

By the time this data reaches CEO level, it has turned into something else. Here’s Uber CEO, Dara Khosrowshahi, interpreting the same data points on the Diary of a CEO podcast (emphasis mine:)

“While 90% of our engineers are using AI tools of some sort, there’s about 30% of them that are using them at a completely accelerated pace. And it [using AI tools heavily] really is changing their productivity in a way that I’ve never ever seen before.”

There’s a step from observing more PRs per engineer, to judging power users as being more productive for that reason. Dara continues:

“I can imagine maybe 5 years from now, as the engineers get more and more productive, that I may not decide to add engineering headcount because at that point instead of adding an engineer, I should add agents and buy some more GPUs from Nvidia. That may be the investment in the future.”

Unsaid in the above is that by that time, only engineers “using AI at a completely accelerated pace” would be employed. Would it also mean that engineers not on the bandwagon are on the way out? I appreciate Dara speaking his mind and shedding light on the thought process of a Big Tech CEO.

Inside large tech companies, it’s becoming a career risk to not use AI at an accelerated pace, regardless of output quality. These large companies are the ones likely to be mulling layoffs, like Meta reportedly preparing to cut up to 20% of staff. And when it comes to identifying redundancies, it’s a fair assumption that things like “AI usage” and “pull requests per engineer” will be taken into account, especially as one theme of such layoffs will almost certainly be that the employer wants to focus more on AI.

So, it’s common sense (and self-preservation) to use more AI, if only not to be seen as unproductive. Their perceived output will rise and engineering leadership will share more reports about productivity being up, and interpreting more code generated and more pull requests as the proof.

4. OpenCode: “more time spent cleaning up”

Dax Raad is founder and CEO of OpenCode, an open source AI coding agent, into which you can plug in models like Claude, ChatGPT, Gemini, and others. It’s an increasingly popular alternative to the likes of Claude Code and Codex. In our recent AI tooling survey, it came up as a tool used nearly as much as Google’s Gemini CLI and Antigravity. A small team works on this increasingly influential tool, and is seeing problems with AI overuse. Dax wrote this note to the OpenCode team (emphasis mine):

Read more

Before we start, I’d like to share updates about data in the two most recent articles:

Uber’s AI adoption numbers. The Dev Platform folks at Uber have been kind enough to share the latest up-to-date numbers on AI adoption, following Tuesday’s article, which reported that 31% of all code is AI-authored. It turns out this was incorrect, due to a bug with one of the tools. Here’s how things look there:

• 84% of devs at Uber are agentic coding users (either CLI-based agents, or making more agentic requests than tab-completion in the IDE).

• 65-72% of code is AI-generated inside IDE-based tools. For AI command line tools like Claude Code the figure is, naturally, 100%.

• Claude Code usage almost doubled in 3 months; from 32% last December, to 63% by February. Meanwhile, IDE-based tool usage (Cursor, IntelliJ) have plateaued.

Separately, last week’s edition of The Pulse reported that Block did not make job cuts between 2022 and 2025, which was incorrect. Layoffs happened in Jan 2024 and March 2025. I have updated my analysis with these details; apologies for the error.

Today, we cover:

1. Staff+ engineers in 2027 and beyond. What happens to the Staff engineer role when agents write more code? Actually, they could be more in demand than ever!

2. New trend? AI token costs are a rising concern for CTOs. Accounts from two engineering leaders who are raising the alarm about steeply climbing AI costs and the need to slow down spending.

3. 10% layoffs at Atlassian: is it AI’s fault? Atlassian says it wants to invest savings in AI, but is there more to it?

4. Industry Pulse. An AI-powered library reimplementation sparks copyleft licensing debate, Anthropic launches $15–25 per-review code reviews, Microsoft ships a Claude-powered, Copilot Cowork clone, and Apple is the lone Big Tech not ramping up AI infrastructure spending.

1. Staff+ engineers in 2027 and beyond

At a recent two-day workshop in Utah, US, named ‘The Future of Software Development’, and organized by Martin Fowler, I was among 50 attendees. We self-organized our own sessions, where everyone could suggest a topic close to their heart, which all went on an agenda:

Read more

Listen and watch now on YouTube, Spotify, and Apple. See the episode transcript at the top of this page, and timestamps for the episode at the bottom.

Brought to You by

• Statsig – ⁠ The unified platform for flags, analytics, experiments, and more. Stop switching between different tools, and have them all in one place. Check out Statsig

• Sonar – The makers of SonarQube, the industry standard for automated code review. Sonar helps reduce outages, improve security, and lower risks associated with AI and agentic coding. See how Sonar is empowering the Agent Centric Development Cycle with new products and capabilities that strengthen the guide, verify, and solve phases of development.

• WorkOS – Everything you need to make your app enterprise ready. Skip the rebuild for enterprise features. Keep shipping. Visit WorkOS.com.

In this episode

Before we start: we recorded this episode with Steve, in Utah, early February, when both of us attended Marin Fowler’s The Future of Software Development workshop. Unfortunately, the audio recording for the episode turned out to be of poor quality, so I published a write-up of this conversation one month ago, as a deepdive: Steve Yegge on AI Agents and the Future of Software Engineering.

The article captured the essence of what Steve shared, but it felt like a shame not to be able to share the conversation, and just how animated and excited Steve got when talking about the software engineering craft. Thanks to the help of software engineer Tatsuhiko Miyagawa and audio post-production software Auphonic, we managed to fix all audio issues, and you can enjoy the full episode, including parts that the deepdive omitted.

Steve Yegge has spent decades writing software and thinking about how the craft evolves. From his early years at Amazon and Google, to his influential blog posts, he has often been early at spotting shifts in how software gets built.

In this episode of Pragmatic Engineer, I talk with Steve about how AI is changing engineering work, why he believes coding by hand may gradually disappear, and what developers should focus on, instead. We discuss his latest book, Vibe Coding, and the open-source AI agent orchestrator he built called Gas Town, which he said most devs should avoid using.

Steve shares his framework for levels of AI adoption by engineers, ranging from avoiding AI tools entirely, to running multiple agents in parallel. We discuss why he believes the knowledge that engineers need to know keeps changing, and why understanding how systems evolve may matter more than mastering any particular tool.

We also explore broader implications. Steve argues that AI’s role is not primarily to replace engineers, but to amplify them. At the same time, he warns that the pace of change will create new kinds of technical debt, new productivity pressures, and fresh challenges for how teams operate.

Key observation from Steve

My 9 observations from this episode:

1. A prototype-as-product model is replacing the build-then-dump cycle. At Anthropic, Steve says teams create many prototypes rapidly and just ship the best one. Claude Cowork reportedly went from prototype to launch in just 10 days. Meanwhile, “slot machine programming” – building 20 implementations and picking the winner – is becoming normal practice for teams.

2. The IDE could be evolving into a conversation and monitoring interface, not a code editor. Steve sees tools like Claude Cowork as the return of the IDE, focused on managing agent workflows above coding by hand. He predicts these new IDEs will focus on conversations with AI agents and monitoring them. Side note: I’m not sure I foresee conversational tools appearing just yet, or IDEs turning into such tools – but we do see tools like Claude Code being wildly popular among devs, as per our latest AI tooling survey.

3. Reading ability is becoming a blocker for wider AI adoption. Some struggle with walls of text that current AI tools produce, and Steve predicts that in the very near future, most people will program by talking to a visual avatar, not reading terminal output because he observes that five paragraphs is already a lot to read for many devs.

4. AI coding has a spectrum, and most engineers trend near the bottom. Steve describes eight levels, from “no AI” to “multi-agent orchestration,” with most engineers currently at levels 1–2: asking an IDE for suggestions and carefully reviewing output. He suspects such engineers will be left behind.

5. Monolithic codebases are a big blocker to AI adoption in enterprises. AI agents have a ceiling of between roughly half a million to a few million lines of code which they can work with, effectively. If your codebase is a monolith that won’t fit in a context window, AI agents won’t work well with it.

6. What software engineers need to know keeps changing. In the 1990s, any decent software engineer knew Assembly, and today almost no decent developer knows it because Assembly has long been superseded by technical progress. What engineers “need” to know these days is different from the ‘90s and that process continues with AI, changing the parts of the craft that are essential for devs. We grumble about this but that won’t change anything by itself.

7. SaaS companies that don’t offer platforms and APIs will be out-competed. Steve uses Zendesk as an example: if your product doesn’t expose APIs, then AI-native companies will just build bespoke replacements. “If Zendesk doesn’t make themselves a platform, then they’ll put themselves out of existence.”

8. There’s a “Dracula Effect” where AI-augmented work drains engineers faster than traditional work because AI automates the easy tasks, meaning that engineers are stuck doing high-intensity thinking all day. Steve says you may only get three daily productive hours at max speed, but during that time, you could produce 100x more output than before.

9. Even if AI progress stalls, it’s worthwhile getting proficient at working with parallel agents. Steve argues that since there’s a model as capable as Opus 4.5 is, we don’t need smarter models but better orchestration layers. The worst outcome for someone who invests in learning AI tools is that they gain a skill set that stays useful, whether the models improve or not!

The Pragmatic Engineer deepdives relevant for this episode

• Steve Yegge on AI agents and the future of software engineering

• Vibe coding as a software engineer

• The full circle of developer productivity with Steve Yegge

• AI Tooling for Software Engineers in 2026

• The AI Engineering Stack

Timestamps

(00:00) Intro

(01:43) Steve’s latest projects

(02:27) Important blog posts

(04:48) Shifts in what engineers need to know

(10:46) Steve’s current AI stance

(13:23) Steve’s book Vibe Coding

(18:25) Layoffs and disruption in tech

(31:13) Gas Town

(40:10) New ways of working

(51:08) The problem of too many people

(54:45) Why AI results lag in business

(59:57) Gamification and product stickiness

(1:04:54) The ‘Bitter Lesson’ explained

(1:07:14) The future of software development

(1:23:06) Where languages stand

(1:24:47) Adapting to change

(1:27:32) Steve’s predictions

References

Where to find Steve Yegge:

• X: https://x.com/steve_yegge

• LinkedIn: https://www.linkedin.com/in/steveyegge

• Website: https://steve-yegge.blogspot.com/

• Vibe Coding: https://itrevolution.com/product/vibe-coding-book

Mentions during the episode:

• Steve Yegge on AI Agents and the Future of Software Engineering: https://newsletter.pragmaticengineer.com/p/steve-yegge-on-ai-agents-and-the

• Amazon, Google and Vibe Coding with Steve Yegge: https://newsletter.pragmaticengineer.com/p/amazon-google-and-vibe-coding-with

• Gas Town: https://github.com/steveyegge/gastown

• Execution in the Kingdom of Nouns: https://steve-yegge.blogspot.com/2006/03/execution-in-kingdom-of-nouns.html

• Rich programmer food: https://steve-yegge.blogspot.com/2007/06/rich-programmer-food.html

• Wyvern on Steam: https://store.steampowered.com/app/1541710/Wyvern

• Claude Code: https://claude.com/product/claude-code

• The death of the junior developer: https://sourcegraph.com/blog/the-death-of-the-junior-developer

• Erik Meijer on X: https://x.com/headinthebox

• Andy Jassy: https://en.wikipedia.org/wiki/Andy_Jassy

• The Death of the IDE? w/ Steve Yegge & Nathan Sobo:

• Nathan Sobo on LinkedIn: https://www.linkedin.com/in/nathan-sobo-92b46720

• Claude Cowork: https://claude.com/product/cowork

• Craft Agents: https://agents.craft.do

• Git Gui: https://git-scm.com/tools/guis

• Fantasia: https://en.wikipedia.org/wiki/Fantasia_(1940_film)

• Welcome to Gas Town: https://steve-yegge.medium.com/welcome-to-gas-town-4f25ee16dd04

• Dolt: https://www.dolthub.com

• The Bitter Lesson: http://www.incompleteideas.net/IncIdeas/BitterLesson.html

• Moltbook: https://www.moltbook.com

• The ‘9-9-6 Work Schedule’ Could Be Coming To Your Workplace Soon: https://www.forbes.com/sites/bryanrobinson/2025/08/04/the-9-9-6-work-schedule-could-be-coming-to-your-workplace-soon

• Perl: https://www.perl.org

• PHP: https://www.php.net

• Anthropic: https://www.anthropic.com

• Purely Functional Data Structures: https://www.amazon.com/Purely-Functional-Data-Structures-Okasaki/dp/0521663504

• Building Claude Code with Boris Cherny: https://newsletter.pragmaticengineer.com/p/building-claude-code-with-boris-cherny

• Gemini: https://gemini.google.com

• The Borderlands Gun Collector’s Club: https://steve-yegge.blogspot.com/2012/03/borderlands-gun-collectors-club.html

• Ralph Wiggum as a “software engineer”: https://ghuntley.com/ralph

—

Production and marketing by Pen Name.

Update on 11 March: the Uber team shared updated numbers as of March 2026:

• 84% of devs at Uber are agentic coding users (either using CLI-based agents or making more agentic requests than tab-completion in their IDE)

• 65-72% of code is AI-generated inside IDE-based tools. This number is, naturally, 100% for AI command line tools like Claude Code.

• Claude Code usage nearly doubled in 3 months — from 32% in Dec to 63% in Feb, while IDE-based tools (Cursor, IntelliJ) have plateaued.

The article is updated with these new and correct numbers.

I spent four years working at Uber until 2020 and experienced firsthand the company’s standout engineering culture. Uber is a company that did the speed run of going from a small startup, through hypergrowth, to being a large company facing major risk during the pandemic, when the rideshare business briefly collapsed. Today, it’s maturing as a publicly traded, profitable company, and employs almost 3,000 people in the tech function.

At the recent Pragmatic Summit in San Francisco, one of the most interesting behind-the-scenes sessions came from the ridesharing company’s principal engineer, Ty Smith, and director of engineering Anshu Chada, who pulled back the curtain on what Uber has been doing with AI tools, internally. They were candid about the amount of work it took to build up Uber’s internal “AI stack,” why all that work was necessary, and also discussed the drawbacks as well as benefits of this rapidly spreading technology. You can watch their presentation at The Pragmatic Summit here.

In today’s issue, we cover:

1. Agentic layers & systems. Four layers spanning an internal AI platform, context sources, industry tools, and specialized agents for testing and code review.

2. Internal tooling: MCP Gateway, Uber Agent Builder, and the AIFX CLI. Uber built several internal tools to make it easier for devs to use AI tools, and to make internal AI agents more effective.

3. How AI changes developer workflows. A move away from single-threaded coding in an IDE, to orchestrating multiple parallel agents. Engineers naturally gravitate toward kicking off new agents, which starts to create resource and cost challenges.

4. Minion: running background agents at scale. Uber built Minion, an internal background agent platform with monorepo access and optimized defaults. It’s a clever abstraction layer that works well in practice.

5. New internal dev tools. More AI-generated code means more code reviews and more noise, so Uber built Code Inbox for smart PR routing, uReview for high-signal AI code review comments, Autocover for generating 5,000+ unit tests per month, and Shepherd for managing large-scale migrations end to end.

6. Challenges. AI adoption is slower than expected, even at a forward-thinking company like Uber. Top-down mandates are less efficient than engineers sharing their wins with peers.

7. Impact in numbers. 92% of Uber devs use agents monthly, 65-72% of code is AI-generated inside IDEs, and 11% of pull requests opened by agents. At the same time, AI-related costs are up 6x since 2024, and token cost optimization is a growing priority.

Longtime readers might recall we’ve covered Uber’s engineering culture over time:

• Developer Experience at Uber – with Uber’s founding engineer on Developer Platform, Gautam Korlam (2025)

• How Uber built its observability platform (2023)

• Inside Uber’s move to the Cloud (2023)

• How Uber is measuring engineering productivity (2022)

• The Program and Platform split at Uber (2021)

Let’s get into it:

AI is not new at Uber, but rolling it out companywide is. The company has used machine learning and AI technologies in many systems, including its Marketplace platform, which are responsible for routing and matching drivers with riders, forecasting demand, etc. What is relatively new at nearly all tech companies is the process of integrating AI across engineering and beyond.

The official strategy at the ridesharing giant is to become a “GenAI-powered” company:

I appreciate Uber sharing this approach openly because while most companies say that they want to be “AI-powered” – however cliche that claim might be – not all provide as much transparency.

It’s worthwhile engineers internalizing how leadership views AI. These folks, in general, see a tool that can bring efficiency everywhere. My take is that in some ways, AI is seen similarly to the cloud, which has been perceived as a means to reduce costs and improve the flexibility and elasticity of hardware resources. Today, AI is seen as the way to increase efficiency and lower costs, such as customer support, software development, the finance function – or any function.

Uber is focusing not on automating everything possible in engineering. Instead, it wants to:

1. Eliminate toil: helping AI do “boring” work like upgrades, migrations, trivial bug fixes, etc.

2. Free up engineers to focus more on creative work.

As Anshu Chada, Engineering Director on Uber’s Dev Platform, puts it:

“What we found is when we push some of the boring stuff to AI – upgrades, migrations, bug fixes – not only does it result in much higher satisfaction from our engineers, but they’re able to push our product and create features for end users in ways that we didn’t even think were possible.”

1. Agentic layers & systems

Uber’s “agentic system” for software engineering is actually made up of several systems:

Categories of systems:

• Internal AI platform. Built on top of Michelangelo, Uber’s ML/AI platform. This layer provides things like a model gateway to proxy to frontier models or internally hosted models.

• Internal Uber context: accessing Uber’s source code, engineering documentation, Slack information, JIRA tickets, etc. These all serve as “memory” for agents to use.

• Industry agents: Uber’s approach is to enable the “latest & greatest” AI agents for engineers, so they support several tools like Claude Code, GitHub Copilot, Codex, and other clients.

• Specialized agents: Uber’s background agent platform, the test generation platform, code review agents, and more.

• Engineering enablement: measuring the efficiency of agents, controlling costs, and educating engineers about which tools to use.

2. Internal tooling: MCP Gateway, Uber Agent Builder, and the AIFX CLI

MCP – the Model Context Protocol – has quickly become the standard way to connect agents and data sources with one another. A frequent analogy is that MCP is like the “USB-C interface for AI agents.” We published a deepdive on the MCP protocol and covered real-world MCP server use cases.

Uber put together a “tiger team” (a temporary unit that gets things done fast) to design the MCP strategy and build the central MCP gateway, which looks like this:

This MCP gateway allows:

• Proxy internal endpoints to MCPs: any internal Thrift, Protobuffer, or HTTP endpoint can be exposed as an MCP server with a simple configuration change. Uber uses the Apache Thrift protocol and Protobuffer protocols extensively for backend service communications

• First-party MCPs: these are exposed as a single, consistent interface

• Third-party MCPs: external MCP servers are also exposed via the gateway, which handles all authentication and authorization tasks.

• Platform concerns: the gateway takes care of authorization, telemetry, and logging in one central place. Plus, it conveniently offers a unified interface to interact with any MCP.

The MCP gateway also provides:

• A registry: to look up MCP servers, and for devs to be able to register their own.

• A sandbox: for devs to experiment with MCP servers without long-winded setups.

Uber Agent Builder

Uber’s Agent Builder product is a no-code solution to build agents that can access Uber’s internal data sources (both MCP servers and Uber data sets), and hand off work to other agents:

The platform includes a tool called Agent Studio, where multi-agent workflows can be visualized, debugged, traced, versioned, and evaluated. This is how it looks:

The agents built in Agent Builder become discoverable through a registry:

Accessing agents via the AIFX CLI

Uber’s Developer Experience platform team had a few issues with deploying AI agent tooling at scale:

• How does the company update all clients when a new version comes out? For example, if a new version of Cursor is released, how can they ensure all devs use the latest one?

• How can the clients be configured with helpful defaults? Uber’s Dev Platform team might have found better defaults for tools, like more helpful models. How are these rolled out to all devs?

• How can devs easily discover MCP servers and configure them for agents?

• How can agents connect to Uber’s background task infrastructure?

The Dev Experience team built the AIFX CLI, which is the AI tooling command line used by all engineers there. Here’s what it looks like:

This tool supports:

• Provisioning AI agents (client tools like Claude Code, Codex, Cursor, and others)

• Finding and using MCP servers

• Running background agent tasks

• Updating AI agents and clients to the latest versions

3. How AI changes developer workflows

The traditional way of building software:

Devs spent some time planning, most time writing code (code authorship), and then some time in code review.

The first AI agent-based workflows were single-threaded: devs worked with a single agent in the command line or inside their IDE:

At Uber, the latest workflows which many software engineers follow are pretty different, involving parallel agents, each kicked off with their own tasks:

As Ty explained, running multiple agents comes naturally to most devs:

“[Once you start using agents] as an engineer, you’re giving a prompt and waiting for something. While it’s running and you’re waiting you’re thinking: ‘What am I going to do? Have a coffee or browse Reddit? Might as well kick off another background agent.’

And so, engineers get into this mode of running several agents at once, right? Both us [at Uber] and a lot of the industry is trying to push towards this.”

4. Minion: running background agents at scale

Read more

Today, we cover:

1. Military strikes take down cloud services for the first time. Drone attacks in Bahrain and the UAE took AWS datacenters partially or fully offline during the early days of the Iran…

Read more

Listen and watch now on YouTube, Spotify, and Apple. See the episode transcript at the top of this page, and timestamps for the episode at the bottom.

Brought to You by

• Statsig — ⁠ The unified platform for flags, analytics, experiments, and more. Stop switching between different tools, and have them all in one place. Check out Statsig.

• Sonar – The makers of SonarQube, the industry standard for automated code review. Proactively find and fix issues in real-time with SonarQube MCP Server by connecting AI tools like Claude Code, GitHub Copilot, and Cursor directly to Sonar’s systematic code analysis engine. Learn how Sonar provides the essential verification layer for the AI development era.

• WorkOS – Everything you need to make your app enterprise ready. Companies like Anthropic, OpenAI, and Cursor already run on WorkOS. See why.

In this episode

Boris Cherny is the creator and Head of Claude Code at Anthropic. He previously spent five years at Meta as a Principal Engineer and is the author of the book Programming TypeScript.

In this episode of Pragmatic Engineer, we went through how Claude Code was built and what it means when engineers no longer write most of the code themselves.

We discuss how Claude Code evolved from a side project into a core internal tool at Anthropic and how Boris uses it day-to-day. We go deep into workflow details, including parallel agents, PR structure, deterministic review patterns, and how the system retrieves context from large codebases. We also get into how Claude Cowork was built.

As coding becomes more accessible, the role of engineers shifts rather than shrinks. We examine what that shift means in practice, which skills become more important, and why the lines between product, engineering, and design are blurring.

Key observation from Boris

Here are 10 observations from this conversation that I found the most interesting:

1. Boris ships 20-30 PRs a day by running 5 parallel Claude instances. Boris works across five terminal tabs (each a separate checkout), starting Claude in plan mode, iterating on the plan, then letting it one-shot the implementation. As he put it: “once there is a good plan, it will one-shot the implementation almost every time.”

2. Boris learned at Meta that code quality has a measurable, double-digit-percent impact on engineering productivity. At Meta, Boris led causal analysis showing clean codebases meaningfully boost output. This learning holds for AI-generated code, too: partially-migrated codebases with multiple frameworks confuse both humans and models. As Boris put it: “always make sure that when you start a migration, you finish the migration.”

3. Claude Code’s “agentic search” is really just glob and grep, and it outperformed RAG. The team tried several approaches to make agentic search better: local vector databases, recursive model-based indexing, and other fancy approaches. All had downsides (stale indexes, permission complexity). Plain glob and grep, driven by the model, beat everything. This approach was inspired by how Boris observed engineers at Instagram searched code when the click-to-definition functionality in Meta’s in-house coding editor was broken.

4. Boris automated himself out of code review well before AI. Boris was one of the most prolific code reviewers at Meta company. And he worked hard to minimize time spent on code review. His system: every time he left the same kind of review comment, he logged it in a spreadsheet. Once a pattern hit 3-4 occurrences, he’d write a lint rule to automate it away!

5. Everyone at Anthropic has the same title (“Member of Technical Staff”), and it’s by design. Without role-specific titles, the default assumption is that everyone does everything: product, design, infrastructure, research. As Boris put it: “It kind of inverts this relationship between people, even if you don’t know each other well yet.”

6. Claude Cowork was built in ~10 days and is growing faster than Claude Code did at launch. The team spotted “latent demand” from non-engineers already hacking with Claude Code (data scientists, finance, sales). The bulk of the engineering complexity for Cowork wasn’t product logic, but around safety: building classifiers, a shipping VM, OS-level protections against accidental file deletion, and rethinking the permission model for non-technical users.

7. PRDs are dead on the Claude Code team: prototypes replaced them. Instead of writing Product Requirement Documents (specs), they build dozens of working prototypes before shipping a feature. Boris: “There’s just no way we could have shipped this if we started with static mocks and Figma or if we started with a PRD.”

8. This is the year of the generalist (and maybe the year of those with ADHD). Boris’s work has shifted from deep-focus single-threaded coding to managing multiple parallel agents and context-switching rapidly. As Boris put it: “It’s not so much about deep work, it’s about how good I am at context switching and jumping across multiple different contexts very quickly.”

9. Fix infra before building a product. Working at Meta, when Boris moved to Instagram, the Python/Django stack was so poor that click-to-definition didn’t work and the type checker was broken. He abandoned his original team, and went straight to Dev Infra, leading migrations from Python to the Facebook monolith and from REST to GraphQL. He did this because you can’t build great products on a terrible foundation, and sometimes the highest-leverage move is fixing the platform rather than shipping features.

10. Could software engineers of today be the medieval equivalents of scribes? Boris brought up an interesting analogy: in the middle ages, scribes were a tiny literate elite employed by often-illiterate kings. When the printing press was invented, scribes technically lost their jobs. Still, many of them became writers and authors, and the market for written work expanded beyond prediction!

Boris wondered if we could see the same pattern with software engineers: coding is becoming accessible to everyone. Could the software engineers of today be building systems that have far broader reach, in the future, than ever than before?

The Pragmatic Engineer deepdives relevant for this episode

• How Claude Code is built

• How Anthropic built Artifacts

• How Codex is built

• Real-world engineering challenges: building Cursor

Timestamps

(00:00) Intro

(11:15) Lessons from Meta

(19:46) Joining Anthropic

(23:08) The origins of Claude Code

(32:55) Boris’s Claude Code workflow

(36:27) Parallel agents

(40:25) Code reviews

(47:18) Claude Code’s architecture

(52:38) Permissions and sandboxing

(55:05) Engineering culture at Anthropic

(1:05:15) Claude Cowork

(1:12:48) Observability and privacy

(1:14:45) Agent swarms

(1:21:16) LLMs and the printing press analogy

(1:30:16) Standout engineer archetypes

(1:32:12) What skills still matter for engineers

(1:35:24) Book recommendations

References

Where to find Boris Cherny:

• X: https://x.com/bcherny

• LinkedIn: https://www.linkedin.com/in/bcherny

• Website: https://borischerny.com

Mentions during the episode:

• Jarred Sumner on LinkedIn: https://www.linkedin.com/in/jarred-sumner-a8772425

• Vladimir Kolesnikov on LinkedIn: https://www.linkedin.com/in/voloko

• React: https://react.dev

• Will Bailey on LinkedIn: https://www.linkedin.com/in/willbailey

• Fiona Fung on LinkedIn: https://www.linkedin.com/in/fionafung

• Inside Meta’s Engineering Culture: Part 1: https://newsletter.pragmaticengineer.com/p/facebook

• Inside Meta’s Engineering Culture: Part 2: https://newsletter.pragmaticengineer.com/p/facebook-2

• Adam Wolff on LinkedIn: https://www.linkedin.com/in/adamwolff

• The Bitter Lesson: http://www.incompleteideas.net/IncIdeas/BitterLesson.html

• How Claude Code is built: https://newsletter.pragmaticengineer.com/p/how-claude-code-is-built

• Mike Krieger on X: https://x.com/mikeyk

• Dario Amodei on X: https://x.com/DarioAmodei

• Ben Mann on LinkedIn: https://www.linkedin.com/in/benjamin-mann/

• Marc Andreessen: The real AI boom hasn’t even started yet:

Lenny's Newsletter

Marc Andreessen: The real AI boom hasn’t even started yet

Marc Andreessen is a founder, investor, and co-founder of Netscape, as well as co-founder of the venture capital firm Andreessen Horowitz (a16z). In this conversation, we dig into why we’re living through a unique and one of the most incredible times in history, and what comes next…

Listen now

3 months ago · 141 likes · 5 comments · Lenny Rachitsky

• Cat Wu on LinkedIn: https://www.linkedin.com/in/cat-wu

• Asana: https://asana.com

• Claude Cowork: https://claude.com/product/cowork

• Felix Rieseberg on LinkedIn: https://www.linkedin.com/in/felixrieseberg

• Electron: https://www.electronjs.org

• Building a C compiler with a team of parallel Claudes: https://www.anthropic.com/engineering/building-c-compiler

• Introducing Claude Opus 4.6: https://www.anthropic.com/news/claude-opus-4-6

• Thariq Shihipar on LinkedIn: https://www.linkedin.com/in/thariqshihipar

• Programming TypeScript: Making Your JavaScript Applications Scale: https://www.amazon.com/Programming-TypeScript-Making-JavaScript-Applications/dp/1492037656

• A General Theory of Reactivity: https://kriskowal.gitbooks.io/gtor/content/intro.html

• Ryan Dahl on X: https://x.com/rough__sea

• Anders Hejlsberg on X: https://x.com/ahejlsberg

• The Three-Body Problem: https://www.amazon.com/Three-Body-Problem-Cixin-Liu/dp/0765382032

• Accelerando: https://www.amazon.com/Accelerando-Singularity-Charles-Stross/dp/0441014151

• Functional Programming in Scala: https://www.amazon.com/Functional-Programming-Second-Michael-Pilquist-ebook/dp/B0C4LX999T

—

Production and marketing by Pen Name.

This article provides a high-level overview of those findings from our latest AI tooling survey. Thank you to everyone who participated!

There are plenty of interesting details, most notably, validation of just how much Anthropic and Claude Code have risen to dominate tooling usage. Claude Code is today nearly as widespread as GitHub Copilot was in our survey three years ago (in the spring of 2023) – which shows how fast the AI market moves.

In today’s issue, we cover:

1. Interesting findings: Claude Code has rocketed to #1 in just eight months, AI is fully mainstream with 95% weekly usage among respondents, and more.

2. Most-used AI tools: Claude Code leads the pack, followed by chatbots and GitHub Copilot. Cursor’s rising fast, and newcomers like Codex and Antigravity are gaining traction. Most engineers juggle two to four tools at once.

3. Popular models: Anthropic’s Opus and Sonnet models dominate coding tasks by a wide margin, with more mentions than all others combined.

4. AI trends: mainstream adoption achieved. 95% of respondents report using AI tools at least weekly, 75% use AI for half or more of their work, and 56% report doing 70%+ of their engineering work with AI.

5. AI agent usage rising: 55% of respondents now regularly use AI agents, with staff+ engineers leading adoption on 63.5% usage in the survey results. Agent users are twice as excited about AI as non-users are.

6. Company size and tool usage: smaller places overwhelmingly favor Claude Code (75% at the tiniest businesses), while large enterprises default to GitHub Copilot. This popularity is probably down to enterprise procurement preferences – and Microsoft’s enterprise marketing efforts.

7. Tools engineers love: Claude Code is the most loved tool at 46%, far ahead of Cursor on 19% and GitHub Copilot at 9%. Senior leaders are especially enthusiastic about Claude Code.

8. Demographics: Overview of who took part in the survey.

Full subscribers also have access to a longer, 35-page report with additional details - linked at the end of this article.

1. Interesting findings

Here are my ten personal, most-interesting findings from this survey:

1. Claude Code has gone from zero to be the #1 tool in only eight months. Released in May 2025, it’s already the most-used AI coding tool, overtaking GitHub Copilot and Cursor.

2. AI is now mainstream. 95% of respondents use AI tools at least weekly, or more often, and 75% use AI for at least half their software engineering work. Among readers of The Pragmatic Engineer, it seems the question is no longer whether to use AI in day-to-day work, but which tools to use.

3. Cursor is catching up fast on GitHub Copilot. As much as we hear about companies dropping Cursor for Claude Code, Cursor is doing more than fine, growing in mentions 35% since our previous survey nine months ago.

4. Most engineers juggle multiple AI tools. 70% use between two and four tools simultaneously, while 15% use five or more.

5. Staff+ engineers are the heaviest agent users. 63.5% use agents regularly; more than regular engineers (49.7%), engineering managers (46.1%), and directors/VPs (51.9%).

6. Codex is seeing explosive early growth. Despite not existing during the last survey, OpenAI’s Codex already has 60% of Cursor’s usage (!!)

7. Using agents correlates strongly with being positive about AI. People using agents are nearly twice as likely to feel excited about AI while non-users are twice as likely to be skeptical.

8. Company size influences tool choice more than preference. Huge companies (10K+) more likely to use Copilot (56%), tiny startups mostly go with Claude Code (75%) and Cursor (42%). It seems like enterprise procurement, not individual preference, is behind this divergence.

9. A tight chatbot race. ChatGPT, Gemini, and Claude as standalone chatbots have nearly equal numbers of mentions, suggesting there’s no clear winner outside of coding-specific tools among software engineers.

10. Directors and senior leaders are especially into Claude Code. The survey finds this tool is twice as popular with these folks as it is at less senior levels, while Cursor gets less love as seniority increases.

Let’s jump into some of the data:

2. Most-used AI tools

Just eight months after its release, Claude Code is already the most-used tool, overtaking both GitHub Copilot and Cursor:

Tools mentioned, in order of popularity:

1. Claude Code: a terminal-first coding agent from Anthropic. We cover its history in the deepdive, How Claude Code is built

2. Chatbots: ChatGPT, Claude, Gemini, and others

3. GitHub Copilot: launched in 2021, it’s the “oldest” AI coding tool on this list

4. Cursor: agent-powered coding IDE. Learn more about it in our deepdive, Real-world engineering challenges: building Cursor.

5. Codex: OpenAI’s AI coding agent. We recently covered how Codex is built

6. Gemini CLI: Google’s command line agent

7. OpenCode: the most popular open source coding agent, where you can swap out the model being used, sidestepping vendor lock-in

8. Antigravity: Google hired the original team behind Windsurf and launched its own agentic IDE

9. JetBrains Junie: the AI coding agent by JetBrains

10. Zed: a fast editor with agentic workflows

11. Windsurf: when Google acquihired the team, it did not buy the product. An AI IDE, now owned and operated by Cognition

12. Amp: a model-agnostic coding agent. A free version is supported by ads

13. Augment Code: an enterprise-focused coding agent

14. Factory: agent-native software development, which calls agents “droids.”

It’s interesting to compare how people answered the same question just nine months ago, last May:

Notable trends:

• Claude Code has massive momentum and is already the market leader. Since being released in May 2025, it has become the most-used AI coding tool among survey respondents. Anecdotally, I’ve heard several teams that have been Cursor power users are introducing Claude Code with great success (while still keeping Cursor in-place.)

• Cursor has grown circa 35% in nine months and now threatens GitHub’s popularity. In this survey, 35% more respondents mention using Cursor than in our previous research, last May. Such growth is impressive: at this rate, Cursor will have more users than GitHub in 6-9 months!

• Chatbot usage remains high. Combined mentions for ChatGPT, Claude, Gemini, Perplexity, and others used outside of coding apps are still higher than for any other tool except Claude Code. For context, the most-mentioned chatbot (ChatGPT, 107 times) only has as many as Gemini CLI (also 107).

• GitHub Copilot usage stable. Nine months ago, 46% of respondents said they used GitHub Copilot. Since then, it’s barely risen.

• Explosive growth for Codex. OpenAI’s Codex wasn’t available during our last survey. But it now already has 60% of Cursor’s usage(!), and could be growing even faster since this survey.

• Up-and-coming tools: OpenCode, Gemini CLI, Antigravity. None of these had launched nine months ago, but today they’re used by around 10% of respondents – no small feat!

• Tools growing more than before: The likes of Zed, Windsurf, Amp, Augment Code, and Factory appear less often in these results, but all are mentioned more by respondents than they were nine months ago.

Chatbot usage: ChatGPT leads – just about

Here’s how mentions of chatbot usage line up in our survey:

Tool usage: multi-tool

Most tech professionals use between two and four AI tools. An interesting detail is how many different ones are mentioned by respondents:

Noteworthy details:

• 70% of survey participants mention using between two and four tools

• 15% use a single tool

• 15% are using 5 or more

3. Popular models

Anthropic’s Opus and Sonnet dominate the ranking of models used for coding.

This is not even a contest: Opus 4.5 and Sonnet 4.5 (latest models at the start of our survey) come up more often than all other models, combined. Anthropic has become the go-to model developer for coding-related work – for now, that is. When this survey launched, Opus 4.6, Sonnet 4.6, and GPT-5.3 were not yet out.

Around 1 in 8 respondents say they just use whatever model is the default at their company. This is interesting to note: these are likely folks who might not bother changing default settings, and just go with whatever’s available. If the default model is powerful enough, that’s fine, but if the company’s default is a cheaper, less capable model, then these people could face a more frustrating experience than those who get to choose what they use.

In the “other” category of models, some other mentions include:

• Cursor’s custom Composer/Composer-1 model, and its “Cursor Auto” auto-select model

• Kimi/Kimi K2.5 — Moonshot

• DeepSeek model variants like R1, V3.2, Coder

• Alibaba’s Qwen/Qwen3

• xAI’s Grok

• Various Mistral models

4. AI trends: Mainstream adoption achieved

How often do people use AI tools? Very often, as it turns out; 95% of respondents are using them weekly, at a minimum:

It’s worth reflecting on this data in relation to insights presented by Laura Tacho at the recent Pragmatic Summit in San Francisco:

It seems that AI is now mainstream in software engineering. Anecdotally, this has been my sense since the beginning of the year: everyone whom I talk with is using AI tooling on a roughly daily basis. Now, there’s data to prove it.

How much software engineering work gets done with AI?

This year, we asked readers to estimate the percentage of their software engineering work that’s done using AI. The results:

The data show that AI is embedded in the workflows of participants in our survey:

• Only 25% of respondents use AI for less than 40% of their work

• 56% do 70% or more of their work using AI

• 75% use AI for at least half of their software engineering work

5. AI agent usage rising

Eighteen months ago, AI usage was mostly for code generation and tab completion. There were one or two respondents who experimented with early AI agents in March 2024, as something equivalent to a junior software engineer.

This year, 55% say they regularly use AI agents. This is 507 people, a massive jump!

The most common use cases for agents:

• Code review and code validation

• Automating manual / annoying tasks

• Bug fixing / investigating bugs

• Code investigation

• Debugging

• “Crafting” code, or “weaving” it together with the agent

Below is a typical-enough comment from one software engineer who uses agents at a smaller company:

“I use agents for pretty much all coding work, mostly prompting with Cursor Chat. I use it for code investigation, bug investigation, creating commits and pull requests. It’s my tool for reviewing code, I am always still in the loop. Almost all of my AI-written code is still reviewed and ‘crafted’. When using it for code review, I find it helpful to chat with and gain understanding, rather than letting it loose on code review. So, I use it for everything, but I am still very much in the loop.”

A common arrangement in the survey is the split-screen setup: a terminal with Claude Code open to drive work, and an IDE also open to review changes made by the agent.

The tools mentioned by those who regularly use AI agents split almost identically to the broader survey results:

Staff+ engineers are the heaviest users of agents. Here’s the data on agent usage by experience level:

This data point is slightly amusing because it shows there’s not much difference between the lowest and highest levels: 46% of leads and engineering managers say they use AI agents regularly, while for Staff+ engineers it’s 63%. Does it suggest the most experienced engineers are also the most curious?

The more someone uses AI, the more they also use AI agents. We segmented the data by the percentage of software engineering work that respondents do with AI:

• Heavy users have AI for 80% or more tasks, or use AI on an hourly basis

• Moderate users have AI for 30-80% of tasks, or use AI on a daily basis

• Light users employ AI for 30% of tasks or less, or use AI on a weekly or monthly basis

The more positive someone is about AI, the more likely they are using agents on a regular basis. In contrast, those negative about AI barely use agents:

One question is whether this finding indicates correlation or causation: that is, does starting to use AI agents more, actually cause people to feel more positive about AI?

A couple of details:

• Using agents seems to make people nearly twice as enthusiastic about AI (61%) as those who do not use them (36%)

• Respondents who don’t use agents are twice as likely to be skeptical (22%) about AI than those using agents regularly (11%)

It looks like that if you don’t use AI agents on a regular basis, you may have a negative opinion about the tools, in general, which could come at the cost of not experiencing what the technology has to offer.

6. Company size and tool usage

In our results, company size and tooling choice correlate for some tools; for example, the smaller a team or company is, the more likely it is to use Claude Code or Codex:

Claude Code is used by a whopping 75% of the smallest companies and teams. This is a big number, far ahead of any other tools. At the smallest places, the most-used tools are, in order:

1. Claude Code: 75%

2. Chatbots: 55%

3. Cursor: 42%

4. GitHub Copilot: 35%

5. Codex: 26%

6. Gemini CLI: 14%

7. OpenCode: 13%

GitHub Copilot overtakes Claude Code at large companies. This confirms what was known: Microsoft is very good at enterprise sales, and at bundling GitHub Copilot in its suite of products:

Cursor and OpenCode usage drops at massive companies with similar usage patterns. Across the board, usage is roughly the same, regardless of company size. We only see a drop at the very largest of companies with 10,000+ employees:

One theory for such a drop at massive companies could be that large companies often build their own internal coding agents that engineers use; e.g., at fintech, Block, the agent is called Goose, Meta has its own agent, as does Google with Jetski (its version of Antigravity) and Cider.

Google’s tools are evenly used across the spectrum of company size. Google’s Gemini CLI and Antigravity are the only tools in this survey whose usage is notably stable, regardless of company size. Both tools hover at around 10% from the smallest to largest workplaces:

The feeling among survey respondents that they can experiment at work, correlates with Claude Code being available. We asked if readers experiment frequently with tools. Below are the “yes” responses by company size:

When these responses are mapped to the percentage of people using Claude Code, there’s a very similar distribution. My theory is that Claude Code is new enough at 9 months old to have not yet been approved at companies with bureaucratic processes for approving new tools, and this is partly why some respondents feel their chance to experiment with the range of tooling is being thwarted.

Engineers at places with lots of red tape for tooling are less empowered to experiment with new AI tools; not just Claude Code, but any new, interesting tool.

7. Tools engineers love

We asked respondents: “Which AI tools do you love using the most, and why?” Below are the tools which respondents enjoy most, in descending order of number of mentions:

Claude and Cursor stand out in terms of how much they are loved. A whopping 57% of respondents mention either Claude Code (46%) or Claude models (11%) as the tools they are attached to. Cursor was at a respectable 19% – double GitHub on 9%.

Other notable tools with two or more mentions:

• Warp: a terminal for building agents

• Zed: a fast editor with agentic workflows

• Amp: a model-agnostic coding agent. A free version is supported by ads

• Cline: an open source coding agent

• RooCode: an open source, AI-powered coding assistant running in VS Code

• Continue.dev: AI checks on every pull request

Claude Code is especially loved by Director-and-above folks. Segmenting the “most loved” responses by level (engineers up to the senior levels of staff+ engineers, leads/eng managers, and Director+ folks):

Both Claude Code and Cursor become less loved – or used! – as seniority goes up, but it’s notable that folks in senior engineering leadership positions are obsessed with Claude Code, but not Cursor.

GitHub Copilot is equally loved by engineering managers as Cursor is — and this is surprising:

Both OpenCode and GitHub Copilot are surprisingly loved by Staff+ engineers. Another unexpected detail is that, despite OpenCode being used about a quarter as much as GitHub Copilot, it rivals the “loved” mentions. For Staff+ engineers, it matches GitHub on those terms:

Finally, when segmenting based on company size, we see the familiar pattern of Claude Code vs GitHub Copilot. Claude Code is less frequently mentioned as a “loved” tool as company size increases.

GitHub Copilot sees the opposite trajectory: it’s more loved within larger companies where it’s likely to be harder to experiment with alternatives.

8. Demographics

In closing, below are details about who took the survey, and how the 906 responses came together.

Engineers comprise 55% of respondents, and engineering leadership another 34%:

Respondents are experienced professionals. The median respondent has 11-15 years of experience:

Company size is also a fairly even split across this group:

Region-wise, most respondents are in Europe or the US:

Takeaways and the full report

We have compiled additional findings from this survey which did not fit in this article: it’s a 35-page article:

Read more

There are several layers to dig into h…

Read more

Listen and watch now on YouTube, Spotify, and Apple. See the episode transcript at the top of this page, and timestamps for the episode at the bottom.

Brought to You by

• Statsig — ⁠ The unified platform for flags, analytics, experiments, and more. See why companies like Notion, Brex, and Atlassian use Statsig.

• Sonar – The makers of SonarQube, the industry standard for automated code review. Join me online at the Sonar Summit on March 3rd, where myself, other eng leaders and devs will share practical strategies for the AI era.

• WorkOS – Everything you need to make your app enterprise ready. If you’re building SaaS — especially an AI product — WorkOS provides all the enterprise building blocks you need. Check out WorkOS.

In this episode

How has the day-to-day workflow of Mitchell Hashimoto changed, thanks to AI tools?

Mitchell Hashimoto is one of the most influential infrastructure engineers of our time, and is one of the most pragmatic builders I’ve met. He is the co-founder of HashiCorp and creator of Ghostty. In this episode, we talk about how he got into software engineering, the history of HashiCorp, and the challenges of turning widely used open-source tools into a durable business. We also go into what it’s really like to work with AWS, Azure and GCP as a startup.

Mitchell shares how he uses AI these days, and how agents have completely changed how he works. We touch on Ghostty, open source, and what’s changing for software engineers and founders in an AI-native era.

Key observation from Mitchell

Here are 9 observations from this conversation that I found the most interesting:

1. Vagrant was created because dev environment setup was an unbillable time sink at a consultancy. At the Ruby on Rails shop where Mitchell worked, jumping onto another client’s project could waste half a day just setting up the environment, plus destroy your existing one. Vagrant solved this in a simple and elegant way, with VirtualBox VMs.

2. Terraform won, despite being 7th to market. Mitchell is frustrated by the narrative that Terraform won because it was first to market in infrastructure-as-code. “We were seventh to market... no one was a clear winner. It was a warring market.” Terraform won through relentless conference presence, community building, and a better developer experience — not timing.

3. HashiCorp had no real business for four years and their first commercial product was a full-on failure. The initial product, Atlas, required customers to adopt the entire HashiCorp stack, which created an unsolvable internal budget problem: no one org within a company would want to purchase the whole stack! HashiCorp pivoted to selling individual services like Vault, and this approach turned out to be a winner.

4. VMware almost bought HashiCorp for ~$100, and Terraform would have not happened if it did. When HashiCorp was only 3 people, VMWare made an initial offer of $20M for the company. Mitchell and Amon (the founders) asked for $100M as the number they would sell for. VMWare took this to their board, where they rejected to buy with a single vote. Mitchell said that Terraform probably never would’ve existed if the VMWare purchase went through.

5. Mitchell’s new rule for building software: always have an agent running in the background doing something. “If I’m coding, I want an agent planning. If they’re coding, I want to be reviewing.” He kicks off tasks before leaving the house — research, edge-case analysis, library comparisons — so work progresses while he drives or is away.

6. Open source is moving from “default trust” to “default deny” — and Mitchell thinks that’s how it should be. This is because AI makes it trivial to create plausible looking but incorrect and low-quality contributions. As he put it: “open source has always been a system of trust. Before, we’ve had default trust. Now it’s just default deny.”

7. Git and GitHub may not survive the agentic era in their current form. Agents cause so much churn that merge queues become untenable, branches proliferate, and repos balloon. Mitchell compares the needed shift to Gmail’s revolution for email: “We’re at the Gmail moment for version control... never delete, archive everything.”

8. The best engineers Mitchell ever hired had boring, invisible backgrounds. “They don’t have social media profiles... they’re honestly nine-to-five engineers. They go back and they don’t code at night.” No GitHub contributions, no public profiles, companies you’ve never heard of. “Every moment you spend on social media is taking away from something else... the best engineers are the ones that context-switch the least.”

9. Mitchell’s advice for AI-skeptical engineers: start by reproducing your research, not your code. “There’s a lot of people like, ‘I don’t want it to write code for me.’ But just delegate some of the research part.” He uses agents for library comparisons, edge-case analysis, and deep research — not just code generation. “You don’t need to pick up on the ‘it must replace you as a person’ kind of propaganda.”

The Pragmatic Engineer deepdives relevant for this episode

• AI Engineering in the real world

• The AI Engineering stack

• Pressure on commercial open source to make more money – and HashiCorp changing its license

• How Linux is built with Greg Kroah-Hartman

Timestamps

(00:00) Intro

(02:03) Mitchell’s path into software engineering

(07:19) The origins of HashiCorp

(15:52) Early cloud computing

(18:22) The 2010s startup scene in SF

(23:11) Funding HashiCorp

(25:23) The Hashi stack

(32:33) Why HashiCorp’s business lagged behind its technology

(35:28) An early failure in commercialization

(38:28) The open-core pivot and path to enterprise profitability

(48:08) Taking HashiCorp public

(51:58) The near VMware acquisition

(59:10) Mitchell’s take on all the cloud providers

(1:06:02) AI’s impact on open source

(1:07:00) Why Mitchell built Ghostty

(1:09:11) Why Mitchell used Zig

(1:10:38) How terminals work and Ghostty’s approach

(1:17:31) AI’s impact on terminals and libghostty

(1:19:13) How Mitchell uses AI

(1:22:02) Ghostty’s evolving AI use policy

(1:28:36) Why open source must change

(1:31:46) The problem of Git in monorepos

(1:36:22) What needs to change to work effectively with AI

(1:39:57) Mitchell’s hiring practices

(1:47:52) Mitchell’s AI adoption journey

(1:50:41) Advice to would-be founders

(1:52:21) Mitchell’s advising work

(1:53:20) What’s changing for software engineers

(1:55:03) How Mitchell recharges

(1:55:50) Book recommendation

References

Where to find Mitchell Hashimoto:

• X: https://x.com/mitchellh

• LinkedIn: linkedin.com/in/mitchellh

• Website: https://mitchellh.com

• Ghostty: https://ghostty.org

• How Mitchell adopted AI: https://mitchellh.com/writing/my-ai-adoption-journey

Mentions during the episode:

• HashiCorp: https://www.hashicorp.com

• PHP Manual: https://www.php.net/manual/en/index.php

• Ruby on Rails: https://rubyonrails.org

• Armon Dadgar on LinkedIn: https://www.linkedin.com/in/armon-dadgar

• Amazon S3: https://aws.amazon.com/pm/serv-s3

• How AWS S3 is built: https://newsletter.pragmaticengineer.com/p/how-aws-s3-is-built

• Vagrant: https://developer.hashicorp.com/vagrant

• Google App Engine: https://en.wikipedia.org/wiki/Google_App_Engine

• Packer: https://www.hashicorp.com/en/products/packer

• Consul: https://www.hashicorp.com/en/products/consul

• Kubernetes: https://kubernetes.io

• Terraform: https://www.hashicorp.com/en/products/terraform

• Vault: https://www.hashicorp.com/en/products/vault

• Nomad: https://www.hashicorp.com/en/products/nomad

• VMware: https://www.vmware.com

• Microsoft Azure: https://azure.microsoft.com

• Google Cloud: https://cloud.google.com

• Zig: https://ziglang.org

• Kitty: https://sw.kovidgoyal.net/kitty

• Alacritty: https://alacritty.org

• Mitchell’s post on X about improving rendering performance:

• Gastown: https://github.com/steveyegge/gastown

• Lobsters: https://lobste.rs

• Real-world engineering challenges: building Cursor: https://newsletter.pragmaticengineer.com/p/cursor

• My AI Adoption Journey: https://mitchellh.com/writing/my-ai-adoption-journey

• The Invisible Life of Addie LaRue: https://www.amazon.com/Invisible-Life-Addie-LaRue/dp/0765387565

—

Production and marketing by Pen Name.

It was well-timed, considering that right now seems like a period of change for tech that’s unfolding faster than before. That was a consensus opinion at both events, also held by veterans like Martin Fowler and Kent Beck, who said things haven’t shifted so rapidly during their 50+ years in the industry.

At the very start of this year, I predicted AI will write almost all code, going forward, and several others have said the same. But at the Pragmatic Summit, I met an embedded engineer writing Assembly and C code who is still writing more of his code by hand than with AI agents – and was the only person I spoke with in San Fran who was not yet “giving in” to AI agents.

Even so, today, this engineer has between a third and a half of their own low-level code being generated by AI agents since the launch of Opus 4.5, and this share keeps on rising. Their view was an interesting counterpoint to the prevailing trend.

This article shares some thought-provoking ideas and conversations from both events, covering:

1. Data vs hype: how orgs actually win with AI. Laura Tacho’s keynote at The Pragmatic Summit. Exclusive data reveals 92% of devs use AI coding tools monthly (!!), “unhealthy” orgs see 2x more incidents, healthy ones have 50% fewer, and other new data.

2. Building world-class engineering orgs in the AI era. The closing session of the summit dug into what AI-native teams look like, in a fireside chat with GitHub’s former CEO and the CTO of Atlassian.

3. The Future of Software Development with Martin Fowler. Laura Tacho, distinguished engineer Annie Vella, Martin Fowler, and myself, look back on the event in Utah. Nobody has the AI shift all worked out, which is reassuring!

4. Mid-level engineers’ quiet crisis. Something I heard that engineering leaders talk about behind closed doors a lot is that mid-career engineers are being left behind by the AI wave. New grads are more productive with the tools, while seniors have more of that all-important experience. Advice on how to catch up fast.

5. Future of refactoring in an AI-native world. Refactoring legacy code has long been an important tool for keeping software maintainable. With AI generating most code now, is there any point? One answer is “hell, yeah!”

6. Future of Agile. It was only fitting to consider the future of Agile on its 25th anniversary, in the company of two of the writers of the Agile Manifesto. Amusingly, one major trend is the return of Extreme Programming (XP) practices which predate Agile.

1. Data vs hype: how orgs actually win with AI

At the debut Pragmatic Summit in SF, Laura Tacho delivered the keynote as a well-known expert on developer productivity, and former CTO of DX. She presented new and exclusive data on AI usage, and what currently works with AI adoption. Her session was one of the most popular at the conference – and you can watch the whole thing here. Full subscribers also get access to Laura’s slides.

New data points on AI adoption shared at The Pragmatic Summit, provided by DX:

Usage:

Time management:

Onboarding impact:

There’s no typical experience with AI, said Laura:

“AI is extremely different in every company because each place has its own problems and its own culture.

Organizational performance is multidimensional and these organizations are just going off to different extremes, based on what they were doing before. AI is an accelerator, it’s a multiplier, and it is moving organizations in different directions.

Some organizations are facing twice as many customer-facing incidents. At the same time, some companies are also experiencing 50% fewer incidents. So, for some companies that have used AI and have a really healthy system, it has amplified that system. They are seeing fewer incidents, moving faster, accelerating with higher quality, higher code maintainability, and higher change competence.

On the other side, we have organizations which were dysfunctional already and are now more dysfunctional: they’re dysfunctional faster. Similarly, organizations are seeing really uneven results economically from using AI.

There are a lot of steep drop-offs when it comes to using AI in a pilot context with production, and then trying to tie it to profit. This is from an MIT study in July 2025, The Gen AI Divide”.

Good analogies for the age of AI might be the space race of the previous century and the age of exploration in the 15th-17th centuries, said Laura.

• Wonder of space: space exploration includes immense hope and grounded scepticism.

• Defining moments: even though the space race was incredibly expensive – or utterly wasteful – landing on the moon was a big moment for humanity.

• Delivering but not on the grandest ambitions: in the 1960s, going to the moon carried the expectation of human habitation on the lunar surface. As yet, that hasn’t happened but landing on the moon created progress back on Earth:

There’s a rare opportunity to channel the energy around AI into practical improvements, said Laura in closing remarks:

“I want to urge you to find that balance between a sense of wonder, aiming for Mars, and a moon colony, but also understanding we need to solve problems here on Earth and we have to live in this reality.

So, please stay grounded, stay skeptical, stay human. Most of all, stay pragmatic”.

Watch Laura's keynote

Access Laura's slides

2. Building world-class engineering orgs in the age of AI

Also at the Pragmatic Summit, Thomas Dohmke (founder & CEO of Entire, and former CEO of GitHub) and Rajeev Rajan (CTO of Atlassian) took part in a fireside chat to explore what a modern, world-class engineering team looks like today. Thomas is building a new, AI-native startup with Entire – an “AI-native GitHub” – while Atlassian has more data than anywhere else on developer productivity after acquiring DX.

Watch the closing session

A number of timely, thought-provoking topics were covered by the pair on stage:

Topic #1: What does an AI-native team look like?

Rajeev (Atlassian):

“It starts with the mindset. You have to really believe in doing AI-native work. Some teams at Atlassian have engineers basically writing zero lines of code: it’s all agents, or orchestration of agents. As a result, teams are not necessarily getting smaller, but they’re producing a lot more, sometimes 2-5x more, and creativity is up.

Efficiency framing is missing the point, it’s more about what you can create now with AI which you could not before”.

Thomas (Entire.io):

“AI native is the new ‘cloud native’ and what we are calling AI native now is going to be very different from AI native in a few years’ time.

There’s a lot of BS out there about how all day-to-day tasks are now ‘AI native’, and using agents for everything. I’m a startup founder: most of the time, I’m still dealing with things like old school HR school systems”.

Topic #2: AI makes being distributed an advantage again

Thomas:

“A remote team is kind of lonely, right? But agents basically gave us these sparring partners, these experts that I can just ask “bring some help in for this topic, I need to write an ADR or solve a problem.

This help has taken it to a new level for remote work. I now have my code review agent, my coding agent, my brainstorming agent, my research agents. In some ways, agents have given an advantage again for being remote-first.”

Rajeev:

“Agents have changed things a lot. I remember the days when I was in the office at 2:00AM writing code and would get stuck. All you can do is read the code, then read the code some more until you figure it out.

Well, now you can ask an agent to explain it, or even better, do the code for you. So, with agents it’s gotten even better and easier for us to be able to be distributed.”

Topic #3: Why restrictive corporate IT gives startups a massive advantage

One hilarious exchange between Rajeev and Thomas was about how large companies are slow with AI due to restrictive IT:

Rajeev: “Over the holidays, I bought a laptop because sometimes you can’t install things because of IT. So, I bought my own personal laptop at the Apple store, installed Claude Code, built an app, a bunch of Python scripts, and things like that. As someone who has grown up writing code to my personal bar, it’s super high – as it is for all my engineers. Now, I can write code much faster with agents and things like that.”

Thomas: “To all startup founders: when an investor asks how you’re preventing the incumbent from doing the same thing you’re doing: just tell them the CTO of Atlassian had to buy a laptop on his own money to start coding!

That’s the best answer I can give because it shows you how even relatively agile companies [like Atlassian, a promoter of Agile] that actually develop agile tools, can’t really do what you can do in five minutes!”

Topic #4: Why CTOs are rolling out agents with top-down mandates

Thomas discussed how engineering leaders introduce agentic tools after playing with them as side projects.

“What happened in the last two years through coding agents like Copilot, Cursor, and Devin, is that many CTOs and CIOs, even in the largest banks, realized they can go back to coding. It no longer takes hours to install all the packages and figure out all the problems.

I actually have friends in these spaces who tell me stories of how every night they’d give the agent a task and say: “build me this.” Then, in the morning they would check in with the agent and instruct it to keep going. And then they have their normal day-to-day as CTO of a big bank.

You do that for two weeks and you realize everything is going to change – and that it has to change in my organization. And in contrast to the Twilio founders – in their case, they had developers decide, bottom up, what tools to use – these CTOs and CIOs say: “I don’t want to hear any excuses. We’re going to roll out agents.”

Still, many banks are on very archaic systems and have developers log into Citrix, and jump through all sorts of hoops to use agents.”

Claude Code was introduced in a similar way at Canadian fintech, Wealthsimple, as we covered two weeks ago.

The fireside chat touched on more subjects, including the changing role of software engineers and engineering leaders, and the biggest changes Thomas and Rajeev expect this year.

Watch the closing session

You can now also watch the other keynote session: How AI is reshaping the craft of building software with OpenAI and the Codex team. Full subscribers can already check out all sessions at The Pragmatic Summit, Q&A included. The other sessions without Q&As will be released next week for everyone.

The remainder of this article covers an event hosted by Martin Fowler.

3. ‘Future of Software Development Summit’ with Martin Fowler

Back in February of 2001, a group of 17 software developers gathered at a ski resort in Snowbird, Utah, where they drafted and signed the famous Agile Manifesto.

Almost exactly 25 years later, Martin Fowler and Thoughtworks organized a retreat at almost the same place in Deer Valley, Utah. He invited around 50 tech leaders from large enterprises and nimble startups, researchers, authors, and experienced, hands-on software engineers. Held earlier this month, it was “The Future of Software Development,” and I was delighted to attend the two-day gathering featuring Annie Vella, Kent Beck, Steve Yegge, Gene Kim, Laura Tacho, – and of course, Martin himself.

This was no attempt to draft a second landmark manifesto; rather, it was an opportunity to share notes and consider where the tech industry might be headed. It was an intense day of back-and-forth about what we’re seeing and what to make of it.

One declaration was drafted. Kent Beck, Laura Tacho, and Steve Yegge, came up with an apposite statement on inflated expectations upon AI to magically improve everything in workplaces:

“Organizations are constrained by human and systems-level problems. We remain skeptical of the promise of any technology to improve organizational performance without first addressing human and systems-level constraints.

We remain skeptical and we remain human”. – Kent Beck, Laura Tacho, and Steve Yegge.

Personal notes

I have my own thoughts on the topics covered:

Everyone, everywhere is adopting AI rapidly, which in some ways is very surprising. Looking back on tech innovation over the last 20 years, such as mobile, cloud, or even crypto – adoption was gradual. For example, native mobile progressed like this:

1. Indie devs: in the first few years, startups, experimental individuals, and teams onboarded to the new technology. For native mobile apps, 2008 was when iOS and Android launched their app stores. Between 2008 and 2010, it was mostly indie developers, small teams, and companies that built their apps.

2. Startups and forward-thinking companies: a few years later, more startups saw the opportunity to jump in. For example, Uber was founded in 2010 and Snap in 2011. Some forward-thinking places also launched in this period, like Starbucks with an iOS app in 2009.

3. Larger tech companies: Years on, some forward-looking, larger tech companies adopted the technology. For example, Bank of America launched an app in 2011 (3 years into native mobile), Yahoo started to go all-in on native mobile apps from late 2012.

4. “Traditional” tech companies finally catch up: Eventually, slow-moving companies also move over when it becomes a baseline expectation to use a new technology. For example, Ryanair launched its native iOS app in 2014, six years after native mobile apps emerged.

Contrast that to today, when it seems like everyone is moving all at once. I talked with folks from companies like John Deere (agriculture), 3M (industrial products), Cisco (networking hardware), Wealthsimple (finance), AWS (cloud), and startups, and none could be called “behind.” Every company is rolling out agentic AI tools; for example, Wealthsimple did a global Claude Code rollout just a week earlier.

Based on past experience, you might expect at least some traditional companies with no exposure to AI to wait and see what happens. But I found none doing so.

Traditional, “old school” companies don’t seem to be lagging behind the pace. A traditional company with 10,000+ developers has more than 50% of software development outsourced. This place founded an AI platform team with a mission to eliminate outsourcing activity via a mix of efficiencies from AI tools and targeted hiring.

They have already built an AI debugging agent popular across the whole business that’s connected to all internal systems like monitoring, logging, and internal data stores. It helps a dev pinpoint any and all errors and bugs. This platform team is building more internal tools and hiring aggressively.

It’s a similar story at other trad places: most are already using AI agents, some are experimenting with AI used for large-scale refactors, and others are seeking use cases for agent swarms.

Smaller engineering teams are a certainty. Around 20 engineering leaders – many at large enterprises, some at tiny startups – said that engineering teams are shrinking. One Head of Engineering at a 200-year-old company in the agriculture sector put it like this:

“We are already seeing the end of two-pizza teams (6-10 people) thanks to AI. Our teams are slowly but surely becoming one-pizza teams (3-4 people) across the business”.

To emphasize, this is one of the most traditional companies which sells physical goods and hardware, and whose business has no disruption to fear from AI. But they’ve also adapted!

Observations from Laura Tacho

Laura was kind enough to share her notes on this event for this article:

Read more