So, I visited the two leading AI labs to see how teams and engineers do things day to day. In this article and in an upcoming follow-up, I’ll share what I learned about how AI is reshaping software engineering principles many of us are accustomed to – and what’s stayed mostly the same despite the AI wave.

In a later article, we’ll compare findings from Anthropic and OpenAI to see what their ways of working might mean for the overall direction of software engineering.

Thanks to Anthropic for showing me inside their lab in San Francisco. I talked with four people:

• Katelyn Lesse, Head of Engineering for Claude Platform, whose organization owns the infrastructure that Claude runs on

• Jarred Sumner, creator of Bun, now at Anthropic on Bun and Claude Code

• Thariq Shihipar, who works across Claude Code engineering and education

• David Hershey, at Anthropic’s Applied AI organization in a role resembling a sales engineer, working with customers like Cursor, Cognition, and Perplexity

Thanks to them, I got a sense of where things are headed at the leading AI lab – and possibly for the wider industry.

Before we continue, The Pragmatic Engineer will be on summer break for the next week and a half. This means no Thursday article this week, and no articles next week. I appreciate your understanding and support!

Back to today’s deepdive, we cover:

1. Complex & long: Claude Managed Agents. One of the most complicated projects took the Claude Platform team six months to ship, and created a new primitive to use at the agent infra level. Infra projects still need re-architecting mid-way through and take time to get right.

2. Twelve-month project done in 11 days: Bun rewrite to Rust. Migrating a 500K+ line project to another language used to take a small team a year, making it impractical. With Fable and $165K of tokens, it recently took the creator of the project less than two weeks.

3. Changing engineering practices. Inside the AI lab with more than 3,500 employees, prototyping is more fluid, verification is more time-consuming than implementation, code review and testing are increasingly done by AI.

4. Team-level changes. Design is more ongoing and less upfront, teams work on more projects, a maximum of two engineers per project, and more.

5. Still the same: two-pizza teams, planning is important, PRDs are relevant in complex projects, context switching is a challenge, the ratio of time spent on coding vs testing not changing that much.

6. Changing the “standout” software engineer archetype? Deep understanding, including of a layer below what you work on, is valuable, along with the ability to coordinate work.

7. Will AI replace software engineering? The more hands-on software engineers get with AI at the lab, the less they fear their jobs are going away.

1. Complex & long: Claude Managed Agents

The Claude Platform team’s most complex project in the past year was building Claude Managed Agents, a pre-built harness for production agents that runs in the cloud on infrastructure managed by Anthropic, or on your team’s own infrastructure, with any sandbox you choose. The project took around six months from idea until launch in April. Katelyn Lesse, head of engineering for Claude Platform, shared the story.

Claude Platform

This team sits between the model/accelerator layer (Claude models operate on GPUs) and the product/application layer (with products like Claude Code and Claude Cowork):

Katelyn on what the Platform team does:

“We’re on the ‘token hot path.’ The prompt comes in, then we tokenize it. Then, things like safeguards and billing all happen within our layer.”

What the Claude team calls “Platform,” I think more of as “API.” Claude Platform operates the API, and owns responsibilities an API would have. Of course, the team does more than that, and Claude Managed Agents is one case we cover here.

The platform layer is being migrated from Python to Rust. Originally, this layer was written for Python for the “usual” reasons at AI companies: it’s a convenient language and AI researchers use Python already, which enables quick iteration. But Python is single-threaded, and at scale, when the API is under high load, it’s not as performant as Rust.

Harness infrastructure demand

The project came together due to customers wanting their own “harness infrastructure”, says Katelyn:

“We started with a model where you get an API to define an agent, then you get an API to start a session with an agent. The reality of what the world wants and needs right now is people running their own infrastructure. So, we started to build a self-hosted sandbox.

But then, what we started to hear from lots of customers is that they’re trying to hack harnesses together, running their own “harness infrastructure,” and this gave us the idea for Claude Managed Agents.”

The largest part: planning

In this project, Katelyn said the single biggest matter was planning:

“There are products you can jump straight to prototyping, but then there are ones where you need to start by architecting it properly. For example, if we build a TypeScript CLI – which is pretty trivial for what needs to be built – we could go straight to prototyping. But with Claude Managed Agents, we needed to first figure out what we are doing.

Of course, we did some upfront prototyping for Managed Agents: hacking and spiking things. But prototyping itself was more about understanding the requirements.

Our planning process looked more like a typical pre-AI planning process. You know how every team has the project, where everyone comes up with some version of the same idea and people keep floating and circling it around until you finally do it? Managed Agents was this for our team. When we started the project, we had documents dating back up to two years about ideas and suggestions.

Post-planning, when the project officially kicked off, a PRD (product requirements document) was created:

“In the end, it was the Product Manager and the Tech Lead on our API Agents team who decided to pull the trigger and kick off this project. We’d get in a room, go through it, and get aligned. But it wasn’t just us: we’d have to align with teams around the business, other cloud providers, and other engineering teams. For example, we have a sandboxing team inside of the Platform org: and so this team was consulted on the design of Managed Agents, given this product would spawn a lot of sandboxes.

Just like before, we had a PRD, it was a Google Doc. We used a Google Doc because we needed to coordinate all interested people. This has not gone away.

Similarly, my product counterpart and I run product reviews.”

Some processes from before AI, like the PRD, are still useful in complex projects today, for getting large groups of people on the same page.

Build for an internal customer first

With planning complete, the team decided to do a “spike” and stress-test the idea and architecture, by building the backend of Claude Code on the web. Remote execution of code with an agent harness was a similarly shaped problem to the one they wanted to solve for customers. The thinking was to start by solving it for the Claude Code team before tackling it in a more generic way for customers.

Internal teams are more fluid than before AI. Katelyn:

“Pre-AI, we might have hit the Claude Code team up with a bunch of big requirements documents, and they would have then hit us back with another set of documents. Now it was much easier: someone on our team built a few components, took it over to the Claude Code team, and they started to hack around it. We could figure out how this component plugs into this part of their product, and the other way around. It was just a faster and easier process, getting this first internal version of the product up and running.

Aligning with other teams on interfaces remains important, and it’s easier. Back in the day, you’d have to come with a fully spec’d interface to use. Now, we could do it a lot more fluidly: we could stand up a stub service that shadowed traffic to start with, and iron out the interfaces with the Claude Code team as we went. They did some hacking on it and gave feedback, we made changes while building out the service under the interface, then went back to make it work.”

They launched a service for Claude Code’s mobile app to spin up a sandbox, boot up Claude Code and run it. The service went to production and the Claude Platform team took the learnings.

Re-architecting midway through

It’s likely a familiar scenario many engineers can relate to, that after planning a project and getting underway, you see that you’re going to need to change the architecture. It happened on this project, too.

The platform team ended up re-architecting Managed Agents based on learnings from the Claude Code “spike.” Re-architecting meant decoupling the “brain” of Claude and its harness from the “hands” (sandboxes & tools that perform actions) and the “session” (the log of events). Each became an interface that made few assumptions about each other.

The team also built an abstraction around vaults and credentials. Credentials can safely be stored inside a vault. All calls using credentials are made via a proxy which has a session token. It is the proxy that fetches the right credentials from the vault: the credentials are never seen by the agent, sandbox, or session. Credentials are only injected at the egress boundary when the service is invoked:

Internal “dogfooding” helped surface hard problems to solve. A few examples:

• Reliability and scalability: these are really hard to do well for agents because if connection to the sandbox is lost, the whole agent dies and you lose state

• Credentials and access control: also hard and problematic, especially when first building the service

The Managed Agents team shared more about this re-architecting project.

The project took about six months, by no means a rapid process. Katelyn emphasized that pre-AI, a project like this would have probably been in the realm of two years. Managed Agents is one of the biggest projects the Claude Platform team has built, and more complex than it looks: for example, adding support for running agents on AWS, GCP and Azure.

2. Twelve-month project done in 11 days: Bun rewrite to Rust

As covered before, Jarred Sumner is the creator of Bun, a popular JavaScript runtime with 22 million monthly downloads currently and Claude Code as a dependency.

Bun is written in Zig, a performant, productive language. However, it’s not memory safe and memory issues kept coming up. Jarred thought that rewriting the project to an also-performant, memory-safe language like Rust could be an option – except that rewrites like this turned out as follies in the past. Jarred (emphasis mine:)

“Historically, rewrites are a terrible idea. Excluding comments, Bun is 535,496 lines of Zig. A rewrite in another language would take a small team of engineers a full year. It would mean freezing bugfixes, security fixes or feature development for that time. The least risky approach to getting something shippable would be a mechanical port from Zig to Rust, with the minimal number of behavioral changes, using the exact same test suite we already use for testing Bun.

Fortunately, Bun’s own test suite is written in TypeScript which means it doesn’t depend on the runtime’s programming language.

A year of zero user-facing impact was not an option we could consider. So, enforcement through code style to fix stability issues was our best bet, and was our plan when we added Rust-inspired smart pointers to Bun’s codebase.

But honestly, I didn’t want to do it. Homegrown smart pointers offer worse ergonomics than Rust, with none of the guarantees.”

But then, Jarred asked if AI could do the heavy lifting and wondered how much the migration could be sped up. In the end, he completed the rewrite from start to merge in 11 days, using 64 parallel agents and $165,000 in tokens at API price. Here’s Jarred on how his AI-heavy rewrite compared:

“By hand, I think this would’ve taken three engineers with full context on the codebase about a year, during which time we wouldn’t be able to improve Node.js compatibility, fix bugs, fix security issues or implement new features. We never would’ve done that. The realistic alternative was to do nothing and keep fixing the bugs at the top of this post forever.”

There was a lot more to the project than typing out the “...make zero mistakes” prompt:

• Jarred made a detailed plan and style guide on how to migrate

• He set up the project so agents would not use Git worktrees which he found slow, but worked on different files in the same codebase

• He created an orchestration system where each AI agent came up with suggestions of what to change, but did not make a change to the file to avoid conflicts; an orchestrator AI agent created the commits

• The most time and tokens went on fixing the compile bugs, tests, and verifying that things worked

• Bun itself has a very robust test harness: when all tests pass, it’s a high-confidence signal that the rewrite works

• Crucially, Jarred is the ultimate domain expert in Bun: he created the project and knows the codebase better than anyone

The rewrite has been shipped to production and powers Claude Code today.

We cover a lot more on this in What can we learn from Bun’s rapid Rust rewrite with AI?

3. Changing engineering practices

So, what has changed in how teams build software at Anthropic, compared to the pre-AI days? That’s the question of this article, and it seems that many things are different. Let’s go through it:

AI lab-specific practices

Some things as normal as breathing at AI labs like Anthropic stand out as different with an outside perspective:

• Everyone runs multiple AI agents all the time. Running 3-10 parallel agents is a given. Folks I talked with had their agents running in the background or cloud.

• No token budget, usage not tracked. One major difference between AI labs and everyone else is that there really is no token limit or token leaderboards that promote tokenmaxxing; people already use agents all the time.

• Very high autonomy. Work is becoming more structured inside AI labs, but there’s still massive autonomy compared to Big Tech and most startups. When everyone has unlimited tokens, it’s pretty easy to prototype any idea.

Prototyping and “spiking” is far more fluid

It was several times faster to prototype early approaches for Claude Managed Agents. Similarly, “spiking” the Claude Code mobile backend implementation was much faster than pre-AI, Katelyn told me.

Verification takes longer than implementation

Jarred made a point about the split between implementation and validation in his 11-day rewrite to Rust. Roughly, it was:

The “implementation” part of rewriting the code from Zig to Rust took about 15% of the time, while 85% went on fixing things up: getting it to compile, fixing tests, verifying that it worked.

Most tokens no longer spent on implementation

Thariq:

“We see that few tokens are spent on actual implementation. Most are spent on discovery of unknowns, prototyping, mocking, and then in verification and testing.”

Jarred’s Bun rewrite echoes this: he spent more tokens on fixing up the implementation and verifying that it worked than on the implementation itself!

Code review and more testing by AI

Jarred:

“Critiquing the code and testing it with agents is a new approach we do a lot more of. I think a lot about trust when you merge a lot of code. How do you merge 100+ PRs a day, and make sure the code works? At this pace, you need to trust the code without the ability to read it all yourself. And I think it’s a few things:

• Code review: it needs to be really good and automated. I’m clearly tooting our own horn here, but I find Claude’s code review to be really good. Claude’s code review catches bugs that would take me an hour of closely reading the code to figure out. The caveat is that it’s expensive!

• Security scanning: for this Rust rewrite we did 11 runs of the Claude Security Scanner.

• Fuzz testing: we’ve also been doing different types of fuzzing (fuzz testing), where we had Claude write a fuzzer for things like parser fuzzing.

Running out-of-process testing, where it happens in a different process/session from coding, is one way to build trust in the code. I expect more of this.”

New pattern: fanning out work to AI

Jarred described a new way he works:

“A new approach I’m using is fanning out a lot of the work to many Claudes at the same time. I did this with the Bun rewrite, but I use it for other work. This approach works very well for me, and I feel it’s pretty underused.”

Time-saving automations powered by agents more widespread

Jarred listed several time-saving automations set up by the Bun team to run an active open-source project with a small team, while the team works on Claude Code:

• Every time someone files an issue, Claude runs to try and reproduce the issue. If it succeeds, it starts another container, which then tries to fix the issue and submit a PR.

• The agent tasked with submitting a PR has to write a test that fails in the system version (the one without the patch) of Bun, and passes in the debug build with the patch, before it is allowed to submit a PR

• There are other automations, like if there is no test, the PR is auto-rejected; all linters are run: Claude Code review is run, CodeRabbit’s code review is run, and the agents go back and forth on the GitHub pull request

Auto-merge of pull requests: coming soon?

Pull requests are merged manually when all quality gates pass, but this could become automatic at some point. Once all the above checks pass, all (AI) code review comments are addressed, tests are added to new code, etc. As an interesting aside, a lot of GitHub activity is Claude talking to Claude!

But manual merging may vanish in low-risk cases, at least for the Bun project. Jarred told me:

“Today, a person presses ‘merge’ but within a few months, I expect:

• Automated reviewer LGTMs

• → another Claude with a fresh context window judges if it’s simple and low blast-radius

• → if it is: auto-merge!”

Test assumptions with each model generation

Inside Anthropic, the team keeps testing their priors. Thariq gave an interesting example:

“The thing with agents is that you have to revisit any assumptions you have made because it can change with a new model generation. For that reason, we deleted 80% of the Claude Code system prompt recently because the model has gotten smarter.

Using HTML is another assumption we needed to re-examine. HTML is one of those things which Claude is a lot smarter at than many of us expected. I’ve started preferring HTML as an output format over Markdown, and see this being used by others on the Claude Code team.

HTML can convey much richer information compared to markdown, HTML documents are easier to read and share.”

4. Team-level changes

At Anthropic, there are also changes in how engineering teams operate, compared to pre-AI.

Read more

Subscribe now

After I recently interviewed Simon Eskildsen, co-founder and CEO of turbopuffer, on the main stage at the AI Engineer’s World Fair, many people at the event told me they found him relatable and inspiring for his choices to stick with one company for close to a decade, his belief in the power of “napkin math” to reveal why products run slowly or cost too much, and for his insight about how too much of VC funding is about ego, not business needs.

This article contains the most interesting parts from that conversation; in particular, the concept of “napkin math” – doing quick calculations to get rough answers – as a way to challenge existing systems to improve. The full 55-minute-long video of the discussion at the AI Engineer’s World Fair is available to watch:

Watch the full interview

Today, we cover:

1. Algorithmic programming speedrun. While in high school, Simon competed in the International Olympiad for Informatics (IOI) which pushed him to learn about writing correct programs that are fast and memory-efficient, and more.

2. Eight years of infra at Shopify. There are many upsides to longer tenure: Simon learned infrastructure concepts, dug deep into databases running across regions, and learned to write software that ages well.

3. “Napkin math” as a superpower. Simon became obsessed with finding the theoretical limits of compute operations, such as sending over data and reading bytes. This held him in good stead at Shopify, and then at his startup.

4. Origins of turbopuffer. When ChatGPT took off, context windows were small, and so stuffing them with the right information was key for AI applications. Fast search was needed, but the search solutions were surprisingly expensive. Using “napkin math”, Simon discovered they were far more expensive than necessary.

5. A new product without VC funding & Cursor as customer no.1. After raising $8M in seed funding, Cursor rolled the dice on the new turbopuffer team after Simon helped with their search & database needs.

6. Reasons to raise venture capital. Fund R&D, fund growth, stroke founders’ egos, and more.

Disclaimer: turbopuffer is a season sponsor of the podcast, but as with all our deepdives, this article is independent of podcast sponsorships.

1. Algorithmic programming speedrun

A self-taught professional, Simon skipped college to work at Shopify and spent nearly a decade there building a variety of systems. His interest in computers started after initially getting into building websites aged 12. Growing up in Denmark, he dabbled in HTML with tools like Microsoft FrontPage and Adobe’s Dreamweaver.

While still a teenager, Simon “hit a wall” by exhausting the Danish-language part of the internet for learning programming, and got into the World of Warcraft MMO game, which helped him acquire English.

After discovering the International Olympiad for Informatics (IOI), he decided to enter, despite a very competitive, multi-stage selection process open to all Danish high schools. IOI problems are pretty similar to Leetcode problems: algorithmic challenges that value correctness, speed, and memory usage.

Simon cleared the online qualification round and was invited to the Danish Nationals. This was more than a competition: a weekend-long bootcamp to teach participants more advanced programming techniques such as recursion (which Simon already knew), the divide-and-conquer algorithm, and dynamic programming; one of the more tricky concepts to master for algorithmic programming.

I tip my hat to the organizers for creating a challenging bootcamp and offering the opportunity for people like Simon, who wasn’t aware of what “NP complete” meant when he entered. He recalls:

“The routine was that every four hours we’d be introduced to a new “programming concept”, and receive ~2-6 tasks where this, combined with previously introduced concepts, had to be applied. All the solutions had to be submitted to the same site I submitted my qualification solutions to, as it was all part of the final evaluation. The tasks were incredibly challenging, like nothing I had ever tried before.

Sometimes in extreme desperation combined with tiredness from the trip, I’d think about taking the next train home. This feeling would disappear with the utter joy and confidence that arose whenever I would finally solve a task, and creep back once again when I found myself still struggling after an hour on a new problem. But this kept me going. By Saturday afternoon, I had almost managed to get up to speed with the others, and was doing the same tasks as them.”

Ultimately, Simon claimed one of six spots in the Danish national team, making it through to the regional finals.

Aware of how little he knew about programming, Simon doubled down to catch up. He realized that most other participants were better prepared and that he had to do something to survive the next qualifying round. So, he got to work:

As Simon recalled:

“I armed myself with a borrowed copy of “The Art of Computer Programming”, worked through the exercises, read up on common algorithms on Wikipedia, completed tasks on USACO, and memorized the critical parts of my Vim config for the competition computers. I managed to create quite an intense training weekend for myself.”

The regional finals were even more challenging than the Nationals, and participants struggled to write performant solutions to the problems. Despite putting in the effort, Simon was pretty sure he was out of the competition, and recorded his learnings:

“[From the programming competition] I learned that you must avoid digging holes. Repeatedly, I found myself so fixated on getting a particular idea to work that I’d get absolutely nowhere. Sometimes, you have to bite the bullet, delete your program, find a new sheet of paper, and start from scratch. A good case of this is when you start working around a general solution to solve specific edge-cases. I learned that there is almost always a simple way to solve a problem without explicitly handling edge-cases. If there are two edge-cases, there’s almost certainly two more. The simple solution will handle edge-cases automatically – even those you might not have considered.”

But as it turned out, two months later he was told he had been selected to represent Denmark in the final phase of the competition. Simon continued to push himself out of the comfort zone of web development, using HTML, a bit of PHP, and getting into algorithmic programming.

Another decision that would later bear fruit career-wise was starting a blog while at high school – though he didn’t know it at the time. In 2010, Simon launched his English-language blog but posted only one or two articles per year; mostly short ones describing problems he’d solved:

• A five-line script to take screenshots and upload them to Imgur, all from the command line (2010)

• Setting up a Ruby HTTP server on NGINX using the library Unicorn (2010)

• Why Simon favors SQL over ORMs: Object-Relational Mappings (ORMs) are frameworks to abstract SQL, these were very popular around that time (2011)

Two posts he published would go on to have an impact on his life. One was about his IOI experience and learnings. The other only came about after he suffered the disaster of fatally dropping his iPhone during his final year in high school.

2. Eight years of infra at Shopify

With his smartphone dead, Simon was forced to switch to an old-school Nokia “dumb” phone. He wrote a short article about the experience titled Why I’m glad my iPhone broke, which went viral on Hacker News, making it to the front page of the site with many comments.

It seems that someone at Shopify, in Canada, read the article and others by him – including the summary of his impressive performance at the International Olympiad for Informatics – as a recruiter from Shopify flew Simon out to interview in Ottawa, Canada, where he was offered a software engineering position at the company.

In his new job on the infrastructure team, Simon made notes about things he didn’t understand and read up on new concepts. As he told me:

“When I started at Shopify, I was insecure about having not studied computer science and my biggest exposure to programming had been the IOI. If nothing else, preparing for the IOI taught me that you can sit down, read a paper, and figure it out if you spend enough time on it. So, I did that repeatedly.

In my first year at Shopify, every time I heard something I didn’t know, I noted it down on a piece of paper. Then, that evening, I would read up about it.

For example, if someone at work mentioned TCP, I assumed that surely they would know exactly what’s in the three-way handshake and how TLS is layered on top. And I also assumed they’d looked at Wireshark and all of that. I don’t think that’s true, but that’s what I thought at the time! So, I dug deep into everything I encountered.”

Working in infrastructure meant solving interesting engineering problems at a time when Shopify was growing 120-140% in load year-on-year; Simon was exposed to problem domains like:

• Ruby on Rails and databases: Shopify was already one of the world’s largest Ruby on Rails monolith applications, and working on infrastructure meant being close to the database layer

• Sharding and cutting over: as Simon’s manager used to say, “you cannot cache writes”, so Shopify had to move from running on a single group of machines to a shard (partitioned) setup of machines. The team did the cut-over (moving to the shard) just a week before Black Friday, the busiest time of the year.

• Multi-data centers: expanding database footprint from one data center (DC) to multiple DCs

• Splitting up key services: Shopify had a 128GB machine (massive for the time) running Redis as a key-value store. Nobody dared touch it until one day the service went down. Then, the team split responsibilities up into separate services.

Simon built a framework to simulate networking conditions called toxiproxy. The idea came to him while attempting to write a more thorough, systems-level test to see how Shopify’s application held up during partial outages. He created a matrix of Shopify’s services, and wanted to write a complete test suite for this matrix, to see if the system was resilient enough to handle some parts of the system being down. For example:

As Simon wrote about this problem at the time:

“Having tests for the matrix was a must; otherwise we couldn’t guarantee the state of the matrix wouldn’t degrade over time. Since the tools mentioned previously require root access, we investigated proxies to simulate latency and downtime at the TCP level, but didn’t find one that suited our needs. We needed an online API to edit proxies and to support deterministic latencies, which made it suitable for integration testing.”

This was the inspiration that led to toxiproxy. As Simon told me on stage:

“Toxiproxy is a proxy that sits in between the application and the databases. With the proxy in place, you could do things like make an API call to the proxy, instructing it to simulate taking the database down, or making it slow.

Over time, we added a bunch of other ways to inject failures. With this proxy, we did not have to mock low-level drivers, but we could test failure handling really well.”

Toxiproxy was open sourced in 2014, and apparently still runs in Shopify’s CI system 12 years later, to Simon’s knowledge!

The biggest benefit of a long tenure at a company was learning to write software that ages well. Simon told me that this was the lesson that made it so worthwhile, and how often the simple solution someone put together in a week or two outlasted the big, multi-team RFC-driven solutions. That was a lesson he still uses today.

3. “Napkin math” as a superpower

While at Shopify, Simon became interested in figuring out the theoretical limits of certain computer operations. For example, how much bandwidth does DRAM have for memory transfer? How long does a round-trip operation to AWS S3 take, and how much does it cost? What does a gigabyte of memory cost?

He sought the actual numbers, wrote a script to collect them, and created a table with the data:

Simon started to memorize key numbers with flash cards. He wanted to be able to instantly recall all important numbers. As he told me:

“Napkin math was essentially just this table that I maintain on GitHub. There’s probably like 50 of these numbers and then a script that generates them all.

For example, what does a gigabyte of memory cost? $2. What does a gigabyte of S3 cost? Two cents. What does a gigabyte of this cost? 10 cents. What does it cost on spot? What does it cost on a three-year commit? I had a massive table, then I created flashcards for almost every single cell so I know all these numbers.

This was a project I started taking on at Shopify because I would review projects and these numbers would be helpful.”

Napkin math enabled him to challenge design decisions based on benchmarks that had issues. As he told me:

“When reviewing a project, a product team would tell me that they chose Database A over Database B because they benchmarked both, and Database A was better. I hate benchmarks because making design decisions based on benchmarks is not a satisfying answer to me.

For example, you’re saying that with Database A, as per the benchmark, it takes 10 seconds to do this one thing. But it should take 10 milliseconds if you do the napkin math. Say, it’s a search query. Okay, you’re searching for three terms then. Each term has this many documents that match it. That’s this many megabytes. We intersect this many lists. You have DRAM bandwidth on multiple cores, at 100 GB/sec. So if you do the math, it should be 10 milliseconds.

So, now you’re telling me that the benchmark for the same thing takes 10 seconds, which means one of us is wrong! Either there’s a gap in my understanding – which is possible! – or the benchmark measures the wrong thing.

Often, it would be things like the person doing the benchmark and not realizing that the query would be a distributed one, running across a hundred different nodes. And in this case, of course the p99 is going to be very high!”

Simon went even deeper into “napkin math” after leaving Shopify. He investigated whether MySQL’s maximum transactions per second is equivalent to fsyncs per second (the number of file writes per second a system can handle). He discovered that MySQL can handle more writes than the operating system could sync to the file, which was a surprise:

“It takes ~3 seconds to perform 16,000 insertions, or ~5,300 insertions per second. This is 5x more than the 1,000 fsyncs per second our napkin math told us would be the theoretical maximum transactional throughput!

Typically, with napkin math we aim for being within an order of magnitude, which we are. But when I do napkin math, it usually establishes a lower bound for the system, i.e., from first-principles, how fast could this system perform in ideal circumstances?

Rarely is the system 5x faster than the napkin math says. When we identify a significant-looking gap between the real-life performance and the expected performance, I call it the “first-principle gap.” This is where curiosity sets in. It typically means there’s (1) an opportunity to improve the system, or (2) a flaw in our model of the system. In this case, only (2) makes sense, because the system is faster than we predicted.”

Simon started to investigate, and learned that MySQL does grouping of transactions (not doing an fsync for every write), and it also does smart merging of multiple fsync operations that would be processed in parallel, effectively doing a “group commit” to further improve performance. This is a learning he’d later use: grouping writes together on top of S3, to reduce latency and improve cost.

Personally, I find it amazing how much you can learn with some measurements and by asking questions about how a system achieves results that go against what the “napkin math” suggested should be its limit!

4. Origins of turbopuffer

“Napkin math” also played a part in the creation of turbopuffer. Simon told me that three things had to combine for the project to come to life.

1. Search was a difficult project at Shopify. Building search was one of Simon’s final projects at the company, and he did not have a good time with it. He used a popular database vendor but struggled to have it perform napkin math. The query plans were not exposed easily by the database, so he could not figure out what was missing and why it wasn’t performing the napkin math without reading massive amounts of source code. Also, the search infra was difficult to operate.

2. Napkin math became a surprisingly efficient reasoning tool. As a toolkit, napkin math gave Simon a way to reason about what could be achievable with a machine, if utilized perfectly.

3. His friends’ startups needed fast search badly because of AI. In early 2023, Simon helped a few friends with infra at their startups, many of whom had the same problem: LLM context windows were very small (4-8 KB), and they needed to fill them with the right parts of documents, but this demanded very fast search. One startup budgeted that their new search vendor would cost $30K/month, while the existing infra bill was only $5K/month! For a bootstrapped Canadian company, the cost was too high, so they didn’t ship the AI feature.

Simon could not stop thinking about why search was so expensive, and why it didn’t line up with what the “napkin math” predicted the cost should be. Eventually, he laid out a basic architecture to make it fast and cheap:

1. Store the data to search in AWS S3

2. Do some clustering and organize the files

3. Get latency down. S3 is cheap and reliable, but has high latency

Driven by curiosity about how that could work, he set out to build it. As he told me:

“I just became fully obsessed that summer [of 2023] with building it. The first version was the simplest possible thing. I’m a very pragmatic person, so I didn’t get buried in detail. I barely read the literature on log-structured merge (LSM).

The simplest way to do this is to run a clustering algorithm on the vectors. You get the clusters, and then put the clusters in files. The files are called ‘cluster_01’, ‘cluster_02’, ‘cluster_03.’ Then you have a file called centroids of the clusters. Then, you search by downloading centroids, looking into them, then downloading the closest clusters.

There were a few optimizations around merging clusters that were adjacent in files, just to control costs and boost performance. But this was the core of it.”

What about performance? Simon:

“For the first version, I didn’t even implement a dedicated caching layer. I just put the reverse proxy (NGINX) in front of S3, and that was it! The performance improvement came from caching all of the S3 objects. It was the simplest possible layer at the NGINX level.”

5. A new product without VC funding & Cursor as customer no.1

I first heard of turbopuffer last year, when I interviewed Cursor cofounder Sualeh Asif about how they built the AI coding harness. From our previously published Cursor deepdive:

“The ongoing need to re-shard as usage grew became error-prone and frustrating. Sualeh told me the biggest lesson the team learned was to avoid sharding where possible in future. So, they looked for a vector database that could support multi-tenancy without requiring manual sharding.

Turbopuffer was a startup that promised this, so Cursor tried it and migrated a good part of their vector search use cases over.”

So, how did a billion-dollar startup come to bet on a tiny, unproven infra product with zero customers? There were a few factors:

First, Cursor was not yet a billion-dollar company in the fall of 2023, but a relatively fresh startup. The company raised $8M in seed funding in the same month. In 2024, the company was valued at $400M, and then its valuation surged to $29.3B. It was sold to SpaceX for $60B this year.

Secondly, it involved a tweet Simon posted after having had enough of building all summer, and wanting to test the waters. As he told me:

“I was so sick of working on [turbopuffer]. I’d been working on this all summer and didn’t know if anyone cared. I only wanted to work on it if anyone cared. So, I decided to put it on Twitter.

At that point, I had a single TMUX instance running on an 8-core node somewhere in GCP. I was thinking: “if someone goes to prod, I’ll set it up properly on multiple nodes.

But first, let’s see if anyone cares.

Anyone who’s worked in the internals of databases would’ve had too much pride to ship anything like that; I was just releasing it like a SaaS project. Why can’t you work on a database like it’s SaaS?”

So, he put out this confident-sounding tweet:

Simon had the confidence to launch his product because he knew it was rock-solid and scalable:

“I knew turbopuffer was reliable. It upheld its invariants. For example, you shut down all the VMs and no data is lost. All the writes are committed directly to blob storage.”

Cursor reached out. The company was an eight-person team at the time, fresh from seed funding and growing fast, but with a search problem that they were looking to partner with a startup on. Simon recounts:

“Knowing Cursor’s founders now, I’m sure they must have sat at the dinner table one day and were like: ‘the unit economics of what we have right now, where all the vectors are in DRAM, are not working.’

They probably were asking why someone had not built a solution where you can put all the vectors from the codebase into S3 (to make it cheap), and move the part of the codebase used in memory (to make it fast). Then, everything sits in blob stores and you just hot load it all into cache. When you open the codebase, after a few seconds it’s all in RAM and the queries are as fast as everything else.

Aman (one of the cofounders) was already talking about using S3 as a key-value cache, which at the time, barely anyone was thinking of for unit economics.”

Simon knew nothing about B2B sales at that point, and was not trying to “sell” anything to Cursor. He exchanged a few emails with the Cursor team about their use case of searching many small local codebases. He wanted to help Cursor with the unit economics, while also proving that turbopuffer works. So, he hopped on a flight from Canada to San Francisco, arrived at Cursor’s office and got down to debugging their database provider:

“When I showed up at Cursor’s office, they were having some Postgres problems. I asked, ‘do you guys have pganalyze?’ And they didn’t, so I was like, ‘Okay, let’s get that going. Let’s look at it. “

And the problem was the same thing as it always is with Postgres: autovacuum hadn’t run enough. And so they had all of these going to heap, when they should be doing index scans, etc. So, we were talking about all of that.

I was just helping them: my “database genes” just kicked in. I think this built enough trust with them to believe that if I know enough to help them with their database, maybe I also know how to build one.”

It was around when turbopuffer’s other cofounder, Justine Li, joined that the work with Cursor kicked off. Simon describes Justine as the best engineer he ever worked with at Shopify, and together, they made more performance optimizations. Cursor migrated their local code-base search over to this brand new product.

Simon promised to reduce Cursor’s bill by 95%, and delivered – thanks to napkin math! Cursor was spending around $80K/month on indexing and search; with turbopuffer in place, this dropped to $4K/month. Simon was confident about making this prediction thanks to napkin math! He did the calculations based on the fundamentals; only counting resource usage that Cursor was using, and not taking a margin for turbopuffer’s operating cost at the time.

6. Reasons to raise venture capital

After securing Cursor as the first customer, Simon was not convinced that turbopuffer should raise VC money. He explained his thinking:

“I understood that if you take venture capital, no matter how many smiles there are in the room, everyone’s expecting to earn a big return on some timeline that makes sense to everyone involved. And ‘everyone involved’ are pension funds in Canada.

But at the time, I did not know if turbopuffer could be a billion dollar company. It felt like a very niche kind of search engine. And that was completely fine with me!

So, I just looked at what we invoiced and I looked at my GCP bill. And I was making this equation that:

Customer Invoices >= GCP bill [the turbopuffer costs]

Justine and I were going to optimize the system until these numbers were roughly equal. And if we could get some other workloads over to turbopuffer, we could then start paying ourselves.

But either way, I didn’t know if I could even go and raise a bunch of money. I didn’t have any relationships. I was an outsider who grew up in Aarhus, Denmark, and moved to Canada.”

A few months later, turbopuffer ended up raising a total of $700K, just to be able to hire two engineers until the end of the year on real salaries. Simon found that most VCs did not take them seriously for wanting to raise too little money, and assumed this signified a lack of ambition! Two years later, with less than $1M in initial funding raised, turbopuffer crossed $100M in annual run rate. So, there evidently was ambition!

Since our conversation, Simon has reflected on the topic of VC capital and built a mental model of six reasons that justify raising external funding:

Reason #1: Research and development. This is what turbopuffer raised for: they needed the money to hire two more engineers until the end of the year, in order to build out more of the platform. Then, they would either get more customers and generate enough revenue to not need funding, or shut down the project if it failed to gain traction.

Reason #2: Growth. When you’ve built something and want to tell the world about it, often, you have to spend money on doing that.

Reason #3: Massaging founders’ egos. Simon:

“This is a very, very popular reason! You see big numbers, you get lots of press. But I think it’s a really dangerous reason to raise money.

I wish this reason was talked about more, because you are diluting all of your employees when you do it. And for some people, it can become a status game. It’s not what we are about: we’re here to build a big business together.”

Reason #4: Employee rewards. A startup is a long journey, and everyone wants to work with the best people. But there are necessarily few of these folks, so you want to reward the standout ones by raising money so that employees can sell their shares. This was why, in December 2025, turbopuffer raised a round where they let employees sell some of their equity: it meant they would not have to wait for a future financial event like an IPO.

Reason #5: Strategic partnerships. It might be the case that raising from a VC whose network of connections is key to your business succeeding, or that taking funding from a strategic investor gives you access to their platform or services.

Reason #6: Mergers & acquisitions (M&A). Purchase another company with the funding in order to expand the business.

Simon encourages founders to be honest about why they raise capital funding – and to watch out if the reason is egocentric!

Takeaways

Watch the full interview here.

It’s inspiring to see how far it’s possible to get with “napkin math” and by understanding the bottom layers. It didn’t sit right with Simon to make decisions about vendors based on hastily written benchmarks that might measure the wrong thing. By understanding the constraints of data transfer latency and storage cost, he found a way to estimate the theoretical lower bounds of the system.

This helped lead to more informed design decisions at Shopify, and showed him there was an opportunity to build a faster, cheaper search product than the status quo.

Getting “lucky” in business requires a lot of skill, and in-person greatly helps with first impressions. Turbopuffer’s first customer being Cursor sounds almost too good to be true. But the account from Cursor cofounder Sualeh Asif – and now from Simon – reveals the ingredients in more detail:

• Spotting a new business need: AI-native startups like Cursor saw their search bills explode, but needed search functionality to offer usable AI products

• Offering a product with a magnitude of lower pricing: what people made suddenly pay attention to turbopuffer was the promise of not “just 20-50%” cost savings, but a seemingly radical, 90%+ reduction in costs. When you’re late to enter a market (like search), you need major differentiation, then deliver on it! This is also what Cursor’s attention.

• Building trust before a sale: Simon helped the Cursor team fix their existing database before discussing using his product

• In-person impressions: if Simon had not flown to San Francisco to meet the Cursor team in person, would they have taken a bet on turbopuffer?

• Launching at the earliest opportunity: none of this would have happened if Simon did not announce the first version of the product when he knew it could work, but was still in a pretty unpolished state!

It’s always helpful to be aware of the dynamic of venture funding – some of which are rarely mentioned. Simon knew investors expect returns and growth on timelines which they set. Raising money is helpful in many situations, but problems arise from egotistical reasoning, and Simon believes too many founders prioritize them – knowingly or not.

“I don’t prompt Claude anymore. I have loops running that prompt Claude and figuring out what to do. My job is to write loops.”

Soon after, Peter Steinberger, creator of OpenClaw, preached loop design in a post:

Elsewhere, Addy Osmani, formerly of Google, wrote an article, ‘Loop Engineering’:

“Loop engineering is replacing yourself as the person who prompts the agent. You design the system that does it instead.”

That’s three mentions in quick succession of this new approach, which is a novel one and therefore pretty abstract to me. To find out more, I turned online to some of the folks who read these articles. In replies, you told me what “loop engineering” means to you and gave some examples of loops in your work.

Today, we cover:

1. Where it began: “Ralph Wiggum” loop. A year ago, software engineer Geoffrey Huntley shared how he builds “loops.” In December, the approach went viral and “Ralph loops” were born.

2. The /goal command ships in all major harnesses. By May this year, the major AI coding harnesses added support to run a loop from a single prompt, using the /goal command.

3. Loops which devs use: triggers and cron jobs. I asked devs how they use loops. Most use cases involve responding to events or running scheduled jobs. They are useful, but don’t feel like brand new workflows.

4. Helpful loops for devs. Open PRs for newly-recorded app issues, have notes ready when an oncall joins an outage Slack channel, “babysitting” and fixing nightly end-to-end tests, and more.

5. Disappointment and “tokenmaxxing”. Several devs reject looping after trying it. Agents drifting, and the “human in the loop” having better results are some reasons. Also, at companies that pay API prices for tokens, loop engineering gets expensive fast.

6. Was looping a hack while tooling caught up? Distinguished engineer Max Kanat-Alexander believes the “loop” might have just been a temporary hack while the harnesses added the ability to do the same from a single prompt.

7. Does “context engineering” matter more for devs? Except for engineers building AI infra, there seems little benefit in going deep into loop engineering. Instead, becoming familiar with AI context windows – also part of building loops – could be more useful.

1. Where it began: “Ralph Wiggum” loop

Exactly a year ago, software engineer Geoffrey Huntley published the article ‘Ralph Wiggum as a software engineer’. The name references the naive son of the local police chief in The Simpsons, who is extremely eager to always be helpful. In engineering, “Ralph” is intended to continuously nudge the agent in the right direction. Geoff described it:

“Ralph is a technique. In its purest form, Ralph is a Bash loop:

while :; do cat PROMPT.md | claude-code ; done

Ralph can replace the majority of outsourcing at most companies for greenfield projects. It has defects, but these are identifiable and resolvable through various styles of prompts.

That’s the beauty of Ralph - the technique is deterministically bad in a nondeterministic world.”

The article expands on the idea of the Ralph loop:

• Start the agent with a prompt that captures the task and defines a goal

• Create a plan of work, each item with a success criteria

• Start the loop:

  • Take one item per loop

  • When the agent is done, check if the goal is achieved

  • If not: start the agent again, with a clear context window

    • Start loops if needed: spawn subagents as and when necessary

Geoff published the experiments he did with this approach, such as building a new programming language last summer, and said it requires skill:

“Engineers are still needed. There is no way this is possible without senior expertise guiding Ralph. Anyone claiming that engineers are no longer required and a tool can do 100% of the work without an engineer is peddling horses***.”

The “Ralph method” blew up late last year with the arrival of better models which were surprisingly capable of building ambitious projects. Software engineer Matt Pocock created a tutorial, ‘Ship working code while you sleep’ with the Ralph Wiggum technique. He said:

“One of the dreams of coding agents is that you can wake up in the morning to working code, where your coding agent has worked through your backlog. It has spit out a whole bunch of code for you to review, and it works.”

Before the Ralph loop, Matt did this in two steps:

1. Ask the agent to create a detailed plan for the work, with tasks broken out

2. Then, in a sequential order, have the agent complete each subtask in a separate run

A problem with this approach is there’s no easy way to add new tasks to the “masterplan”. Software engineers know that most plans do need to be modified, often while the work is ongoing. In contrast, with Matt’s take on the Ralph method, the “masterplan” is continuously updated in a “master PRD”. Here’s the prompt he gives the agent:

1. Choose the next feature: Find the highest-priority feature to work on and work only on that feature

2. Have tests pass: check that the tests pass (via pnpm test)

3. Update the master tracker: update the PRD with the work done

4. Log work: append your progress to the progress.txt file

5. Commit: make a git commit of the feature

This style of working is more of a “dynamic Kanban”:

The “Ralph method” is all about working around context window limitations. Back in mid-2025, the maximum size of a context window was around 200,000 tokens. That’s not enough for more ambitious tasks, so it’s necessary to break up agent runs into smaller ones and run them, one by one. In this context, here’s where the Ralph method works:

• Have a goal for a project, and keep running (or re-running) agents until this goal is reached

• Persist work done in a “compressed” manner on the filesystem (as logs or an updated plan)

• Start agents with fresh context to minimize “context rot”

• Allow each agent to add or modify the “masterplan” if needed

2. The /goal command ships in all major harnesses

For a few months, building a Ralph loop meant doing it yourself: setting up the loop, state tracking, deciding how the agent can add tasks and when to stop. But things changed once coding harnesses made it easy to run these loops.

April: Codex ships /Goals

About six months after the “Ralph technique” started gaining wider traction, Codex shipped the “Goals” feature in Codex. From the documentation:

“Goals are persistent objectives in Codex that keep a thread working toward a defined outcome across turns. A Goal gives Codex a completion condition: what should be true, how success should be checked, and what constraints must stay intact.”

Also from the docs (emphasis mine):

“A normal prompt says: do this next thing.

A Goal says: keep working until this outcome is true. In a normal request, Codex works through the immediate instruction, reports a result, and waits. With a Goal, Codex has a durable target attached to the thread. After a turn finishes, it can inspect the current evidence and decide whether the objective is satisfied. If the answer is no, and the Goal remains active and within budget, Codex can continue from the latest state.”

A visual representation:

Here’s an example of using a goal in Codex:

/goal Reduce p95 checkout latency below 120 ms on the checkout benchmark while keeping the correctness suite green

That’s a clear enough “end criteria” to just hand off to the agent, which then breaks up the task, spawns subagents, and runs until complete. So, how did OpenAI build Goals? They used files, logs, running tests, and lifecycle controls:

In this way, “Goals” feels awfully similar to a Ralph loop, except compressed into a single command! It feels like the Codex team took the idea of the Ralph loop, built infrastructure around it, took care of coordinating agents by not having them step on one another, dealt with state, running of tests, starting and stopping agents, and then added functionality such as being able to set a budget.

May: Hermes and Claude Code follow with /goal

Three days later (May 2), Hermes agent, a popular AI agent framework and OpenClaw rival, shipped their implementation of /goal. From the docs:

“[/goal] It’s our take on the Ralph loop, directly inspired by Codex CLI 0.128.0’s /goal by Eric Traut (OpenAI). The core idea — keep a goal alive across turns and don’t stop until it’s achieved — is theirs. The implementation here is independent and adapted to Hermes’ architecture.”

Less than two weeks later – on 12 May – Claude Code also shipped their /goal command. It’s identical in what it does to Codex. From Claude Code:

“The /goal command sets a completion condition and Claude keeps working toward it without you prompting each step. After each turn, a small fast model checks whether the condition holds. If not, Claude starts another turn instead of returning control to you. The goal clears automatically once the condition is met.”

A few months before, in March, Claude Code shipped the concept of scheduling an agent with the /loop command. It is basically what JavaScript’s setTimeout() function would be equivalent to: repeat a task after a given interval until the work is done.

By May, running a Ralph loop had become as simple as giving a single command in one of the major agent harnesses. It’s as if AI labs noticed user demand to do more with agentic loops which were hard to set up, and built ways to make it simpler. For instance, in open source agent harnesses like OpenCode, there are plugins like the /goal plugin. For the minimalist coding agent, Pi, the /goal command can be added as a package.

3. Loops which devs use: triggers and cron jobs

By May, we had access to the /goal primitive. So, what use cases were Boris Cherny and Peter Steinberger referring to in terms of spending most of their time on designing loops, instead of writing prompts? I asked around for examples of “loop engineering” from fellow devs. Based on ~210 replies, mostly from X and LinkedIn, it seems that triggers and cron jobs are two very common use cases for loop engineering:

Triggers / automations: an agent kicks off when an event happens. The event could be an error being logged, a new ticket created, customer support feedback received, etc.

Pre-AI, these events were typically triggered by a webhook, and kicked off things like a Slack bot posting in a channel, triggering a system, or being the starting point of a Zapier or an n8n integration.

Cron jobs: many devs see “loop engineering” as kicking off jobs that involve agents on a cadence. This is fundamentally the same as scheduled cron jobs. From director of engineering, Oded Messer:

“The idea is that strategic workflows that are repeatable and automatable can be done so with an agent. OK. But if my strategic workflow is automatable then it either becomes tactical if the AI is capable enough or it’s just a high level old-school-automation I can set up like a cron or a trigger.

The name suggests simplicity and repetition. Sometimes it feels like AI enthusiasts forgot automation was a thing before LLMs.”

4. Helpful loops for devs

Below are some workflows with AI agents that run on a regular basis as a /loop command, or when triggered:

Development-related work

Open PRs for newly recorded app issues. Software engineer Ivan Pantić:

“App encounters a problem and creates a sentry issue. Then:
→ Cron tells agent to check sentry and open PRs.
→If there’s no active PR, agent tackles the issue and creates one
→ If PRs aren’t reviewed, agent pings the devs via slack

In this flow, there is only one PR open at a time.”

Fix flakey tests. Paul D’Ambra, software engineer at PostHog:

“/loop pull the next flakey test from the trunk API. run it to check if it flakes locally, if it does open a PR with the fix... which netted me 13 PRs to stabilise some of our tests.”

Triaging issues and outages. Software engineer Ivan Abad:

“A new alert/exception pops up in a channel.
→ The agent investigates the issue
→ if it is a code change, implements the change
→ creates the PR
→ pings the human for review.

Same applies for customer tickets or incidents. By the time you get paged and connect to the incident the agent already triaged everything and often located the root cause.”

Review design plans. Artem Nikitin, software engineer at Elastic:

“What I’m finding myself doing often recently is to review design/implementation plans in a loop.

Usually, agents only find a few issues during a normal run and then find more on subsequent runs.

So I’m now asking them to run in a loop until they find 0 new major issues.”

Daily/nightly work

Daily product improvements. Jack D, founding engineer at Schematic:

“We have a loop which reads the logs for the last 24 hours, user feedback and makes PR with fixes - we still review the PRs it makes though!”

Nightly end-to-end test run babysitted by an agent. Utku K, engineering manager:

“nightly e2e runs on the frontend app. When a test fails, an agent investigates first to work out whether it’s a real regression or a false negative. If it’s a real bug, the agent attempts a fix, reruns the test, and keeps iterating until it passes or hits a retry cap and escalates... Either way it lands as a PR ready for review by morning.”

More complex development work

Build new telemetry integrations and verify they work. Lawrence Jones, software engineer at Incident.io:

“A lot of the AI runbook that we use to build new telemetry integrations uses loops such as executing the query, verifying that it executed correctly and iterating on the query plan/output format/whatever until it is happy with the outcome.”

Do a long-running migration, mostly autonomously. Startup founder Rafel Mendiola:

“Recently, I converted my startup’s codebase from a regular React app to a React Native app using Expo.

There are two ways I could have done it:

• In the traditional software engineering style, I could have created a large epic with 50-100 different tickets. I started building myself the infrastructure for that, it felt like it was way too much work.

• What I did instead was create a skill that would let an agent figure out a small to medium-sized piece of work or piece of code to convert, given certain detection mechanisms and guidelines, and then do that conversion and keep track of the migration progress. Then I put that skill on a cron job. This was a lot easier to handle cognitively than managing a large migration plan. It also ran every 30 minutes, not nightly or weekly.”

Productivity-related workflows

Daily tasks executed. Aaron Stannard, creator of Akka.NET:

Read more

For this final part of our series on the tech hiring market during the first half of 2026, I spoke with more than 50 hiring managers, software engineers, and engineering leaders. Thank you to everyone who contributed!

For more on this topic, check out our analysis of what the data says about the market in Part 1 and Part 2 of this mini-series.

Overall, an appropriate description of the employment market as many folks experience it right now would be “weird”. This is a characteristic that’s not easy to see in the data, but is clear from talking to people and hearing their anecdotal, personal accounts of job hunting this year. I think the data in the first two articles of this series failed to capture just how unusual things are. So, in today’s issue, we attempt to shed some light on the weirdness, covering:

1. “Catch-22:” nobody finds each other. Hiring managers struggle to find experienced folks, who barely get any replies when applying for jobs. How’s that work?

2. No trust. Is AI to blame? AI-enhanced resumes read as incredible, but hiring managers often face disappointment. Some places don’t bother reading inbound applications as a result.

3. Hot market for some, but tough for most. For those in AI Engineering, ML, or FDE, the market is incredible. For everyone else, it’s much less great.

4. Higher hiring bar & lower compensation – but not for everyone. Many candidates are unhappy with offers that are the lowest in years. This doesn’t apply to AI Engineering positions or at AI businesses, however.

5. Engineering leader recruitment: also weird for senior ICs. Senior engineering leaders are struggling to find opportunities, or may turn them down in favor of fractional roles or to work on their own startup.

6. US market trends. Folks experiencing the “best market ever” are likely in the US, where a talent shortage is a bigger complaint than it is elsewhere.

7. Trends in the UK, EU, and rest of the world. “Ghosting” is more commonplace than in the US, “fake applicants” a bigger issue, remote roles are going extinct, and more.

For more details on the hiring market, see also:

Part 1: what the data says:

• Software engineering recruitment: trending up, mostly

• Big Tech and publicly-traded companies

• Who’s hiring the most software engineers?

• AI engineering: explosive demand

• Who’s hiring the most AI engineers?

• Is AI engineering replacing software engineering hiring?

Part 2, what the data says, continuied:

• Top AI labs are now more attractive than Big Tech

• Harder for new grads & interns to get hired

• Mobile and frontend demand drops, AI & FDE surges

• AI engineering comp > software engineering comp

• Management’s “great flattening” continues

• Big Tech seniority & tenure keep rising

• Interview preparation signups: what do they indicate?

• Where engineers go after Big Tech

1. “Catch-22:” nobody finds each other

The phrase “Catch 22” refers to a paradoxical problem, whose solution is blocked by the problem itself. The term originated in a famous World War II novel of the same name, and it also describes pretty accurately what I see in today’s tech job market. Hiring managers are saying that highly-skilled talent (typically senior+ engineers) is not available to be recruited, at the same time as experienced, proven professionals find their applications ignored by employers.

What seems paradoxical here is how both can be true. It’s as if recruiters and potential candidates aren’t hearing each other. Of course, there’s some nuance:

Mike Julian, CEO of DuckBill Group, which is hiring software engineers, replied to my post with this observation:

“We get about 1,000 applications a day on inbound and maybe two of them are even relevant to the posting.

I mostly no longer look at inbound seriously because it’s so c***. I’d almost certainly miss a great inbound submission if it came in.

All of our recent hires have been via network and us reaching out to folk on LinkedIn. Biggest hurdle we have to outreach is thin LinkedIn profiles and little other online presence.”

From the other hiring managers we talked with, more themes emerged:

More inbounds than ever & also more noise

“I’ve never seen so many inbounds and strong resumes. I’m hiring for lots of roles; for one software engineering position in Seattle, we have had 800 resumes inbound over a three-month period. I’ve never seen anything like this! These resumes are not low-quality either: they are people who have worked at MSFT, AWS, other large tech companies, and have solid skills.” – Head of Engineering, Series B startup, Seattle, US

“It’s difficult to find good candidates among all the noise. Have had two open senior engineer positions for months, and the only good interviews or offers we’ve extended were in network.” – Engineering Manager, late-stage startup, US

“A glut of vastly underqualified people completely drowning your hiring pipeline. This is what I’m seeing, and it’s making inbound a useless channel.” – Engineering Director, Big Tech, US

“From what I hear from recruiters, every job posted has 1,000+ applicants, and 98% of them are considered unqualified.” – fullstack engineer, Middle East

“We are a small company and gave up on inbound hiring. We got too many AI applications. Much lower signal-to-noise compared to past experience.” – Director of Engineering, Canada

Experienced engineers struggle to get interviews

“Even when you’re good, it feels like there’s a LOT of noise to cut through to get noticed. I’m quietly looking for a new role in devtools, and fit that ‘product engineer’ profile perfectly. I feel like the demand for great people is higher than ever, but I can’t figure out where on that bar I fall.” – Tech lead, seed-stage startup, US

“As an experienced engineer & successful-ish founder I have yet to get to a phone screening in this job market. There is clearly something amiss, just a wall of noise preventing any signal from getting through.” – Software engineer + founder, 8 years experience, US

A staff engineer at a late-stage startup in the US summed things up:

“It feels like everyone who has a good job is holding onto it for dear life, and THOSE are the people we want to hire.”

“Tale of two cities:” in demand or not

A director of engineering at a Big Tech in the US, identifies two distinct groups in the market:

“If you are at the top of your game, have AI experience, and are senior enough, you can write your own ticket. If not, then the job market is tough! This job market is like the tale of these two very different cities.”

Observations from some job seekers echo this: demand feels strong for “AI-adjacent” engineers (those building AI systems), especially in the US. Standout engineers with a strong network are still in demand, and we covered the story of one such person in “How to be a 10x engineer” – interview with a standout dev. For everyone else, it’s a struggle to get an interview!

Referrals: more important than ever?

Several experienced engineers currently on the job market say they only get interviews when they personally know someone at a company:

“Larger companies have to have some serious AI going on to sift through resumes because they are being bombarded. It really is the case that you don’t get your resume seen unless you know people who can vouch for you.” – Technical Program Manager, Big Tech, US

“Referrals are a lifeline. It’s impossible to get interviews for Staff or Principal Eng positions by cold applying. The only interviews I am getting from a cold-apply are Senior-level roles.” – Principal Engineer, 10 YOE, US

From a couple of hiring managers:

“The only interviews or offers we’ve extended were to devs already in our network.” – Software architect at a mid-sized company, US

“I filled an engineering manager position in record time thanks to a person in my network looking for this job. If it would not have been for my network I would probably have been looking for someone for a couple of months.” – VP of Engineering, private equity-funded company, Germany

2. No trust. Is AI to blame?

Hiring managers repeatedly tell us they no longer trust what they read in resumes, and that some candidates even turn out to be fake.

Polished CVs, weak candidates

A common gripe among hiring managers is that resumes are highly optimized by AI, but candidates turn out not to have the experience they claim:

“CVs are high-quality, but the people behind them are not. Almost every resume looks impressive. However, the quality of the conversations does not match it at all.

One recent example: I interviewed a senior candidate who had spent five years at a US-based cloud consulting company, most recently as an architect. I asked which architectural principles or patterns he had used in his projects. His answer was: “Daily standup, sprint planning, and retrospective.” I clarified that I meant from a tech perspective, not process perspective. He confidently replied: “Yes, daily standup, sprint planning, and retrospective.” – Engineering manager, large company, Berlin, Germany

“Ten years ago, we would have appreciated resumes tailored to a role that was posted, now it’s just lazily thrown together with AI.” – Engineering Manager, mid-sized company, Canada

AI-keyword stuffing is rampant, according to one head of engineering in the UK:

“Lots of people are rebranding themselves as senior AI Engineers and demanding much higher salaries. Their resumes now have lots of AI-related keywords mentioned, like RAG, evals, inference… but when digging deeper there is little substance. Many of them are seeking a senior level salary (£90k–£140k) when they are barely showcasing mid-level skills.”

Cover letters are as good as dead, several hiring managers tell us. The reason is that they’re always AI-generated, boring to read, and pointless. More than 18 months ago, we reported on how cover letters were being made redundant by AI in How GenAI is reshaping tech hiring.

An engineering manager at a UK e-commerce agency summarizes what’s happening:

“’Claude; write me a CV that matches this job spec, then auto send’. This seems like the name of the game for most applicants.”

Fake candidates

Does anything encapsulate the challenges of today’s job market for recruiters more succinctly than candidates who do not exist, even when they appear to be sitting in an interview? A year ago, we covered an “AI faker” applicant caught by a security startup, who was potentially a state agent from North Korea.

Such incidents are becoming more common at US, UK and EU companies which hire for remote roles:

• “There are a lot more fake candidates applying, leveraging AI for not just resumes, but also interviews. In extreme cases, the interviews are being outsourced so that a different person shows up for the interview. It feels a bit like playing captcha with them during interviews.” – Senior EM, private-equity backed company, Bay Area, US

• “The second person we interviewed was clearly a North Korean scammer, writing questions into an LLM, reading the response, easily tripped up, and other interviews were background noise in the room.” – Staff Engineer, UK

• “Many applicants that looked a good fit turned out to be someone else in the Asia Pacific region doing interviews with an AI in the background. It’s easy to spot because of the ‘lag’ in a naturally flowing conversation.” – head of engineering, mid-sized company, Germany

Cheating in remote interviews by using AI is also commonplace. A software engineer based in Finland told us:

“A couple of times we saw an applicant who was using AI to answer the questions, and this was sooo weird. I honestly didn’t know how to react, so we decided to cut the interview short. We then shared a note with the recruiters on how to spot this.”

3. Hot market for some, but tough for most

The market appears sharply bifurcated: amazing for AI and a few specialist roles, and a major struggle for everyone else. Some accounts from folks benefitting in the current conditions:

“The market is really good as an (AI) engineer with the right experience. I’m not actively looking but get 2–3 messages a day. When I was hiring for AI engineers at my last AI startup, the market for these folks was terrible. It’s very hard to find good talent, and people that were great on paper did poorly in interviews.” – AI Engineer, 5 YOE, New York, US

“It seems like demand for senior positions is still there, but only if your profile really matches what the company needs. For example, I’ve been working on different data pipelines for a while, and finding such a position is relatively simple. However, breaking into anything else is not! Every startup wants to hire only people experienced in that particular thing they need.” – Software Engineer, 13 YOE, Big Tech, UK

Here’s what two job seekers who are struggling at present say:

“I sent my handcrafted resume to 30–40 positions, and heard back from zero. Eventually, I got an interview after a recruiter reached out via LinkedIn; I don’t know how I would’ve found anything, if it was not for this!” – Software engineer, 5 YOE, Amsterdam, Netherlands

“As a developer without a specific “specialization,” I’m struggling to get any interviews. I’ve spent a few years as frontend, then another few as backend, and most recently working on DevEx. In this market, I just don’t get any callbacks for interviews, not even a first round interview.” – Software engineer, 6 YOE, London, UK

AI/ML/FDE market on fire

We see in the data that AI engineering is seeing explosive demand, and that forward deployed engineers (FDEs) are also seeing a massive spike in demand. For more detail on this, check our deepdive on what FDEs are, and why they’re hot right now.

Anecdotal evidence from job seekers suggests that being in these fields means the market is as good as it gets, right now:

“It’s the greatest job market I’ve ever seen. I’m an L5 former FDE, now SWE, who has worked on LLM apps for ~2 years. The inbound top of the funnel is bonkers, and I find myself saying “no” to places that I would have once killed to work at.” – AI Engineer at an AI decacorn, San Francisco, US

“I have seen huge interest in FDEs and AI Engineers as an engineer who’s been interviewing.” – Software Engineer, 5 YOE, New York, US

“It’s not hard to get interviews as an ML/AI engineer. The technical bar is about the same as late 2022. If anything, onsites seem to be of fewer rounds for the same level. Companies seem to move fast to get candidates onsite.” – ML engineer, ex-Meta, Bay Area, US

An engineer at Apple is surprised by the demand for their skills from AI companies:

“Got a lot of responses from cold applications for AI roles, and ended up with two offers at AI infra companies. I ended up getting a significant pay bump beyond what Apple offers for the same level!”

A software engineer at well-known company in the Bay Area, also finds the job market is good:

“I got offers for Senior SWE with Stripe and Rippling, was rejected from Snowflake… Anecdotally, the job market is better compared to 2025 for regular SWE positions in the Valley.”

EM and Staff+ profiles are near-impossible to fill

A repeat complaint from hiring managers in the US is how challenging it is to recruit solid engineering managers and Staff+ engineers:

“It’s extremely difficult to hire EMs and Staff+ engineers. It’s much easier to hire folks with less than 10 years of experience. This is despite us offering 90th percentile comp via Pave, and having a hybrid and good culture.” – Fractional VP of Engineering, Series B, New York, US

“My team has been looking for a new engineering manager for three months and we barely had any good applicants. It’s a super weird dynamic right now.” – Senior Infrastructure Engineer, Series D Fintech, company with offices in the US and EU

“The most desirable candidates right now are EM or staff-level folks who can keep up with uncertainty in the business. We are having trouble hiring folks who roll with the constant change that is now very typical at startups.” – CTO at a San Francisco-based startup

“Specialist” profiles are hard to recruit

A few examples:

• Distributed systems engineers: “We can’t find enough qualified distributed systems engineers, cloud infra etc. I have 20 reqs [open positions] right now.” – Recruiter at a hyperscaler, US

• Product engineers: “‘Product engineer’ has been a hard profile to find. It is also hard to find someone with a decent design eye who can also build full stack. The hardest thing to hire for has been taste + trust… I’d rather hire someone who is ‘behind’ on AI, but has great taste/judgment than someone with complex agent setups and prompt libraries.” – Tech Lead, seed-stage startup, Los Angeles, US

• Senior engineers, in general: “As a hiring manager trying to hire seniors right now, it has felt pretty difficult. Most applicants are totally unqualified, and the qualified ones do poorly in an interview.” – Engineering manager, robotics company, Canada

Silence for many

One in five respondents detail how difficult it has been to get responses from recruiters:

“Things are looking bleak. Now, applications are going into a void. Recruiters aren’t reaching out immediately as before. Of 10 or so applications sent out, I’ve gotten one interview scheduled that was then rescinded because it looks like the team was laid off.” – Frontend engineer, 10+ YOE, Southeast US

“The job market has been a desert. I’ve gotten back into applying, job agencies seem to have few roles they’re looking to fill. The companies I have applied to myself, never reply.” – Software Engineer, 8 YOE, Southern California

“Recruiters are ghosting me! Every time when I respond to recruiter reachouts, they ALL ghosted me after a few messages and have similar stories from friends.” – Software Engineer, 7 YOE, ex-Meta, Switzerland

“So far, I have a hard time even getting past the hiring manager and I’m rejected before I can do the proper technical assessment.” – Software Engineer, 7 YOE, Helsinki, Finland

Even folks at Big Tech face a lack of response. Here’s a Technical Program Manager currently at Meta who used to be very in-demand:

“I put a few feelers out when Meta announced layoffs, and it’s just radio silence. I don’t want to sound cocky, but with my resume, you at least get a chat with a recruiter, usually right away. But Google and Anthropic just ghosted me.”

4. Higher hiring bar & lower compensation – but not for everyone

In a “normal” market, when the hiring bar goes up, so does compensation. But we heard anecdotes about the hiring bar going up, with the compensation on offer trending down!

Read more

Three weeks ago, at Craft Conference, in Budapest, Hungary, I opened the event with a keynote titled ‘Slow Down to Speed Up’.

As with most of my talks, it came together in stages, including with some input from full subscribers to the Pragmatic Engineer, with whom I shared my thinking in advance, in ‘Ideas: slow down to speed up when working with AI agents’. Thank you for all the comments!

As fate would have it, just two days beforehand, social media giant Meta appositely provided a real-world case study for my talk, with its most embarrassing outage of all time: users could simply ask the Meta AI to change the email of any account, and the bot happily complied – even if the account belonged to someone else entirely – including a former US president. It was a timely example to kick off the talk with. Check out the full keynote that’s available to view on YouTube:

Watch the keynote video

In this article, I summarize the key parts of my Craft Conference keynote in detail, and some responses received at the event. Full subscribers also have access to the slides, here, and at the foot of this article.

We cover:

1. Meta: “AI psychosis” in effect? Meta has been destroying its engineering org, and an obsessive focus on AI seems to be one reason for it. For more on this question, check out this deep dive.

2. Everything’s changed in six months. From around November last year, things changed with a more capable generation of AI agents like Opus 4.5 and GPT-5.4.

3. How are tech companies changing how they work? Anthropic, OpenAI, Google, Uber, startups, and traditional companies.

4. Trends. Individual productivity is up, but team productivity’s flat, tokenmaxxing and tooling adoption, vanishing middle management, CEOs and CTOs back to coding, and more.

5. Trends across software. Falling software quality, GitHub’s constant reliability woes, AI slop overwhelming devs who care about quality, and more.

6. Advice for software engineers and engineering leaders. Suggestions to help future-proof a career.

7. Feedback. “It’s happening here too!” is a common theme, and relief for some that it’s not unique to their own workplace.

1. Meta: “AI psychosis” in effect?

I thought it was a made-up story when I read that Meta had enabled account takeovers via a “zero auth” policy; i.e., simply asking the Meta AI bot was sufficient to change any account’s email address. After all, shipping such a regression would fly in the face of security measures, code reviews, automated testing, and metrics. Plus, the company has dedicated Integrity teams whose mission statement is to ensure something like this never happens… And yet, this bug shipped.

It went undetected by anyone at Meta, and high-profile accounts like that of former US president, Barack Obama, were taken over as a result. Instagram’s dedicated Integrity team seems to have discovered the embarrassing issue via the news.

As mentioned, it was two days before the Craft keynote, so there was enough time to ask around at Instagram and Meta. Engineers at the company there told me this disaster was caused by AI-generated, AI-reviewed code, along with layoffs, and by forced reassignments from Integrity teams and elsewhere onto AI labeling and related duties.

The problem at Meta seems to be that leadership is aggressively pushing AI, while withdrawing resources and headcount from areas responsible for security, quality, and reliability. Since last week’s deepdive into what’s been happening behind the scenes was published, I’ve learned further details:

• Integrity teams at WhatsApp have been hit hard by layoffs and enforced data-labeling reassignments

• Instagram’s design team suffered a 44% cut in headcount during layoffs

• The Developer Documentation and Support team had a full 95% headcount reduction during layoffs

• Data labeling at the ADO group goes beyond “just” labeling; there are many AI training tasks to do. But these are repetitive, unless you get really creative.

Based on everything I heard from talking with Meta folks, AI-induced behavior was indeed at the heart of this outage. AI-generated, AI-reviewed code, and security teams being gutted, were also factors in the beyond-embarrassing incident. As reported in last week’s deepdive:

• Instagram’s Trust and Safety Team lost around 50% of its staff to data labeling and layoffs. Some of the most senior folks were drafted onto AI training tasks.

• AI-generated changes with zero human input, with just an additional AI code review, have been very common in recent months across the codebase. The change that caused this outage looked like one of these

• Normally, the Trust and Safety team would be on top of monitoring and alerting of security breaches, but it is currently in full disarray due to rapid, internal disorganization”.

If major changes like data labeling assignments and staff tracking are undone, then perhaps things at Meta could return to normal. But so far, the most being done is that leadership has boosted budgets for snacks, travel, and events. Hardly the change needed to restore morale and the former culture!

The comparison to the Lumon corporation in the hit show, Severance, was duly made:

Meta’s worst-ever outage can be interpreted as a warning about what happens when there’s so much focus on AI that the basic health of a company’s main – money-spinning – products is neglected. Instagram, WhatsApp, and Facebook generate the bulk of revenue for Meta, but the company is reallocating more engineers to training the coding model, and aggressively cutting the headcounts of vital orgs to do so – up to the point of not having oncall coverage for key services, and security teams being too stretched to do their jobs.

Am I missing some insight about why it’s more important to build a state-of-the-art, likely-closed AI model that’s good at coding, than it is to keep operating revenue-generating businesses with stable infra?

2. Everything’s changed in six months

Independent, experienced software engineers with zero affiliation to AI labs have been saying for a few months that how we do software engineering has been transformed.

David Heinemeier Hansson (DHH), creator of Ruby on Rails in January:

”Just [in] summer 2025, I spoke with Lex Fridman about not letting AI write any code directly, but it turns out part of this resistance was simply based on the models not being good enough at the time! I spent more time rewriting what it wrote, than if I’d done it from scratch. That has now flipped.”

Simon Willison, creator of Django, in May pinpointed the start of the change to late last year:

“The models released in November 2025 elevated agents to being genuinely useful. We’ve had six months to get used to that idea now; it’s no wonder companies are beginning to spend real money on this technology.”

Teams using agents now ship 5x as many pull requests as two years ago. Here’s data from Linear:

Devs using AI harnesses are producing 2.5x as much code versus 18 months ago. Data from Cursor shows that their users, on average, went from adding 3,500 lines of code in January 2025 to 8,600 today:

The size of pull requests is up 3x versus 18 months ago. Also from Cursor:

More AI changes are accepted without human review. Data from Cursor shows a big jump in changes being accepted without human review from around February this year, when Opus 4.7 and GPT 5.5 launched:

We’re seeing a lot more code generated, and less of it than ever being reviewed by devs. In the relatively short time since AI agents became really good last November, there are more pull requests generated by devs, those pull requests are getting better, and code reviews are harder to keep up with. And so, reviews are less stringent and more changes are shipped to production sans human review! As per my discussions with Meta engineers, these kinds of AI-generated, AI-reviewed pull requests [at Meta, they’re called diffs] are what caused the most recent, embarrassing outage at Instagram.

3. How are tech companies changing how they work?

Details from a few larger tech companies:

Anthropic: all-in on AI agents. In March, Boris Cherny, creator of Claude Code, was on the Pragmatic Engineer podcast and shared some details:

• He personally runs ~5x agents parallel, and ships 20–30 PRs/day

• Product requirement documents (PRDs) are dead & prototypes have replaced them inside Anthropic

• ~100% of Claude Code was generated by Claude in March

• ~70-90% of code inside Anthropic was generated by Claude

• Claude Cowork – another billion-dollar product in terms of revenue potential – was built in just 10 days

Since then, Boris has shared that his workflow has changed to setting up loops to run agents.

OpenAI: moving much faster with AI agents. OpenAI’s Codex team was on the main stage at The Pragmatic Summit in February. Tibo Sottiaux (head of engineering, Codex, OpenAI) shared interesting details on how software development is done in the Codex team:

• There’s a “fix this” button integrated into the internal OpenAI mobile app. It makes one-shot fixes to bug reports, which devs review and can merge

• AI code review for all code changes. With a tiered approach, some changes can be merged with just AI review, and more important ones need an extra human review

• Most devs run several agents in parallel, often walking around with their laptop lids open, so the machine doesn’t enter sleep mode and suspend agents

• Code isn’t really written by hand anymore on the Codex team, and is also less common on other teams too

• “Taste” is becoming a core skill for working at the company

• Codex improves itself: it runs its own test suite, runs improvement tasks overnight, and during team meetings it takes actions on topics discussed

Google: AI widespread. Gemini is not as capable at coding as Claude or Codex, as acknowledged by Google’s CEO, but it’s widely used companywide. The less capable coding model could be hurting AI adoption compared to other companies.

Uber: in-house AI infra. We covered in-depth how Uber uses AI for development, touching on internal systems like:

Uber’s MCP Gateway:

Uber Agent Builder:

The AIFX command line interface:

Minion: background agents

Code Inbox:

Smart Assignments as a neat feature of Code Inbox:

Risk Profiles: another smart feature inside Code Inbox:

uReview, Uber’s AI code review tool:

Autocover and Shepherd for large-scale migrations:

Uber is a good case for learning how much of internal developer infra needs to be rebuilt in order to work well with AI agents. Uber built all the tools above because they needed new, better ways to integrate AI agents into the developer workflow, but couldn’t find anything that worked up to requirements. I’d also point out how much time and effort Uber invested in making code review more efficient. Devs are, indeed, getting overloaded with AI code reviews and Uber’s Code Inbox tries to separate the important pieces of code to review from unimportant ones.

Startups are jumping into using AI agents, although their integrations are more basic. In preparation for the keynote, I talked with several startups about their AI usage. Harnesses like Claude Code, Codex, Cursor, OpenCode and others are popular, and I also noticed most startups are heavily integrating AI agents into Slack, so devs can kick off bugfixes or small feature requests straight from the chat tool.

I observed startups being the most likely to experiment with new AI dev tools; from code review, all the way to AI incident management tools.

“Traditional” companies are also heavily investing in AI dev tools. At the recent Pragmatic Summit in San Francisco, Laura Tacho shared interesting details:

• In February, 18,000 Cisco developers used Codex for complex migrations, code review, and refactoring. This was very early – Codex was just starting to gain industry-wide adoption!

• JP Morgan Chase built a multi-agent framework for annotation, using multiple specialized agents to label customer interaction data, and judge agents to aggregate and rank results. These are pretty advanced use cases!

In general, “traditional” companies do not seem to be lagging behind in using, paying for, and adopting AI agents and AI developer tools.

4. Industry trends

There are trends I’ve observed around the adoption of AI dev tools:

Read more

Subscribe now

Many subscribers expense this newsletter to their learning and development budget. If you have such a budget, here’s an email you could send to your manager.

For two decades, Meta had a unique, high-performance engineering org; right up until around April of this year. For the first 20 years of the company’s existence, it had a “move-fast-and-break-things” culture, and in the early 2020s this shifted to a “move-fast-with-stable-infra” one. Engineers I know at the company were empowered to do good work, focus on impact, and to balance business interests with solid engineering.

But in the past few weeks, all that has changed, as if the leadership has been following detailed blueprints on how to demolish a proven, successful engineering culture in the most ruthlessly efficient way possible.

For the past few weeks, I’ve been sharing how bad things are inside the social media company for engineers in one of Silicon Valley’s most prestigious workplaces. In this article, we walk through what’s happened, and ask what’s going through the minds of leadership who are reducing software engineering there from the profit center that it was between 2004 until very recently, to the disdained cost center that it has become in just a few weeks.

We cover:

1. Meta’s pre-AI engineering culture

2. Investing in AI and pressing engineers to always use it

3. Core engineering folks feel treated like trash

4. Most embarrassing-ever outage

5. Internal mess

6. Self-inflicted wounds

7. Is “AI psychosis” just a Meta issue?

1. Meta’s pre-AI engineering culture

I’d split Meta’s engineering culture into two eras: “move fast and break things”, and then “move fast with stable infra.”

“Move fast and break things”

In the 2010s, Facebook’s unconventional engineering culture had grown somewhat legendary in the tech industry, as the company went against conventional best practices and succeeded massively.

In 2012, when Facebook hit the billion-users landmark, the company produced a small physical book about its culture which was placed on employees’ desks. Presented with retro propaganda design, it was dubbed the “little red book”, co-opting the name of a famous volume of the thoughts of Chairman Mao, (1964).

At around 70 pages long, Facebook’s version codified its engineering culture: speed, fearlessness, taking ownership, and thinking outside of the box.

Back then, mantras in Facebook’s little red book were also in print across campus, and included:

• Move Fast and Break Things

• Done is Better Than Perfect

• Fail Harder

• What Would You Do If You Weren’t Afraid?

• Every Day Feels Like a Week

• The Wright Brothers Did Not Have Pilot Licenses

• The Foolish Wait

• Fortune Favors the Bold

There was genuine focus on building good products. Also from the book:

“Move fast with stable infra” culture

In 2022, I did what is one of the longest deepdives we’ve published on the topic of Meta’s engineering culture. By then, things had evolved, and much of any former recklessness was gone, replaced by the principle of moving fast, but with stable infra. Here’s how I described Meta’s engineering culture then:

“The culture is incredibly engineering-centric: much more than most of Big Tech. This might come from Mark Zuckerberg being an engineer himself, or because much of the innovation in the early days of Facebook came from engineers.

Focus on individual impact. Impact has been the bread and butter of the focus at Facebook. This is very true since the early days, and the focus on generating impact remains.

One detail in common with most Big Tech firms is that both the engineering culture and general culture focus so much on individual impact. This results in some people focusing on short-term, measurable wins and assuming that teamwork and split wins between groups might be less rewarded.

The lack of rigid processes. Facebook seems to have the least amount of processes or standardization across all of Big Tech. Don’t even try to compare it to Amazon’s engineering culture and the countless formal processes there. But even compared to companies like Google, Microsoft or Uber, Facebook’s processes are much looser. Most of this comes from the engineering-centric nature of the company and engineers disliking processes.

Surprisingly little emphasis on testing, documentation or code comments. You’ll find shockingly little automated testing and documentation at Facebook, compared to the rest of Big Tech. Inline code comments are also very rare.

A founder-engineer driven company. Facebook is one of the few Big Tech firms whose founder is an engineer, and still is the CEO. Netflix is the other one where founder and co-CEO Reed Hastings was also a software engineer before starting the company. Amazon was the other example of this until recently, but it’s not the case at Google or Apple. There are good examples of smaller companies like Cloudflare, but they’re all younger than Facebook.

Bootcamp. A unique onboarding process, unlike what any other Big Tech firms offer. We cover this more in the Bootcamp & onboarding section.”

Also, Facebook, as a product, has one of the most sophisticated auto rollout systems in the industry. Instagram has a battle-tested infrastructure where it was almost trivial to launch a new social network (Threads) with 100 million users served in its first week.

Engineers whom I knew inside the company are capable, motivated, and product-minded, and their work was appreciated. CEO, Mark Zuckerberg, was influential: he personally coded the first version of Facebook, had stayed close to engineering, and valued software engineers very much. Engineers there felt they were working inside a profit center.

2. Investing in AI and pressing engineers to always use it

Meta has been the only company among the big five of Apple, Microsoft, Amazon, Google, and itself not to own a hardware platform or operating system. Apple has the iPhone, iPad and Macs, Google has Android, ChromeOS and Pixel phones, Microsoft has Windows, and Amazon has the Kindle.

Stepping back, it looks as though the Mark Zuckerberg of today has resolved not to miss a platform opportunity, after the company failed to build its own mobile OS or mobile phone during the 2010s.

This is one reason for investing so much in virtual reality (VR) with Oculus, and in augmented reality with the Meta Glasses. Facebook changed its name to Meta in 2021, back when it looked like VR – and the metaverse – could be massive. Billions was spent on ensuring Meta would be the market leader in this space. But once again, VR didn’t go mainstream; since the end of the pandemic, popular interest in the segment has died down considerably.

When it became clear that AI would become a mega-trend in 2022, Zuckerberg didn’t miss it: he assembled the internal FAIR group (Fundamental AI Research team) as well as a GenAI product organization and released a series of open-weight AI models:

• Llama 1: released in Feb 2023, three months after ChatGPT, built by FAIR

• Llama 2: in June 2023, built by the GenAI product organization (as well as all subsequent Llama models)

• Llama 3: in April 2024. This model was Meta’s most competitive LLM of all, and gained momentum in adoption across the industry

• Llama 4: in April 2025. This model was deeply disappointing

In June that year, Meta acquired a 49% stake in Scale AI to reboot its AI efforts for a whopping $14.8B, and brought in Scale AI’s CEO, Alexandr Wang to take over Meta’s AI strategy. The acquisition of Chinese startup Manus AI for $2B is currently in question after China blocked the deal from being completed.

Based on the investment made into Scale AI and Wang, it’s pretty clear that Meta – and Zuckerberg – is determined to build a state-of-the-art LLM that can be competitive with the latest versions of Claude and ChatGPT. But Meta has to start pretty much from scratch, and it’s up to Alexandr Wang to deliver.

Scale AI brings in a very specific kind of expertise to Meta, as one of the best in the industry in:

• Training data and labeling: Scale started, and is still best known, as a provider of high-quality labeled datasets for machine learning and AI training, including code, text, image, video, etc.

• RLHF and fine-tuning: A RLHF (reinforcement learning from human feedback) flow which Scale runs, where people give feedback for foundation models, as a “human in the loop” data engine that many leading AI labs use to create better LLMs.

Wang seems to have a very broad reign to do what he has been an expert in: creating training data, doing data labeling and RLHF. This is being pulled off with the labor of Meta’s engineering workforce, and by surveilling it.

Problem #1: Tracking keystrokes and mouse clicks, with no option to opt out. In late April, Meta told engineers they were being enrolled into a system that tracks every keystroke and click, to produce training data for Meta’s new AI. There’s no way to opt out.

Needless to say, this is invasive and raises privacy questions: If you log into your personal bank account, does the tool track you? What about when you’re writing a personal email, or responding to a personal call? Meta held no consultation and there are no workarounds; just a top-down decision being pushed through.

This month, Reuters reported that people’s concerns there are finally being heard:

“Meta is dialing back elements of its plan to collect employee mouse movements, keystrokes and other actions for use as AI ​training data, it said in an internal memo on Tuesday, following weeks ​of angry pushback from staffers.

New controls will allow employees to pause ⁠the data collection for up to 30 minutes at a time and ​request exemptions from the initiative, according to the memo, authored by Stephane Kasriel, ​a vice president in Meta’s AI model-building Superintelligence Labs unit.”

From talking with current Meta engineers, I understand the logging system has not been rolled out in the UK due to data protection regulation.

Problem #2: 30-50% of engineers on core teams have been forcefully reassigned to data labeling and RLHF, upsetting folks even more. Also starting in late April, product engineering teams received a mandate from above, whereby 30-50% of engineers were to leave the team and join the ADO org (Agent Data Optimisation).

“Forceful” reassignment is very relevant here because of Meta’s traditional engineering culture. Between its founding in 2004 and until last year, Meta gave engineers autonomy to choose where they work and what they work on. This was structural to how the company worked:

• Engineers were not hired for a specific team (save for at the Staff+, levels, in some cases). They were hired to the company

• During a 6-week bootcamp, new hires got familiar with Meta’s engineering culture and chose a team

• Team matching meant talking with multiple teams who had headcount, doing small work with them, and finding a match

• Internal transfers were easy, and often initiated by engineers

Team selection via bootcamp started to die down in around 2024, but any Meta engineer with at least two years’ tenure knows that previously they chose what to work on, and of course, could pick the most impactful thing to work on. And then, out of the blue, they’re assigned to a division where the impact is not clear, the work is menial, and doing it too long will surely hurt their career prospects.

“Data labeling” is more involved work, even though a bit repetitive. There are labeling tasks, where you create a website, then look at it and decide if it looks good or not, and give this feedback: this is the “typical” data labeling. But this is the less frequent type of work. But then there are more involved AI training tasks, which looks like this:

• Come up with a task that the AI should do

• Then write the tests that confirm the result

• Package all of this up into a Docker container, using Harbor framework

• Then read the code that the AI writes - often doing this based on feedback from several models, and give it feedback

This work is not easy to do — and you can see why you need good software engineers to do it! — but it gets repetitive really quickly. Most engineers who I talked to said that they found it hard to do this work motivated day-in, day-out. Then again, I did talk with an engineer inside the org who said that by varying the technologies they use, and challenging themselves, they found this work motivating and interesting! This engineer expects there to be more software engineering work coming up, after the phase of training passes. There’s a lesson here: when life gives you lemons, you can either complain about it, or you can make lemonade, like this dev has done so.

This training work is secretive across the AI industry, and vendors are offering $100+ per hour for devs to do this type of work. There are rumors across the industry that OpenAI and Anthropic spends more on these training environments (to make the models great at coding) than the model training run itself!

Infrastructure and security teams were hit especially hard by reassignments. I talked with several engineers in infra orgs, who had 30-50% of their teams drafted into the ADO org. And in some cases, it was the best engineers who left.

One engineer told me that the whole situation feels like the movie, The Hunger Games, when tributes are randomly selected and then removed from their environment, to something completely different. Except, at Meta, many more folks are being affected, with between three and five from a 10-person team going from building products used by hundreds of millions, to giving human feedback on AI-generated GitHub repos, over and over. So, a wider impact than in the Hunger Games, but with less drastic consequences.

Around 6,500 people are in the ADO org, more than at OpenAI and Anthropic. Roughly four to five thousand of these are software engineers. Meta has around 25,000 engineers, meaning that one in every 5-6 software engineers may now find themselves doing data labeling full time.

As you can imagine, people are actively open to new positions, and nobody is updating their job title on LinkedIn and elsewhere to “data labeling at Meta.”

I’ve spoken with people in this role and they don’t like doing it, and feel upset about the top-down decision making. The silver lining is that they still have a job, have retained their salary, and were not part of layoffs. They still have time to leave Meta for something that pays comparably and is not a data labeling job.

3. Core engineering folks feel treated like trash

Problem #3: a month-long waiting game, stoking fear across the company. On 20 April, Reuters reported that Meta planned to lay off 10% of staff in a month’s time, and Meta confirmed the news, meaning there was a period of four weeks when everyone knew that they could be unemployed very soon.

Forced reassignments to data labeling started to happen. As I covered at the time:

“Understandably, there are mixed feelings about this redeployment [to data labeling], with layoffs coming soon. On Wednesday, 20 May, Meta will announce layoffs. Perhaps those moved to do data labeling could actually be “safer” than colleagues on product teams. Of course, this is speculation, but it would be cruel if Meta cut devs reassigned to data labeling.”

Problem #4: Performance review is hyper-aggressive at Meta, so devs optimize all metrics. The internal performance review process, PSC (Performance Summary Cycle), is very stringent, compared to Google and Apple, I’ve learned. Managers inside Meta “fight” over the pay packets of their employees, which involves “knocking down” the packet of engineers on other teams, so their direct reports are ranked higher. It’s common to weaponize metrics in this process – be that business impact, the number of code reviews, number of lines of code written, pre-AI (see our Coding Machine archetype podcast on this.)

Quotas are handed down to managers for the splits of the workforce to be put in each ‘bucket’, and the internal politics gets heated as managers try to get their reports into higher buckets.

After a few years, engineers at Meta learn that the best way to not get a bad PSC rating is to have all metrics – impact, code committed, and other numbers – higher than their peers’ are. Learn more about the internal politics of performance calibrations.

Problem #5: tokens are measured as part of perf, so devs aggressively optimize for it. When layoffs were confirmed, engineers also learned that managers shall inspect token count during perf reviews. This raised worries that those with low token counts might be marked as underperformers and dismissed.

So, what is the natural reaction to this as an engineer at Meta? They started using AI tools for the sake of generating more tokens. This happened while Meta had an internal token leaderboard, encouraging tokenmaxxing. As I wrote on 16 April:

“As per The Information, Meta employees used a total of 60.2 trillion AI tokens (!!) in 30 days. If this was charged at Anthropic’s API prices, it would cost $900M. Of course, Meta is likely purchasing tokens at a discount, but that could still come in at $100M+ – in large part from senseless “tokenmaxxing”.”

The biggest problem: people stop caring about real work and focus on performative work. Let’s check the four ingredients that Meta’s leadership has decided to introduce to their workplace:

1. Tracking the keyboards and mouse clicks of all engineers, where legally possible

2. Reassign a good chunk of engineers to fulltime data labeling

3. Let staff know that 10% of them will be laid off

4. Have a culture where devs optimize for any and all metrics measured during PSC

5. Measure token usage as part of PSC

Shake this mix up well, and what do you get? Two things:

1. Everyone overuses AI to boost their personal stats. An engineering workforce that pretends to work with as much AI, and as little human input, as possible. It’s a strange incentive where an outage caused by a failure to review code properly is not grounds for dismissal, but writing code by hand – instead of having an AI agent write it – could cost you your job

2. Every longer-tenured engineer is seeking a new job, or at least considering it. Those who have been around at Meta longer term have seen enough. Let me describe this visually:

Fresh data seems to confirm that starting in May, a lot more engineers at Meta are looking for an “out.” Here’s how signups to leading Big Tech interview preparation service interviewing.io from Meta have changed over the last year and a half: we can see a massive jump in May vs the year before, as shared by founder and CEO Aliner Lerner with us:

(We covered more data from interviewing.io and other interesting data sources in the deepdive State of the software engineering job market in 2026)

To its credit, Meta has given out generous retention equity packages to several engineers considered key on the remaining teams. These packages make it harder to get matching compensation elsewhere. Still, I talked to one engineer who got an equity top-up, and said that this approach helped him decide to leave as soon as possible because he feels bitter about the lack of autonomy and having no control over things.

4. Most embarrassing-ever outage

Meta’s core infra and security teams have suddenly found themselves severely understaffed. Most folks are pushing AI-generated code merged with AI-only reviews, without paying much attention to quality. After all, they’re dealing with the possibility of unemployment, while firefighting to operate a team without its best engineers whose headcount has been cut in half, all with the knowledge that AI usage could affect their own job security.

Two weeks ago, on 30 May, the most embarrassing outage in Meta’s history happened. Here’s software engineer Siddharth Sundharam’s summary (emphasis mine):

“Yesterday, a slew of Instagram accounts, including some high profile ones like the Obama White House account, seemingly got hacked.

Look, I’m no spring chicken. I’ve spent almost a decade and a half identifying vulnerabilities and exploits at unicorn scale, but this is hands down the most unserious, “almost too stupid to be true” of them all.

The Takeover Flow:

Step 01: Faking the Location & Initiating Support. All the attacker needs to kick this off is your account username. Then, they hop on a VPN or proxy close to your city so Instagram’s security algorithms don’t suspect a thing. (You can quite easily get this from your public profile or “About” section or a hundred other ways.) Once it looks like the request is coming from the correct region, they tell the Meta support AI that the account is hacked and ask it to send the verification codes to an arbitrary email address they control.

Step 02: That’s It. Really, that’s it.

The first proper zero auth password reset I’ve seen in production. There appears to be no additional check as to whether the email being given is actually something the user has used before. Once the AI sends the security code to the attacker’s email, the attacker passes it right back to complete the verification. The platform hands over a fresh password reset link, granting full ownership to the attacker.”

This is a security breach in which Meta left its extra-secure, reinforced front door unlocked, so that anyone could come in, and there was no alarm to notify anybody when it happened! It seemed Meta only noticed when users started reporting it on social media!

From talking with folks inside Meta, I’ve learned that AI was at the heart of this outage. AI-generated, AI-reviewed code, and security teams being gutted were together the cause of this beyond-embarrassing incident. I poked around, and here’s what I gathered:

• Instagram’s Trust and Safety Team lost around 50% of its staff to data labeling and layoffs. Some of the most senior folks were drafted onto AI training tasks.

• AI-generated changes that saw no human input, just another AI code review, were very common during the last two months, across the codebase. The change that caused this outage looked like one of these

• Normally, the Trust and Safety team would be on top of monitoring and alerting of security breaches, but it is currently in full disarray due to rapid, internal disorganization.

Meta’s Chief Security Officer resigned the very next day. The outage was resolved on Monday, 1 June, and an investigation started as part of the SEV process. On Tuesday, Meta’s Chief Information and Security Officer (CISO), Guy Rosen, announced his departure.

Coincidence? I suspect not: the CISO might have stepped down if they warned against the Security org being gutted but were then ignored, and so no longer trusts leadership. I also imagine the CISO didn’t have the idea to move a good half of Instagram’s security team over to data labeling.

As a note, the previously worst outage was all Meta services going down for seven hours in 2021, due to a DNS / BGP configuration issue. It was a bad outage, but Meta handled the follow up well, in my opinion. After that 2021 outage, Meta shared a postmortem and apology. It has not done so for the latest Instagram account takeover outage.

5. Internal mess

Wired shares more details on just how bad the situation is inside Meta, right now:

“Someone interrupted a livestreamed, employee-only presentation at Meta earlier this week with an expletive-filled outburst about “being the company’s bitch,” according to a recording heard by WIRED. The individual then asked the people leading the call to write to a specific Meta AI executive and “tell him that he’s a piece of shit.”

The incident, which took place on a call open to thousands of employees, reflects growing frustration inside the company’s Applied AI team, which was formed in March to support the work of AI researchers at Meta Superintelligence Labs. Three current employees tell WIRED there is widespread dissatisfaction with how Meta assembled the unit of about 6,500 engineers and product managers and the drudgework they allege they have been assigned to improve AI models.

“It’s literally the gulag,” one of the employees claims. “You have zero purpose in life all of a sudden, you barely interact with anyone, you just have these tasks every week.”

There’s more: Meta’s Chief Product Officer, Chris Cox, reportedly admitted to staff that Meta’s upper leadership (the folks above him, meaning the C-level at Meta) created the mess. Also from Wired:

During a meeting this week open to all employees at Instagram, Meta chief product officer Chris Cox addressed the “difficult” and “brutal” environment created by the “insanity of this company” in the past few months, according to a recording heard by WIRED. Cox applauded Instagram employees for launching features and serving around 2 billion users amid what he compared to “running a marathon in the middle of a hailstorm and then, like, your teammate gets replaced and then we’re recording you.”

“It’s like what the fuck,” he said, drawing laughs, before repeating himself. “It is like what the fuck.”

6. Self-inflicted wounds

So, is there an ultimate source of the “insanity of this company”, as CPO Chris Cox put it? Engineers whom I talked to point the finger at two individuals: Mark Zuckerberg and Alexandr Wang. Zuckerberg has full control over the business, and has made the decisions to reallocate a good part of engineering folks to data labeling, to roll out tracking software, and to lay off 10% of staff when Meta achieved record revenue and profits. As the CEO, the buck clearly stops with him.

But it’s hard to unsee that – outside of layoffs – everything that Meta is doing is taken from the Scale AI playbook, and that surely comes from Wang:

• Mandatory keystroke and mouse tracking to generate training data

• Forced data labeling with 4,500+ engineers is to generate high-quality RLHF, surely for Meta’s under-construction coding LLM

• Taking away the best engineers from the heart of the business is surely signed off by Mark Zuckerberg, believing that it is more important for Meta to train a coding AI than it is to operate its core business like Instagram, Facebook or Messenger reliably. Oh, did I mention that on Saturday (12 June) Facebook and Instagram had another SEV0, that is, a full-on outage?

Before all this happened, Meta was on track to overtake Google as the world’s #1 ads business by the end of the year. But for some reason, Mark Zuckerberg decided that building a coding LLM is more important.

Meta’s leadership is now trying to undo all the damage they have done. Wired reports that Meta CTO, Andrew Bosworth, admitted to staff that the AI reorg was atrocious and committed to better communication in the future.

To me, it looks obvious that Zuckerberg doesn’t care how engineers feel about the massive changes, and that Bosworth likely ignored the chaos, all while engineers know for a fact that the next AI model matters more than they do to the business. Bosworth also said that employees will have access to AI coaching tools. Very considerate, given the situation!

Based on all I’ve learned, Meta’s engineering culture is dead because leadership has made it clear that engineering at the company is a cost center.

Needless to say, I hope my assessment is way off, but I’ve seen nothing yet from Mark Zuckerberg and Alexandr Wang – the two executives creating this current mess – to suggest it is. There may be a short time period where, if major changes like data labeling assignments and staff tracking are undone, then things at Meta could return to normal. The longer the current conditions persist, the more tenured engineers will surely leave.

7. Is “AI psychosis” just a Meta issue?

It’s tragic to see a technical founder at Meta so focused on AI that he neglects the engineers who built the heart of his company. But is Meta a one-off exception?

Mitchell Hashimoto (creator of Ghostty, founder of HashiCorp) says he is seeing similar behavior by other founders (emphasis mine:)

“I strongly believe there are entire companies right now under heavy “AI psychosis” and it’s impossible to have rational conversations about it with them. I can’t name any specific people because they include personal friends I deeply respect, but I worry about how this plays out.

I lived through the great MTBF vs MTTR (mean-time-between-failure vs. mean-time-to-recovery) reckoning of infrastructure during the transition to cloud and cloud automation. All those arguments are rearing their ugly heads again but now it’s... the whole software development industry (maybe the whole world, really).

It’s frightening, because ‘psychosis folks’ operate under an almost absolute “MTTR is all you need” mentality: “it’s fine to ship bugs because the agents will fix them so quickly and at a scale humans can’t do!” We learned in infrastructure that MTTR is great but you can’t yeet resilient systems entirely.

The main issue is I don’t even know how to bring this up to people I know personally, because bringing this topic up leads to immediate dismissals like “no no, it has full test coverage”, or “bug reports are going down” or something, which just don’t paint the whole picture.

We already learned this lesson once in infrastructure: you can automate yourself into a very resilient catastrophe machine. Systems can appear healthy by local metrics while globally becoming incomprehensible. Bug reports can go down while latent risk explodes. Test coverage can rise while semantic understanding falls. Changes happen so fast that nobody notices the underlying architecture decaying.

I worry.”

The takeover outage at Instagram was exactly like this: the engineering team dropped the quality bar for AI-generated and AI-reviewed code, probably expecting that they could recover quickly from failures. And they did indeed recover… after the damage was done, high-profile Instagram accounts were hacked, and the system was compromised, all very publicly.

Mitchell highlights the specific concern of founders over-estimating the capabilities of AI, and consequently casting aside sensible safeguards when shipping code to production.

Takeaways

Most of us probably have something to learn from the disastrous events at Meta caused by hyper-focus on AI to the exclusion of people who are the lifeblood of that company. In some good news, I’m hearing that in the UK, some of the 10% layoffs have suddenly been cancelled: at the end of the mandatory consultation period, several infra and security teams are learning that no engineers on their team will be let go, as originally expected.

Meta has a booming business, and is already a beneficiary of AI via increased ads revenue. Meanwhile, my Facebook feed is filled with fake, AI-generated videos, with hundreds of comments from bots and people who seemingly don’t realize it’s AI. It all seems like just more content for Meta to show ads next to.

And yet, despite business booming, Meta’s leadership has gone on a crusade to inflict the most damage possible on its engineering org. Apparently, they’re now learning that most of it was pointless.

If you’re in a leadership position and feeling the temptation to make drastic org changes for AI-related reasons, take a deep breath and see where it left Meta. Meanwhile, If you’re an engineer at a company whose leadership is over-indexing on AI, consider forwarding this article as additional context.

If you’re hiring standout engineers who are extremely hands-on with AI, then it’s never been easier to get talent from Meta, than right now. Every engineer I know at the company is an extremely early adopter of AI, and knows how to build products and AI infra. These folks have soured on the company and its leadership. Meta’s loss of talent will be the gain of other startups and the rest of Big Tech; it’s one benefit of AI that’s probably a bit unexpected – not least of all by Meta!

It seems like the old mantra of “move fast and break things” has now extended to Meta’s engineering org itself, with the company’s rush to over-invest in AI, so it will avoid missing the latest mega-trend in the tech industry.

First of all, a big thank you to partner teams for sharing exclusive details for this deepdive:

• Interviewing.io: anonymous mock interviews with engineers from top companies. Thanks to founder and CEO, Aline Lerner.

• Workforce.ai, built by Live Data Technologies, which monitors 1M+ job changes and 300M+ employment validations monthly, across companies, roles, levels, functions, industries, and locations. Special thanks to Alex Hamilton for his input.

• SignalFire: a VC firm with a standout data analysis team. Ordinarily, their data is used to give their portfolio companies a major commercial advantage, but they made an exception to share some for this article. Special thanks to Asher Bantock.

• TrueUp: a platform that scans every open job in Big Tech, top startups, and scaleups, typically paying in the top two tiers of the trimodal software engineering compensation model. Thanks, Amit Taylor.

Today, we cover:

1. Top AI labs are now more attractive than Big Tech. Anthropic is most in demand among job candidates for interview preparation services. Along with OpenAI, it’s almost certainly the place with the most competition for jobs in tech.

2. Harder for new grads & interns to get hired. Data shows that intern intakes have fallen, even as software engineering recruitment recovers. Large tech companies take on half as many interns as before, and candidates’ work and educational backgrounds matter more than ever.

3. Mobile and frontend demand drops, AI & FDE surges. Frontend engineer titles are disappearing fastest across the industry, followed by native iOS and Android ones.

4. AI engineering comp > software engineering comp. AI engineers are more in demand than software engineers, and get higher compensation offers, especially with equity. At the 80th percentile in the US, $300K+ base salaries are the norm now for senior engineers.

5. Management’s “great flattening” continues. There are fewer engineering managers for each engineer across the industry, and fewer VP of engineering and director of engineering posts at Big Tech.

6. Big Tech seniority & tenure keep rising. Since the end of zero interest rates in 2023, it’s as if there’s fewer ways to tempt Big Tech workers to switch jobs, so they remain in situ.

7. Interview preparation signups: what do they indicate? Companies doing mass layoffs tend to see the biggest surge in devs signing up to practice interviews. A list of the top 20 companies from where engineers are preparing to interview elsewhere.

8. Where engineers go after Big Tech. From Amazon, they go pretty much everywhere. From Google, Apple & Meta, it’s mostly to AI labs. Microsoft is where the most ex-workers become their own bosses by working for themselves next.

As a reminder, in Part 1, we covered:

• Software engineering recruitment: trending up, mostly

• Big Tech and publicly-traded companies

• Who’s hiring the most software engineers?

• AI engineering: explosive demand

• Who’s hiring the most AI engineers?

• Is AI engineering replacing software engineering hiring?

See Part 3 for stories from hiring managers and job seekers, covering:

1. “Catch-22:” nobody finds each other

2. No trust. Is AI to blame?

3. Hot market for some, but tough for most

4. Higher hiring bar & lower compensation – but not for everyone

5. Engineering leader recruitment: also weird for senior ICs

6. US market trends

7. Trends in the UK, EU, and rest of the world

Let’s get into the latest data:

1. Top AI labs now more attractive than Big Tech

In Part 1 of this mini-series, we cover the exploding demand for AI engineering:

AI engineering job openings have increased 60% in the past year at top companies, while software engineering openings grew by 7% in the same places. We also found that Big Tech is significantly growing AI engineering headcount:

Anthropic: most in demand

New data suggests that the two biggest AI labs are attracting the most candidates to apply for their AI engineering roles, which is pretty predictable.

Interviewing.io is a job interview preparation service which offers coaching for clients who are getting ready for interviews at specific companies. Based on the number of mentions by clients, Anthropic is the one most candidates are preparing for with paid coaching, and it’s not even close:

It’s also notable that OpenAI (16% of candidates) gets around the same share as Google (17%) and other large tech companies (17%). Combined, Anthropic and OpenAI account for 51% of all interviewing.io coaching requests. For context, interviewing.io only added coaching for frontier labs this year!

There are a few potential causes of the surge of interest in Anthropic:

• OpenAI replaces Anthropic as AI supplier for the US’s novel ‘Department of War’. In early March, the US Government controversially declared Anthropic a “supply chain risk”, and appointed OpenAI as its AI supplier, after Anthropic raised concerns about the future use of AI in mass surveillance and fully autonomous weapons. This raised suspicions that OpenAI agreed to cross ‘red lines’.

• Anthropic’s market dominance continues. Claude Code is the most popular developer tool, as found by our AI tooling survey in February. It seems little has changed.

• Anthropic’s value exceeds OpenAI’s. In March, Anthropic raised a $65B funding round at a $965B valuation, making it more valuable than OpenAI for the first time.

• Anthropic files to go public first. Last week, Anthropic filed to go public, beating OpenAI which has done so a week later.

Anthropic also recruited the most in-demand AI researcher, Andrej Karpathy, in May. My sense is that between the two labs, Anthropic has more momentum for the time being, and has perhaps acquired a ‘halo effect’ with its seemingly principled stance. It’s not surprising that it’s attracting more candidates.

Where are AI labs hiring from?

We looked into the sources of recruits to the three most in-demand AI labs: Anthropic, OpenAI, and Google DeepMind. Here’s what we found:

Where Anthropic hires from, in order of popularity:

1. Google (often Google DeepMind)

2. Meta

3. Stripe

4. Microsoft

5. Amazon (AWS)

6. Databricks

OpenAI:

1. Google

2. Meta

3. Apple

4. Stripe

5. Statsig (after OpenAI acquired Statsig)

6. Microsoft

7. Amazon (mostly AWS)

8. Databricks

9. Airbnb

10. NVIDIA

Google DeepMind:

1. Internal transfer

2. Meta

3. Microsoft

4. Amazon

5. Windsurf

Anthropic has the highest retention rate of all AI labs. Data from SignalFire found the 2-year retention rate (percentage of employees who stay 2 years) is:

• OpenAI: 67%. This is consistent with the rest of Big Tech

• Google DeepMind: 78%. Well above the rest of Big Tech

• Anthropic: 80%. Standout, industry-wide!

Consistent with SignalFire’s 2025 finding, OpenAI 2-year retention was 67% (FAANG-level) versus Anthropic (80%), and DeepMind (78%).

2. Harder for graduates & interns to get hired

It’s well known that it’s getting harder to be hired as an early-career engineer, and new data underlines this.

Intern intakes down since 2022

Live Data Technologies looked at software engineer vs engineering intern hiring trends at 30-80 US-based tech companies, pinned to 2019 hiring numbers (100% being that year’s total number of hires). The spread is wide because Live Data Technologies selects the top few dozen companies that meet their criteria for a “large public tech company” in their database.

The findings:

Zooming into intern hiring, here’s a visualization of it as a percentage of all appointments:

Alex Hamilton, analyst at Live Data Technologies, says:

“We’ve seen overall software engineering hiring start to come back since the 2023 tough market. However, intern intake just kept falling alongside it, which isn’t what you’d normally expect.

Historically, intern programmes have tended to bounce back pretty quickly once companies start hiring again. That hasn’t happened this time, and 2024 and 2025 are the first years in the series where the two lines are moving in opposite directions.

Where you do see companies holding intern intake steady or growing it, it’s almost always a reflection of where they are as a business, be that earlier-stage or faster-growing companies, rather than any kind of broader market recovery”.

Graduate jobs trending down

Anecdotally, we hear new grads continue to have a hard time finding a position. Our new recruitment data on major US tech companies confirms it:

“New grads” in this data are software engineers who graduated less than a year before getting a job as a software engineer. In 2025, just one in 10 engineering hires at larger companies were recent grads, down from nearly three in 10 in 2023.

Pedigree matters more for new grads

We looked closely at the places from where new graduate software engineers are joining US-based tech companies, and found the share of successful candidates from “elite” universities is growing:

By “elite” universities, we mean one of the top 20 US colleges for computer science, such as MIT, Stanford, Carnegie Mellon, UC Berkeley, Harvard, Caltech, Georgia Tech, and Cornell.

Obviously, the influence of these places’ reputations is not a new thing, it’s what makes them “elite” universities, after all. But with new grad hiring down across tech, even graduates from these universities can expect fewer opportunities than before.

Even so, the pedigree that comes from graduating from a well-known university, or doing an internship at a well-known company, becomes ever more significant as the job market tightens.

3. Mobile and frontend demand drops, AI & FDE surges

Here’s interesting data showing the shifting prevalence of job titles on sites like LinkedIn over time:

Some takeaways:

• AI engineering’s on fire. This is not surprising and is evident throughout our study.

• Forward Deployed Engineers (FDE) are growing rapidly. We covered the sudden demand for FDEs in 2025, and this year we’re seeing the FDE role heat up again.

• Modest increase in sales engineers: Sales engineers help close large, B2B-type, deals, and are typical at companies selling to enterprises. The rise in prevalence of this position suggests more companies are targeting enterprise-scale clients. Also, my sense is that FDEs can operate like sales engineers.

• There are fewer native mobile engineers. In 2022, I observed a drop in demand for native iOS and Android engineers. Cross-platform frameworks being more capable today may contribute to fewer places investing in native applications, and a fall in demand for this discipline overall. Is the “golden age” of native mobile development over, with its standalone native iOS, native Android, and web teams for a single product?

• Frontend-only engineers are disappearing. This is one of the most interesting trends in the data. I’ve observed full-stack engineers become the norm at many places, who can do both frontend and backend development. Especially with AI, there is no reason a proficient frontend engineer should not work on backend as well, so, I expect “pure” frontend engineers will be employed only in larger companies, where demand exists for things like building a design system. We cover more on this topic in the deepdive, Design systems for software engineers.

4. AI engineering comp > software engineering comp

One poorly-kept secret in tech is that although software engineering compensation is very good at Big Tech and top startups, it’s superior for AI engineering jobs at the same places – and even better at leading AI labs:

Read more

I’m in Budapest, Hungary, this week, for Craft Conference, where I’ll be giving a keynote presentation alongside other speakers, including software engineering legend Kent Beck, who’s been on the podcast, Hillel Wayne, a formal methods expert and the author of ‘Logic for Programmers’, and Titus Winters, lead author of Software Engineering at Google.

The title of my keynote is “Slow down to speed up”, and I’ve been thinking about this topic a lot recently. Here are some things I’ve been seeing that I feel are relevant…

AI coding tools now used by pretty much all software engineers – that’s fast!

Read more

A year ago, most engineers found it harder to get responses to job applications, while hiring managers found it harder to hire than before. We’ve now spent two months gathering fresh, never-before-shared data from a range of sources to find out if that paradoxical situation still exists or not.

We collected the data with software engineer Jessica Salmon – who’s joined the team as a tech industry analyst. A big thank you to partner teams who shared exclusive details for this deepdive:

• TrueUp: a platform that scans every open job in Big Tech, top startups, and scaleups, typically paying in the top two tiers of the trimodal software engineering compensation model. Thanks, Amit Taylor.

• Workforce.ai, built by Live Data Technologies, which monitors 1M+ job changes and 300M+ employment validations each month across companies, roles, levels, functions, industries, and locations. Special thanks to Alex Hamilton for his input.

Today, we cover:

1. Software engineering recruitment: trending up, mostly. More software engineering job postings in the UK and the US, and a decline in Germany and France. “Top” tech companies are hiring 20% more vs a year ago.

2. Big Tech and publicly-traded companies. Meta went on a monster hiring spree for the last two years until layoffs, last week. Microsoft and Amazon were flat, while Google and Apple consistently hired devs. Stripe, Shopify, and Atlassian all hired more than Big Tech.

3. Who’s hiring the most software engineers? Apple, Amazon, and IBM are the top 3 by number of positions listed, while Meta dropped off the top 20 with layoffs there. The fastest-growing companies seem to be in fintech, observability, and security.

4. AI engineering: explosive demand. Most tech companies seem to be prioritizing AI engineering recruitment over software engineers. Observability companies appear to be hiring fastest.

5. Who’s hiring the most AI engineers? Apple, Google, and TikTok have the most openings, and many larger tech companies have 50-100% more AI engineering job listings than a year ago.

6. Is AI engineering replacing software engineering hiring? The data doesn’t say, but are the basics of AI engineering becoming a baseline skill for software engineering job candidates?

See Part 2 for details on:

• Top AI labs are now more attractive than Big Tech

• Harder for new grads & interns to get hired

• Mobile and frontend demand drops, AI & FDE surges

• AI engineering comp > software engineering comp

• Management’s “great flattening” continues

• Big Tech seniority & tenure keep rising

• Interview preparation signups: what do they indicate?

• Where engineers go after Big Tech

See Part 3 for stories from hiring managers and job seekers, covering:

1. “Catch-22:” nobody finds each other

2. No trust. Is AI to blame?

3. Hot market for some, but tough for most

4. Higher hiring bar & lower compensation – but not for everyone

5. Engineering leader recruitment: also weird for senior ICs

6. US market trends

7. Trends in the UK, EU, and rest of the world

For more on the tech jobs market of 2025, check out last year’s deepdive.

1. Software engineering recruitment: trending up, mostly

Top tech companies hiring more, slowly but surely

TrueUp tracks open positions at top-paying companies, which are usually US-based. Among this group, recruitment of software engineers has been steadily increasing since March 2023:

US growth, but less in Canada and Europe

Let’s take a look at broader industry trends via software developer jobs listed on Indeed.com, the world’s largest job aggregator site. As a “jobs aggregator”, Indeed not only displays paid-for ads but also crawls many other career sites and lists the vacancies.

Looking at trends over the last five years, the number of software engineering jobs listed is still lower than in May 2021:

Obviously, 2021 was the pandemic which created a commercial boom for the tech industry, so it’s not exactly an ideal comparison. Zooming into the last twelve months shows encouraging growth signs:

Outside of the US, let’s check out software engineering vacancy trends, per Indeed:

In this list, the US and the UK are the only two countries where vacancies are up; Canada is flat, while Germany and France have seen declines. To me, it suggests US-headquartered companies are hiring more devs, mostly in the US and some in the UK, whereas European-headquartered companies are more cautious about recruitment.

The software engineering industry still grows

Let’s ask how much the software engineer population has grown or shrunk in the last seven years. Here are some numbers from profiles, as tracked by Workforce.ai:

Let’s zoom into this chart:

There are a few interesting details:

• Hiring mostly occurs between the months of March and June, and there’s very little net growth in the second half of the year. This makes sense as tech companies set annual headcount budgets at the beginning of the year and commence hiring from then, with most hires evidently made in the following months. Hiring budgets are usually spent by mid-year.

• The industry seemingly shrank in 2023. That year, it was only in Q2 that more software engineers were hired than left their jobs. It was the worst period for software engineers in two decades.

…and also in 2024 and 2025 during Q3. Mass layoffs mostly happened between August and October, the charts show.

2. Big Tech and publicly-traded companies

Big Tech: modest jobs growth – but not at Meta

Looking at the two years since May 2024, of the five largest tech companies, only Meta grew headcount aggressively:

In that time, Meta increased its headcount by nearly 20%, so perhaps its recent 10% layoffs could be considered as not a total surprise. As for other tech giants, their software engineering headcount also changed in the last two years:

• Apple: +10%

• Google: +5%

• Microsoft: -1.1%

• Amazon: -1.3%

Visualizing this:

Apple and Google seem the most “stable” places when it comes to growing the software engineering function; Apple has not had mass layoffs for decades. In contrast, Meta is the most volatile place to work; hiring devs quickly and then letting them go. Elsewhere, Amazon and Microsoft are slowly employing more software folks.

Meta’s hire-then-fire rollercoaster

Meta warrants a closer look, where morale is reportedly as low as it’s ever been right now, due to layoffs of 10% of staff, while thousands of software engineers have been reassigned to manual data labeling from previous product development or infra work. From 2004 until 2022, Meta never did mass layoffs, but in the past three years there have been three sizeable redundancy rounds (at the end of 2022, early 2024, and last week), with the most rapid hiring growth of any tech giant sandwiched in between:

Before 2022, Meta ranked with Google as one of the most stable places to work in tech. Safe to say, the company is currently the riskiest place to be in job-security terms – even if Mark Zuckerberg has said there will be no more mass layoffs this year.

Publicly-traded tech companies: mostly flat recruitment

A couple of years ago, larger tech companies were hiring even more than most of Big Tech. Software engineering headcount grew in the past two years at Shopify (+36%), Stripe (+29%), Atlassian (+23%), Snap (+15%) and Spotify (+6%).

But that’s changed recently. In the past year, headcount growth has slowed at Stripe (+15%), Atlassian (+11%), Shopify (+6%), Snap (+2%), and Spotify (+1%),

This data does not capture Atlassian letting go of 10% of staff this month, and Snap laying off 16% in April.

Among this group, Stripe stands out as a place that keeps hiring at a higher pace than nearly all of Big Tech. Atlassian and Shopify have grown their software engineering headcount faster than Google, Microsoft, and Amazon did in the past two years.

3. Who’s hiring the most software engineers?

So, which companies are hiring the most? Let’s look at this via open positions and growth comparisons.

Companies with the most open roles

“Top” tech companies with the largest number of software engineering openings:

Compared to last year:

Observations:

• The top three are unchanged: It’s still Apple, IBM, and Amazon

• New entrants to the top 20: Accenture (consulting), Tesla, Cadence (hardware), Hewlett Packard Enterprise (HPE), and SpaceX.

• Dropouts: Meta and Oracle. Last year, Oracle was #4 on the list with 1,396 open software engineering roles, but the company announced up to 30,000 layoffs in March. Also no longer leading in jobs available are Speechify, NTT Data, Meta, and DXC Technology.

• Big Tech companies hiring more than last year: Apple and Google, which has +62% more engineering roles advertised than last year.

• Big companies hiring less vs last year: Meta, Oracle, and TikTok all have much fewer openings listed. For Meta and Oracle, there’s little surprise given recent layoffs. TikTok was in US regulatory limbo until January, when a new TikTok USDS joint venture was established.

• Hardware companies hiring more software talent: Micron, Qualcomm, and AMD all have considerably more software engineering openings than last year.

Where are the “top” software engineering jobs?

Most of these jobs are in the US, just as most companies leading in recruitment numbers are US-headquartered. Other leading locations for jobs are India, the UK, and Canada:

Fastest-growing: fintech and security

Then there are large companies where recruitment has been among the fastest anywhere in software engineering:

Software engineering headcount growth at some companies was significant over the past two years:

• Ramp (fintech): +94%

• Wiz (security): +84%

• Datadog (observability): +68%

• Rippling (fintech): +55%

• Figma (design tooling): +41%

• Netflix (streaming): +37%

Of this group, Datadog, Figma, and Netflix are publicly traded, and the largest by headcount:

• Datadog: AI agents are driving demand for observability through the roof, and Datadog is the leader of the pack. There’s demand to expand their infra and offerings. AI companies are also customers: OpenAI famously – or infamously! – spent around $170M on Datadog in 2025.

• Figma: the design tooling company went public in 2025 and is heavily investing in products related to AI, such as Figma AI.

• Netflix: the streaming behemoth is transitioning from “just” being a video streamer to becoming a broader entertainment platform, doing live events, advertising, gaming, and more. We did a podcast with Netflix CTO, Elizabeth Stone, in 2025 about Netflix’s engineering culture.

4. AI engineering: explosive demand

AI engineering demand is only going in one direction: upwards, at a fierce tick. Last September, our previous State of the Tech Job Market report found that AI Engineering was the hottest segment in tech for job listings growth. Today, the trend seems to continue:

Read more

1. Tradeoffs of AI tooling. Positive and negative sentiments, like how AI tools often mean less time spent on tedious, repetitive work, but that it often leads to unrealistic business expectations.

2. Adopting AI at scale is hard. Companies are struggling to achieve adoption that feels productive at the team and org levels. Our survey finds the benefits of AI heavily depend on the engineering culture that was in place before.

3. Impact on codebase quality. Codebase quality seems to be decreasing, but management at most places does not care. Meanwhile, maintenance duty is falling upon a shrinking number of engineers who still understand increasingly complex codebases.

4. Less experienced engineers and AI. For these folks, AI seems to be less helpful and they rack up higher AI token bills. Maybe they need more space to learn, mentorship, and support.

5. AI tooling ‘addiction’. Using AI agents “feels like a slot machine” encouraging “just one more prompt”-type behavior, while some folks think that the pricing of plans is built in a way to “lure” them to prompt more and more.

6. Changes since 2024. Fewer devs are negative about AI, but there’s not all that much more positivity. The models have become much higher quality, and better tooling improves trust.

7. Where are we headed? The concept of code ownership seems to be eroding, and collaboration within teams becomes less important.

Previously in this series, we covered:

• AI tooling for software engineers in 2026: a detailed summary of survey responses, covering the most-used AI tools, trends, AI agent usage, company size and usage, and tools engineers love.

• The impact of AI on software engineers in 2026: key trends. Part 1. Concerns about mounting AI costs, more engineers hitting usage limits, and AI tools having uneven effects upon different types of engineers

Full subscribers can access a more detailed report.

1. AI tooling tradeoffs

What can be said about the impact of AI tooling on engineers this year? Well, based on the responses that readers sent in to our survey, there are some common patterns:

Code and output tradeoffs

Productivity tradeoffs

Research published two weeks ago (5 May) by Microsoft claims AI expands the pool of people who can do high-value work. Our survey found similar, while MS’s findings are based on Microsoft 365 Copilot chat usage.

Obviously, Microsoft has an interest in showing the benefits of its AI tools, but in this specific area, it’s telling that both studies agree AI does allow devs and non-devs to expand the type of work they do.

Software engineering tradeoffs

With AI tools, their ‘mileage may vary’ among individuals. For example, some respondents told us they can spend much more time in a “flow state” thanks to AI tools, as they don’t have to wait for input from peers, can keep unblocking themselves, and have fewer interruptions.

But others say the opposite: that because they can start so many more tasks in parallel with AI tools, they keep context switching which knocks them out of a flow state!

The positives and negatives of these tools seem dependent on the environment respondents work in, individuals’ personality traits, and where users are on the AI learning curve.

2. Adopting AI at scale is hard

One pattern in the responses is that adopting AI at company-scale remains challenging, including:

• Costs: a growing concern covered in depth in Part 1

• Usage: getting people to use AI tools continuously is not always straightforward

• Onboarding and education: at larger companies, there needs to be support to help devs make the most of the tools

• Reviewing AI-generated output: code review is a particular pain point

• Integrating with internal systems: AI tools are more helpful when they are seamlessly integrated with internal systems, hence why many larger companies use in-house, deeply embedded coding agents

Adopting AI at company-level doesn’t lead to a magical fix for engineering problems, while the benefits of AI in an organizational sense seem to depend on what was in place before.

Case in point, AI seems to amplify pre-existing engineering culture. AI doesn’t change the underlying quality of an organization’s engineering culture. Teams with strong engineering practices get more positive benefits out of AI than those without. Teams that see benefits from AI tools already had:

• Guardrails: testing and automation around the codebase and deployments

• Documentation: they recorded their architectural decisions and engineering practices

• A quality codebase: AI agents will replicate patterns already in a codebase

A few quotes on this from respondents to our survey:

“AI is an amplifier, not a fixer. Good software engineering practices get multiplied. So do the bad ones. Embedding this properly in teams is exciting and important”. – Staff+ engineer at a large company in Europe

“I feel like AI allows both faster prototyping and increased velocity on iterations to production software; it relies on existing best practices / project templates our team already have”. – Solutions Architect at a small company in the US

A workflow that makes one dev “10x” more productive may not work for another. This is another reason why rolling out AI tools doesn’t seem to magically make everyone more productive. A senior engineer working at a large company in Canada told us:

“It feels like AI workflows are very idiosyncratic in that some people derive (I hate this framing, but…) 10x more productivity benefit from them than other apparently equally clever, educated, and diligent developers. It feels like finding a workflow that clicks with your own habits and heuristics is more important than finding a global optimum for everyone”.

AI amplifies individuals differently, so the team impact is messy to figure out. A US-based principal engineer at a large company reflected on feeling disconnected from colleagues because of how they use AI differently:

“I use AI in what I think is probably a more sophisticated way than most of my colleagues, so there can be a disconnect between my work and theirs, which is not good news because I am “The Principal” on the team”.

“The tool that works for you” approach can lead to tooling chaos, even at a team level. While it’s empowering to allow devs to choose the AI tools they feel are the most helpful, over time it becomes chaotic when teams can choose their own tech stacks, and when at large companies there are dozens of different technologies. A staff+ engineer at a 200-person business in the Middle East wrote in their response:

“We’re still trying to figure out how to deal with tooling consistency on a team level. It’s one of our biggest struggles, but possibly more due to company structure than anything else. Everyone is using different tools with little coherence. It’s been rough.”

Some companies have briefly rolled back AI to deal with the negative effects first. From an engineering lead at a 10,000+ person company in Europe:

“Since the AI boom, the quality of technical writing and reasoning from senior engineers in my org has significantly deteriorated. There’s an overwhelming volume of low-quality work product that is generated entirely or in part by AI, which has made it very difficult to conduct meaningful review of RFCs or code. We’ve also seen costly production incidents caused by code written and/or approved by AI, and – while my employer initially bought heavily into the hype – we have now rolled back some of our AI tools to deal with the drop in quality.”

3. Impact on codebase quality

A concrete pattern in our survey data is that codebase quality is decreasing due to AI. The contributing factors are not surprising:

• “AI slop”: more low-quality code generated, such as duplicated, verbose code, and poor abstractions

• Too many code reviews, which means review quality slips

• More bugs: due to faster code output and less strict reviews, more bugs sneak into codebases

We discuss the degradation of products and codebases in a recent deepdive, Are AI agents actually slowing us down?

One CTO at a European startup lists their negatives about increased AI usage:

1. “A lot of tiny bugs and low code quality if you are not careful, verify carefully, and have good structure and guardrails

2. AI agents generate too much and repetitive code, making systems harder to maintain

3. Developers lose understanding of the codebase and become numb to bad architecture and bad developer experience”

According to our survey, management often seems unfazed by decreasing quality, and instead focuses on the higher output. A principal DevOps engineer at a large European company said:

“In our company, we hand AI tools to inexperienced engineers who can’t distinguish good code from bad code and it’s falling on deaf ears in our leadership. They only seem to care about short to mid-term cost savings.”

The maintenance burden of AI-generated code is falling on the fewer engineers who understand and care. A staff engineer at a European company listed the problems that AI-generated code is causing:

• “Drive by” contributions are up: many more occasional non-core-engineer contributors adding code but not sharing the maintenance burden

• Contributing without adding guardrails: many engineers and most of engineering leadership are not using reasonable guardrails like tests

• AI slop from folks who have nothing to do with the codebase: huge volume of slop incoming from people who don’t understand the codebase, but will commit and create PRs without fully understanding what they’re doing

• Complexity is exploding: thanks to the above

The maintenance budget is falling upon fewer devs, while the task of refactoring bloated codebases and reducing complexity is left to those still sufficiently in touch with the codebase, thereby making the maintenance burden even worse.

But some leaders “get it.” A CEO at a 20-person company told us:

“While AI has made generating code ‘cheaper’, the monitoring and maintenance worry me; the things that have traditionally cost the most in software. We’re increasing the rate of shipping large amounts of code with less understanding and increasing the unpredictability, so how do we work the predictability back on top?”

There is industry pressure on companies to adopt AI tooling and impose its usage upon engineers, driven by a mix of factors:

• Seeing actual benefits of AI and hearing that other teams and companies enjoy success with it

• Fear of being left behind by competitors, or becoming less relevant

• Anxiety about investor interest if a company is seen as not adopting the latest AI tools.

This often leads to:

• Top-down mandates to use AI

• Expectation of headcount reduction, with smaller teams doing the same amount or more

• Management treating AI productivity gains as a baseline rather than a bonus.

One staff engineer at a 10,000-person US company explained it like this:

“AI is part of almost every work conversation. The entire company expects it to increase productivity and reduce the need to hire people. I keep trying to get better at using it and trying to make it more reliable so I can do more. I do worry about the quality of the work and atrophy of certain skills. It’s unclear to me if those skills even matter anymore.”

Pushing AI adoption blindly triggers red flags. Respondents shared what makes them worry about things going south in their workplaces:

• Focus on tracking AI usage, but not the quality of the output. This will likely lead to product regressions and unhappy customers

• Pushing for universal adoption. Some companies target 50%, 80%, or 100% AI usage for certain tasks, seemingly blind to how some targets can worsen the quality of output, or simply create wasteful usage. See the trend of tokenmaxxing.

• A focus on velocity, but without recognition of quality work. Expecting more velocity and output seems to be the baseline, and there is no recognition for work truly well done.

The “move fast and break things” mantra famously championed by Facebook seems widespread across the industry with AI tools. A senior manager working at a large, European-headquartered company told us:

“I see a trend: move fast and break things, and end up breaking things too often. We have to learn to focus on testing and resiliency a lot more, as with AI-driven development we introduce more bugs than before. But the velocity gain is bigger for now”.

Output over quality is leading to the death of code review at some places. As a lead engineer working at a small company summarized:

“We’re at the death of code review. I used to do very deep code reviews where I’d take the time to understand the architecture and organization and provide feedback on maintainability and efficiency. I have no motivation in spending that time to review a giant PR where it’s clear that even the original author didn’t bother to do that”.

4. Less experienced engineers and AI

The first generation of software engineers who have never developed without the help of AI are now entering the industry. Here is a response from a young engineer working at a startup as an intern:

“I have never worked as a developer without AI. Writing this scares me a bit, actually, but it’s the truth!”

But this will be the new reality for those joining the industry. So, what needs to happen to help a new generation of “AI-native” grad engineers grow professionally?

AI is an amplifier which could amplify the lack of experience. A staff engineer in the US at a large company told us:

“Agentic AI is a fascinating mirror. It can code as well as the user who drives it. If that user is a junior engineer, now you have a faster junior engineer. If the user is a staff engineer, now you have a faster staff engineer.

What agentic AI doesn’t do is magically convert a junior engineer into a staff engineer, because the user driving it still needs enough experience to know what a good solution looks like”.

A junior engineer in Australia shares their experience of how frustrating working with AI tools is:

“I think AI agents are great for vibe coding or prototypes where the code quality and functionality doesn’t matter that much. I think it’s also useful for senior engineers who know what they’re doing.

For junior engineers like myself, these AI tools are stressful to use. I don’t have the experience or knowledge to tell AI exactly what to do or quickly confirm its output, so I spend a lot of time on just triple checking and redoing stuff. I’m overall frustrated, but I’m trying to embrace it as we’ve been asked to by the company”.

Less experienced engineers seem to use more AI tokens and rack up higher bills. Several respondents observed this: director-level folks noted that junior engineers are in the top-spender category in their orgs, and it is junior devs who spend tokens on unproductive use cases.

There should be more space for junior folks to grow because they use AI more, one staff engineer respondent said:

“Companies need to give some breathing room to Junior engineers and help them learn and acquire knowledge using AI tools as a booster and not as a replacement”.

Junior folks seem to be delegated fewer opportunities that could help them grow. This is because senior people can turn to AI for tasks, including those which they would have previously delegated to an intern or new grad. A few responses mention this:

“AI allows me to have work done that I would usually delegate to a junior or pay a SaaS for; e.g., writing drafts, summarizing the news. “ DevSecOps lead at a small company, Europe

“I’ve begun to automate any repetitive task that we previously relied on juniors and offshore contractors for.” - Engineering manager, at a large company, US

“I no longer have to delegate work by writing a very long document and briefing a junior engineer.” - Principal engineer, large company, Europe

Why not consider mentoring junior devs in your organization? It’s clear that less experienced engineers are having a rocky start to their careers, so delegating stuff to them instead of to AI could be of high value for newer generations of talent.

5. AI tooling ‘addiction’

It seems that the rapid feedback loops of AI-assisted development create addictive tendencies, and there’s a noteworthy presence of “addiction lingo” in some responses to our survey:

Read more

Four decades ago, the writer of ‘The Mythical Man-Month’ (1975), drew on folklore about werewolves to publish a paper about the prospects of a so-called silver bullet for software development that would make professionals much more productive at their craft.

Frederick P. Brooks published “No Silver Bullet – Essence and Accident in Software Engineering” in 1986, and as the title suggests, it is pessimistic about the existence of any silver bullets. The term refers to a super weapon capable of dropping otherwise near-unstoppable werewolves and other creepy supernatural beings in European folk tales.

Since its release, this paper might have become even better-known than Mythical Man-Month (MMM). In 1995, the second edition of that book included Brooks’ later essay as chapter 17, along with an additional chapter of reflections.

In this article, we look into whether the essay was correct in its disbelief in silver bullets, or whether any did indeed slay the beast of unproductivity for developers over the course of time. Also, how does AI agents generating so much code, as of today, challenge the entire premise – or not?

We cover:

1. “No silver bullets” – why has it held up? No single new technology or methodological breakthrough by itself introduced magnitudes-of-improvement to the areas that really matter in software engineering. Is that unusual?

2. Is SRE a silver bullet? Google’s Search team introduced the SRE discipline, and won orders-of-magnitude superior reliability to its competitors. But why only Google Search?

3. Was open source + GitHub a silent silver bullet? No development had a bigger impact on the wider tech industry than the open source wave since the 2010s. Has it been a silent silver bullet, an overlooked cause?

4. Could AI be a silver bullet? At first glance, AI generates 100x-or-more code output. But productivity, reliability, and simplicity improvements are a bit unimpressive – at least for now.

Brooks was a computer scientist who led IBM’s System/360 and OS/360 operating systems development, ‘The Mythical Man-Month’ was published in 1975. Last year, we did a deepdive into this engineering classic (Part 1, Part 2, Part 3, Part 4), delving into its predictions and legacy.

1. No silver bullets?

The paper delves into folklore for its motif, a ‘silver bullet,’ and uses it to pose the question of whether there would be any “silver bullets” on the horizon (in 1986) that could be similarly fatal to software engineering complexity. From the paper (emphasis mine:)

“Of all the monsters who fill the nightmares of our folklore, none terrify more than werewolves, because they transform unexpectedly from the familiar into horrors. For these, one seeks bullets of silver that can magically lay them to rest.

The familiar software project has something of this character (at least as seen by the non-technical manager), usually innocent and straightforward, but capable of becoming a monster of missed schedules, blown budgets, and flawed products. So, we hear desperate cries for a silver bullet, something to make software costs drop as rapidly as computer hardware costs do.

But, as we look to the horizon of a decade hence, we see no silver bullet. There is no single development, in either technology or management technique, which by itself promises even one order of magnitude improvement in productivity, in reliability, in simplicity.

Skepticism is not pessimism, however. Although we see no startling breakthroughs, and indeed, believe such to be inconsistent with the nature of software, many encouraging innovations are under way. A disciplined, consistent effort to develop, propagate, and exploit them should indeed yield an order-of-magnitude improvement. There is no royal road, but there is a road.

The first step toward the management of disease was replacement of demon theories and humor theories by the germ theory. That very step, the beginning of hope, in itself dashed all hopes of magical solutions. It told workers that progress would be made stepwise, at great effort, and that a persistent, unremitting care would have to be paid to a discipline of cleanliness. So it is with software engineering today.”

In 1995, Brooks revisited his idea that silver bullets weren’t real in the software domain. From the Mythical Man-Month’s anniversary edition:

“No Silver Bullet” asserts and argues that no single software engineering development will produce an order-of-magnitude improvement in programming productivity within ten years (from the paper’s publication in 1986). We are now nine years into that decade, so it is timely to see how this prediction is holding up.

Whereas The Mythical Man-Month generated many citations but little argument, “No Silver Bullet” has occasioned rebuttal papers, letters to journal editors, and letters and essays that continue to this day.

Most of these attack the central argument that there is no magical solution, and my clear opinion that there cannot be one. Most agree with most of the arguments in “NSB,” but then go on to assert that there is indeed a silver bullet for the software beast, which the author has invented. As I reread the early responses today, I can’t help noticing that the nostrums pushed so vigorously in 1986 and 1987 have not had the dramatic effects claimed.”

Brooks re-concluded that there had been no technological breakthroughs of the type postulated in NSB.

But motivation can also have silver bullet-like effects and always has had, he found via more research into scientific evidence that motivation can boost productivity. In his own words:

“Since “NSB,” Bruce Blum has drawn my attention to the 1959 work of Herzberg, Mausner, and Sayderman.

They find that motivational factors can increase productivity. On the other hand, environmental and accidental factors, no matter how positive, cannot; but these factors can decrease productivity when negative. “NSB” argues that much software progress has been the removal of such negative factors: stunningly awkward machine languages, batch processing with long turnaround times, poor tools, and severe memory constraints.”

Today, it’s a long time since the mid-nineties; with the benefit of hindsight, were there any silver bullets flying between then and 2022, which fit the bill as slayers of unproductiveness? I suggest a few, below. If you can name other silver bullets since the launch of Windows 95, please do so in the comments!

• Version control: (late 1990s.) CVS, Subversion, and later, Git. Version control allowed engineers to collaborate much more fluently, leading to more teamwork and – in some cases – less full-on solo labor.

• IDEs: (early 2000s). Modern IDEs like Visual Studio, IntelliJS, and others make context-rich editing easy and fast. They also allow for faster, less error-prone refactoring and more efficient debugging.

• CI/CD and automated testing: (mid-2000s). CI systems started to spread during the 2000s with the likes of CruiseControl (2000s) → Jenkins and SaaS CI solutions from the 2010s (e.g., Travis CI, CircleCI, GitLab CI, GitHub Actions).

• Open source and package managers: (2010s). Open source has been around for decades, but GitHub’s rapid adoption made it easier to create and discover, coupled with package managers in the Node, Python, and other language ecosystems to build on top of open source solutions.

• StackOverflow: (2010s). The popular programming Q&A site made it easier to get unstuck by finding solutions to common problems, with the capability to ask questions and get responses from the large user community within hours. By 2025, the site was pretty much dead.

• Cloud: (early 2010s). AWS launched in 2006, then Azure and Google Cloud in 2008, and they went mainstream in the 2010s. Today, cloud is everywhere, and increasingly more infrastructure startups build on top of hyperscalers (the biggest public cloud platforms), such as Vercel / Netlify (platform-as-a-service), Supabase / Turbopuffer (databases) and others.

• Platform teams & DevEx teams: (mid-2010s). Mid-sized and larger tech companies created dedicated platform teams to own infrastructure and internal platforms. In the 2020s, larger tech companies have created ‘Developer Experience’ teams to build better internal dev tools and workflows. We previously covered Uber’s program/platform split, and Uber’s developer experience evolution.

• SRE: the Site Reliability Engineer (SRE) profession grew during the 2010s, much influenced by Google.

These technologies increased developer efficiency and productivity, but none by itself was a productivity accelerator in isolation.

Obviously, by 2022 the craft of building software had developed greatly since ‘No Silver Bullets’ came out; and was more efficient, faster, and more collaborative than ever. One highly anecdotal way to identify this is via the disappearance of cake from some tech workplaces. Back in the day, cake was distributed at work for major product milestones being hit: the shipping of a new product was often marked with awards and tasty baked treats – at least on teams building browsers, like the IE and Firefox teams.

But by the 2010s, shipping frequency had increased by so much and was an everyday, unremarkable occurrence at some places, according to Matt Brubeck, a former engineer on the Firefox team:

“Back when Firefox 2 was released (six years ago this week!), the Internet Explorer team started a friendly tradition of sending Mozilla a cake as congratulations. This continued for Firefox 3 and Firefox 4. After Firefox switched from major releases once or twice a year to incremental updates every six weeks, they sent us a cupcake for the next few updates instead. :)” Mozilla engineer, Matt Brubeck

Today, Firefox ships a stable version about once a month, as does Chrome. In this context, marking each release with more cake could inadvertently cause some health issues on the team – too much cake, that is! From this September, Chrome will switch to shipping every two weeks.

Agile and Scrum is worth a mention; not as a technology, but a methodology: Scrum encourages teams to move in smaller cycles and deliver more frequently, via sprints that typically range from a week to a month. In the early 2000s, this methodology spread quickly and brought efficiency improvements to many tech companies. However, by the early 2020s, many startups and some of Big Tech had moved on, as covered in How Big Tech runs tech projects and the curious absence of Scrum:

“Scrum got in the way of shipping on a daily basis. The whole idea of Scrum revolves around Sprints, of committing to tasks at the beginning of the sprint, working on these during the sprint, and demoing what we did at the end.

The process felt unnatural and like it had been forced on a fast-moving web team. We soon moved to a more fluid way of working, taking the Kanban approach. We stopped caring about sprints, and dropped most rituals that come with Scrum. We just cared about knowing what we’re working on now, and what it was we’d get done next.”

Basically, Scrum worked and still does so for teams wanting to shorten shipping cadence from months to weeks. But for teams shipping daily, it often gets in the way.

One area that improved significantly has been the pace of shipping incremental software. In 1975, shipping software several times per day with elements like version control, CI/CD, feature flags and engineers being oncall, might have sounded far-fetched. Back then, software delivery was measured in months and years. In this way, we’ve perhaps made improvements overall in the regions of 10x to 100x over the years.

But that came via combinations of new tools like version control and CI/CD, new approaches & methodologies, and testing – and also from shifting constraints; for example, it’s now possible to revert backend changes rapidly, and code shipped in binaries can be controlled by feature flags in many cases.

Even so, improvements were mainly in iteration speed and not necessarily in the complexity of the software shipped. With all that progress, shipping complex and high-quality software still takes comparable time, often years, as 50 years ago. A prime example is the upcoming video game, Grand Theft Auto VI, probably by now the most highly-anticipated game ever, which is set to launch in November, after at least six years – and potentially 12 – of total development time:

• Initial planning started in 2014 (12 years of development)

• Development started in earnest in 2020 (circa six years of full-on development)

• The studio, Rockstar, confirmed development was underway in February 2022 (at least 4.5 years of full development)

The video game development timeline is as long as it ever was, and even longer, as developers strive to meet players’ expectations on things like graphics, lighting, and physics. GTA 6 looks like being the most complex installment in the long-running series. So, perhaps there’s not been much change in software delivery timelines because when we have more capabilities to work with, the goals get more ambitious and the bar for “standout” software keeps rising.

2. Is SRE a silver bullet?

Brooks’s definition of a silver bullet:

“A single development, in either technology or management technique, which by itself promises even one order of magnitude improvement in productivity, in reliability, in simplicity.”

In simplicity and productivity terms, I struggle to name a single approach that delivered a 10x-or-more improvement by itself. But in the area of reliability, one company that has pioneered novel approaches since the 2000s is Google. Google.com is probably the single most reliable piece of internet software of all. In the last 15 years, Google Search has suffered a single outage, on 8 August 2022, which lasted around an hour. Otherwise, there have been no global outages (of course, there have been several for other Google services).

In 2003, Google created the ‘Site Reliability Engineering’ (SRE) role. SRE veteran, Dave O’Connor, shared with us:

“The fervent belief of Google’s founders was that speed and reliability mattered more than features. This belief was coupled with the understanding that we couldn’t achieve it traditionally, which made it an existential issue. The level of investment in building out all layers of the serving stack was a case of “because we can”, but also “because we have to, as nowhere else does what we need”.

There was never a question of whether traditional ‘ops’ would work at Google. We needed a specialized role, staffed by folks familiar with the problem space and engineering methods required to make it work.

In 2003, the SRE role was born. Ben Treynor Sloss had been tasked with building Google’s “production team” and in his own words, he built “what happens when you ask a software engineer to design an operations team.” This turned into the birth of the SRE function at Google. From the outset, SRE was staffed in varying measures by systems/operations experts and software engineers. A large part of the remit of the team was to build the tools and practices required to operate Google’s fleet.”

Over time, the rest of the industry caught on to SRE and DevOps. From our SRE deepdive:

“Eventually, other companies caught onto the scaling issues, especially the hyperscalers. Each had their own approach, but over time, the notion grew industry-wide that making things reliable was a real-life engineering discipline, not simply ‘ops’.

This step saw a number of terms coined to describe this engineering, including ‘DevOps’. At its core, this was the notion that the disciplines and practices of reliability engineering should be ingrained into the overall engineering organization. At places other than Google, this mostly took the form of combined developer/operations roles (i.e. “you build it, you run it”), which differed from Google’s implementation, but the practices were similar.

Around this time, Google started opening up about SRE, eventually publishing the first SRE book, and follow ups. Conferences such as USENIX SRECon, Devops Days, and other movements have solidified reliability engineering as a discipline that scales well beyond Google. Indeed, the company has become a consumer of many state-of-the-art developments.”

So, at Google Search, the SRE role could be described as a genuine silver bullet for the tech giant. The company’s obsession with reliability helped it build what is probably the most reliable public-facing service of all. On the assumption that SRE plays a significant role in the approach, I would feel comfortable with calling SRE a silver bullet for Google Search.

SRE, as a concept, is commonplace across Google, but the reliability of its other services is not so impressive. For example, Google Cloud has had many outages, and Gmail also goes down every now and then. I’m sure that without SRE, reliability would be worse, but in general, Google services’ availability these days is probably a magnitude higher than the availability of most online services in the 2000s.

Similarly, GitHub has an SRE role but the service is at zero nines of availability, partially explained by a 3.5x increase in load in two years. But in other ways, the zero nines is likely self-inflicted.

This makes me wonder if the existence of silver bullets depends greatly on teams and individual contexts. SRE seems like a good case to consider:

• a “silver bullet” for Google Search

• … but not for other Google services

• … and definitely not for the broader industry

Could it be that when implemented in the right place, in the right way, and with the correct investment, then SRE – and an incredible focus on reliability – will yield a 10x-or-higher increase in reliability?

My hunch is that Google Search has such standout reliability not just because of SRE, but because Search might be the only organization in Google with reliability as a founding value, embedded in the team’s culture, with unmatched investments of time and money.

Google has published several books that explain their techniques and practices, but for other teams to get those results, they would need to invest similarly in reliability.

3. Was open source + GitHub a silent silver bullet?

Perhaps there’s a silver bullet which is easily missed: open source. In the first-ever Pragmatic Engineer Podcast episode, I asked software engineer Simon Willison what the biggest “productivity leaps” have been during his career. He named open source:

Read more

Nine years later, he felt the time was ripe for an updated edition, with cloud computing much more widespread than in 2016. So, Martin teamed up with software engineer and investor, Chris Riccomini, a former colleague at LinkedIn and the author of The Missing README, for a full refresh of the book which brings it right up to date for the present day.

Martin was recently on The Pragmatic Engineer Podcast, where we discussed this updated volume and many related cloud computing matters. We also looked into some topics that have become less relevant over time, like details on MapReduce.

I asked Martin if this newsletter could share an excerpt of the updated edition of the book about a timeless, important topic, and he generously agreed. So, today we cover:

1. Cloud versus self-hosting tradeoffs

2. Doing the right thing as a software engineer

These excerpts are only part of the book; the first edition has been on my shelf for years and is now in well-worn condition. I jumped at the chance to get the second edition, and if you’re interested in building resilient systems, I recommend it as an excellent resource.

Get the second edition of the book

My usual disclaimer: as with all my recommendations, I was not paid for this article, and none of the links are affiliates. See my ethics statement for more.

The excerpt below is from “Designing Data-Intensive Applications” second edition, by Martin Kleppmann and Chris Riccomini. Copyright © 2026 Martin Kleppmann, Chris Riccomini. Published by O’Reilly Media, Inc. Used with permission.

1. Cloud versus self-hosting tradeoffs

This excerpt is from Chapter 1: “Trade-Offs in Data Systems Architecture”

For anything that an organization needs to do, one of the first questions is whether it should be done in-house or outsourced. That is, should you build or should you buy?

Ultimately, this is a question about business priorities. A common rule of thumb is that things that are a core competency or a competitive advantage of your organization should be done in-house, whereas things that are non-core, routine, or commonplace should be left to a vendor [20]. To give an extreme example, most companies do not fabricate their own CPUs, since it is cheaper to buy them from the semiconductor manufacturers.

With software, two important decisions to be made are who builds the software and who deploys it. The spectrum of possibilities is illustrated in Figure 1-2. At one extreme is bespoke software that you write and run in-house; at the other extreme are widely-used cloud services or SaaS products that are implemented and operated by an external vendor and that you access only through a web interface or API.

The middle ground is off-the-shelf software (open source or commercial) that you self-host, or deploy yourself – for example, if you download MySQL and install it on a server you control. This could be on your own hardware (often called ‘on-premises,’ even if the server is in a rented datacenter rack and not literally on your own premises), or on a virtual machine (VM) in the cloud (infrastructure as a service, or IaaS). There are more points along this spectrum, such as taking open source software and running a modified version of it.

A related question is how you deploy services, either in the cloud or on premises – for example, whether you use an orchestration framework such as Kubernetes. However, choice of deployment tooling is beyond the scope of this book, since other factors have a greater influence on the architecture of data systems.

Pros & Cons of Cloud Services

Using a cloud service, rather than running comparable software yourself, essentially outsources the operation of that software to the cloud provider. There are good arguments for and against this approach. Cloud providers claim that using their services saves time and money and allows you to move faster compared to setting up your own infrastructure.

Whether using a cloud service is actually cheaper and easier than self-hosting depends very much on your skills and the workload on your systems, however. If you already have experience of setting up and operating the systems you need, and if your load is quite predictable (i.e., the number of machines you need does not fluctuate wildly), then it’s often cheaper to buy your own machines and run the software on them yourself [21, 22].

On the other hand, if you need a system that you don’t already know how to deploy and operate, adopting a cloud service is often easier and quicker than learning to manage the system. Hiring and training staff specifically to maintain and operate the system can get very expensive. You still need an operations team when you’re using the cloud, but outsourcing the basic system administration can free up your team to focus on higher-level concerns.

Outsourcing the operation of a system to a company that specializes in running it can potentially result in better service, since the provider gains operational expertise from providing the service to many customers. On the other hand, if you run the service, you can configure and tune it to perform well on your particular workload. A cloud service would likely be unwilling to make such customizations on your behalf.

Cloud services are particularly valuable if the load on your systems varies a lot over time. If you provision your machines to be able to handle peak load, but those computing resources are idle most of the time, the system becomes less cost-effective. In this situation, cloud services have the advantage that they can make it easier to scale your computing resources up or down in response to changes in demand.

For example, analytical systems often have extremely variable load. Running a large analytical query quickly requires a lot of computing resources in parallel, but once the query completes, those resources sit idle until a user makes the next query. Predefined queries (e.g., for daily reports) can be enqueued and scheduled to smooth out the load, but for interactive queries, the faster you want them to complete, the more variable the workload becomes. If your dataset is so large that querying it quickly requires significant computing resources, using the cloud can save money as you can return unused resources to the provider rather than leaving them idle. For smaller datasets, this difference is less significant.

The biggest downside of a cloud service is that you have no control over it:

• If it is lacking a feature you need, all you can do is politely ask the vendor whether they will add it; you generally cannot implement it yourself.

• If the service goes down, all you can do is wait for it to recover.

• If you are using the service in a way that triggers a bug or causes performance problems, diagnosing the issue will be difficult. With software that you run yourself, you can get performance metrics and debugging information from the operating system to help you understand its behavior, and you can look at the server logs. With a service hosted by a vendor, you usually do not have access to these internals.

• If the service shuts down or becomes unacceptably expensive, or if the vendor changes their product in a way you don’t like, you are at their mercy; continuing to run an old version of the software is usually not an option, so you’ll be forced to migrate to an alternative service [23]. This risk is mitigated if alternative services expose a compatible API, but for many cloud services there are no standard APIs, which raises the cost of switching, making vendor lock-in a problem.

• If the cloud provider is in another country and a political conflict arises between that country and your own, you risk being locked out of the service due to imposed sanctions.

• The cloud provider needs to be trusted to keep the data secure, which can complicate the process of complying with privacy and security regulations.

Despite all these risks, it has become more and more popular for organizations to build new applications on top of cloud services, or to adopt a hybrid approach in which cloud services are used for some aspects of a system. However, cloud services will not subsume all in-house data systems. Many older systems predate the cloud, and for any services that have specialist requirements that existing cloud services cannot meet, in-house systems remain necessary. For example, very latency-sensitive applications such as high-frequency trading require full control of the hardware.

Cloud-Native System Architecture

Besides having a different economic model (subscribing to a service instead of buying hardware and licensing software to run on it), the rise of the cloud has also had a profound effect on how data systems are implemented on a technical level. The term “cloud native” is used to describe an architecture that is designed to take advantage of cloud services.

In principle, almost any software that you can self-host could also be provided as a cloud service, and indeed, such managed services are now available for many popular data systems. However, systems that have been designed from the ground up to be cloud native have been shown to have several advantages: better performance on the same hardware, faster recovery from failures, being able to quickly scale computing resources to match the load, and supporting larger datasets [24, 25, 26]. Table 1-2 lists some examples of both types of systems.

Layering of cloud services

Many self-hosted data systems have simple system requirements; they run on a conventional operating system such as Linux or Windows, they store their data as files on the filesystem, and they communicate via standard network protocols such as TCP/IP. A few systems depend on special hardware such as GPUs (for ML) or remote direct memory access (RDMA) network interfaces, but on the whole, self-hosted software tends to use generic computing resources: CPUs, RAM, a filesystem, and an IP network.

In a cloud, this type of software can be run in an IaaS environment, using one or more VMs (or instances) with a certain allocation of CPUs, memory, disk, and network bandwidth. Compared to physical machines, cloud instances can be provisioned faster and come in a greater variety of sizes, but otherwise they are similar to traditional computers: you can run any software you like on them, but you are responsible for administering it yourself.

In contrast, the key idea of cloud-native services is not only to use the computing resources managed by your operating system, but also to build upon lower-level cloud services to create higher-level services. For example:

• Object storage services such as Amazon S3, Azure Blob Storage, and Cloudflare R2 store large files. They provide more limited APIs than a typical filesystem (basic file reads and writes), but they have the advantage that they hide the underlying physical machines; the service automatically distributes the data across many machines so that you don’t have to worry about running out of disk space on any one machine. Even if some machines or their disks fail entirely, no data is lost.

• Many other services are, in turn, built upon object storage and other cloud services. For instance, Snowflake is a cloud-based analytical database (data warehouse) that relies on S3 for data storage [26], and some other services, in turn, build upon Snowflake.

As always with abstractions in computing, there is no one right answer to what you should use. As a general rule, higher-level abstractions tend to be more oriented toward particular use cases. If your needs match the situations for which a higher-level system is designed, using the existing higher-level system will probably meet your needs with much less hassle than building it yourself from lower-level systems would. On the other hand, if no high-level system meets your needs, building it yourself from lower-level components is the only option.

Separation of storage and compute

In traditional computing, disk storage is regarded as durable (we assume that once something is written to disk, it will not be lost). To tolerate the failure of an individual hard disk, RAID (redundant array of independent disks) is often used to maintain copies of the data on several disks attached to the same machine. RAID can be implemented either in hardware or in software by the operating system, and it is transparent to the applications accessing the filesystem.

In the cloud, compute instances (VMs) may also have local disks attached, but cloud-native systems typically treat these disks more like an ephemeral cache and less like long-term storage. This is because the local disk becomes inaccessible if the associated instance fails, or if the instance is replaced with a bigger or a smaller one (on a different physical machine) to adapt to changes in load.

As an alternative to local disks, cloud services also offer virtual disk storage that can be detached from one instance and attached to a different one (e.g., Amazon EBS, Azure managed disks, and persistent disks in Google Cloud). Such a virtual disk is not a physical disk, but rather a cloud service provided by a separate set of machines that emulates the behavior of a disk (a block device, where each block is typically 4 KiB in size). This technology makes it possible to run traditional disk-based software in the cloud, but the block device emulation introduces overheads that can be avoided in systems that are designed from the ground up for the cloud [24]. The use of virtual disks also makes the application very sensitive to network glitches, since every I/O operation on the virtual block device is a network call [27].

To address this problem, cloud-native services generally avoid using virtual disks and instead build on dedicated storage services that are optimized for particular workloads. Object storage services such as S3 are designed for long-term storage of fairly large files, ranging from hundreds of kilobytes to several gigabytes in size. The individual rows or values stored in a database are typically much smaller than this; cloud databases therefore typically manage smaller values in a separate service and store larger data blocks (containing many individual values) in an object store [25, 28].

In traditional systems architecture, the same computer is responsible for both storage (disk) and computation (CPU and RAM), but in cloud-native systems, these two responsibilities have become somewhat separated, or disaggregated [9, 26, 29, 30]: for example, S3 only stores files, and if you want to analyze that data, you will have to run the analysis code somewhere outside of S3. This implies transferring the data over the network.

Furthermore, cloud-native systems are often multitenant, which means that rather than having a separate machine for each customer, data and computation from several customers are handled on the same shared hardware by the same service [31]. Multitenancy can enable better hardware utilization, easier scalability, and easier management by the cloud provider, but it also requires careful engineering to ensure that one customer’s activity does not affect the performance or security of the system for other customers [32].

Operations in the Cloud Era

Traditionally, the people managing an organization’s server-side data infrastructure were known as database administrators (DBAs), or system administrators (sysadmins). More recently, many organizations have tried to integrate the roles of software development and operations into teams with a shared responsibility for both backend services and data infrastructure; the DevOps philosophy has guided this trend. Site reliability engineers (SREs) are Google’s implementation of this idea [33].

The role of operations is to ensure that services are reliably delivered to users (including configuring infrastructure and deploying applications) and to ensure a stable production environment (including monitoring and diagnosing any problems that may affect reliability). For self-hosted systems, operations traditionally involve a significant amount of work at the level of individual machines, such as capacity planning (e.g., monitoring available disk space and adding more disks before you run out of space), provisioning new machines, moving services from one machine to another, and installing operating system patches.

Many cloud services present an API that hides the individual machines implementing the service. For example, cloud storage replaces fixed-size disks with metered billing, where you can store data without planning your capacity needs in advance, and you are then charged based on the space used. Moreover, many cloud services remain highly available, even when individual machines have failed.

This shift in emphasis from individual machines to services has been accompanied by a change in the role of operations. The high-level goal of providing a reliable service remains the same, but the processes and tools have evolved.

The DevOps/SRE philosophy places greater emphasis on the following:

• Setting up automation; preferring repeatable processes over manual one-off jobs

• Using ephemeral VMs and services rather than long-running servers

• Enabling frequent application updates

• Learning from incidents

• Preserving the organization’s knowledge about the system, even as individuals come and go [34]

With the rise of cloud services, a bifurcation of roles has occurred. Operations teams at infrastructure companies specialize in the details of providing a reliable service to a large number of customers, while the customers of the service spend as little time and effort as possible on infrastructure [35].

Customers of cloud services still require operations, but they focus on different aspects, such as choosing the most appropriate service for a given task, integrating services with each other, and migrating from one service to another. Even though metered billing removes the need for capacity planning in the traditional sense, it’s still important to know what resources you are using for which purpose so that you don’t waste money on cloud resources that are not needed. Capacity planning becomes financial planning, and performance optimization becomes cost optimization [36]. Additionally, cloud services do have resource limits or quotas (such as the maximum number of processes you can run concurrently), which you need to know about and plan for before you run into them [37].

Adopting a cloud service can be easier and quicker than provisioning and running your own infrastructure, although you still have to learn how to use the cloud service and perhaps work around its limitations. Integration among services becomes a particular challenge as a growing number of vendors offer an ever-broader range of cloud services targeting different use cases [38, 39]. ETL is only part of the story; operational cloud services also need to be integrated with each other. At present, we lack standards to facilitate this sort of integration, so it often involves significant manual effort.

Other operational aspects that cannot fully be outsourced to cloud services include maintaining the security of an application and the libraries it uses, managing the interactions between your own services, monitoring the load on your services, and tracking down the cause of problems such as performance degradations or outages. While the cloud is changing the role of operations, the need for operations is as great as ever.

2. Doing the right thing as a software engineer

The excerpt below is a section from Chapter 14, “Doing the Right Thing”

In the final chapter of this book, let’s take a step back. Throughout, we have examined a wide range of architectures for data systems, evaluated their pros and cons, and explored techniques for building reliable, scalable, and maintainable applications. However, we have left out a fundamental part of the discussion, which we should now fill in.

Every system is built for a purpose; every action we take has both intended and unintended consequences. The purpose may be as simple as making money, but the consequences may be far-reaching. We, the engineers building these systems, have a responsibility to carefully consider those consequences and to ensure that our decisions do not cause harm.

We talk about data as an abstract thing, but remember that many datasets are about people: their behavior, their interests, their identities. We must treat such data with humanity and respect. Users are humans too, and human dignity is paramount [1].

Software development increasingly involves making important ethical choices. There are guidelines to help software engineers navigate these issues, such as the ACM Code of Ethics and Professional Conduct [2], but they are rarely discussed, applied, or enforced in practice. As a result, engineers and product managers sometimes take a cavalier attitude to privacy and the potential negative consequences of their products [3, 4].

A technology is not good or bad in itself – what matters is how it is used and how it affects people. This is true of a software system such as a search engine in much the same way as it is for a weapon like a gun. The ethical responsibility is ours to bear; it is not sufficient for software engineers to focus exclusively on the technology and ignore its consequences.

In contrast to much of computing, however, the concepts at the heart of ethics are not fixed or determinate in their precise meaning; they require interpretation, which may be subjective [5]. What makes something “good” or “bad” is not well defined, and serious discourse on the subject among computing professionals is lacking [6]. Reasoning about ethics is difficult, but also too important to ignore. What does this entail? “Ethics” are not a checklist with which to comply; it’s a participatory and iterative process of reflection, in dialogue with people involved and accountability for the results [7].

Predictive Analytics

Predictive analytics is a major part of why people are excited about big data and AI. It’s also an area that is fraught with ethical dilemmas. Using data analysis to predict the weather, or the spread of diseases, is one thing [8]; it is another matter to predict whether a convict is likely to reoffend, whether an applicant for a loan is likely to default, or whether an insurance customer is likely to make expensive claims [9]. The latter have a direct effect on people’s lives.

Naturally, payment networks want to prevent fraudulent transactions, banks want to avoid bad loans, airlines want to avoid hijackings, and companies want to avoid hiring ineffective or untrustworthy people. From their point of view, the cost of a missed business opportunity is low, but the cost of a bad loan or a problematic employee is much higher, so it is expected for organizations to be cautious. If in doubt, they are better off saying “no”.

However, as algorithmic decision making becomes more widespread, someone who has (accurately or falsely) been labeled as risky by an algorithm may suffer a large number of “no” decisions. Systematically being excluded from jobs, air travel, insurance coverage, property rental, financial services, and other key aspects of society is such a large constraint on an individual’s freedom that it has been called “algorithmic prison” [10]. In countries that respect human rights, the criminal justice system presumes innocence until proven guilty; on the other hand, automated systems can systematically and arbitrarily exclude a person from participating in society without any proof of guilt and little chance of appeal.

Bias & discrimination

Decisions made by an algorithm are not necessarily any better or any worse than those made by a human. Everyone is likely to have biases, even if they actively try to counteract them, and discriminatory practices can become culturally institutionalized. There is hope that basing decisions on data, rather than subjective and instinctive human assessments, could be more fair and give a better chance to people who are often overlooked or disadvantaged in the traditional system [11].

When we develop predictive analytics and AI systems, we are not merely automating a human’s decision by using software to specify the rules for when to say “yes” or “no”; we are leaving the rules themselves to be inferred from data. However, the patterns learned by these systems are opaque: even if the data indicates a correlation, we may not know why. If the input to an algorithm carries a systematic bias, the system will most likely learn and amplify that bias in its output [12].

In many countries, anti-discrimination laws prohibit treating people differently depending on protected traits such as ethnicity, age, gender, sexuality, disability, or beliefs. Other features of a person’s data may be analyzed, but what happens if they are correlated with protected traits? For example, in racially segregated neighborhoods, a person’s postal code or even their IP address is a strong predictor of race. Put like this, it seems ridiculous to believe that an algorithm could somehow take biased data as input and produce fair and impartial output from it [13, 14]. Yet this belief often seems to be implied by proponents of data-driven decision making; an attitude that has been satirized as “machine learning is like money laundering for bias” [15].

Predictive analytics systems merely extrapolate from the past; if the past is discriminatory, they codify and amplify that discrimination [16]. If we want the future to be better than the past, moral imagination is required, and that’s something only humans can provide [17]. Data and models should be our tools, not our masters.

Responsibility and Accountability

Automated decision-making raises the question of responsibility and accountability [17]. If a human makes a mistake, they can be held accountable, and the person affected by the decision can appeal. Algorithms make mistakes too, but who is accountable when they go wrong? [18] When a self-driving car causes an accident, who is responsible? If an automated credit scoring algorithm systematically discriminates against people of a particular race or religion, is there any recourse? If a decision by your ML system comes under judicial review, can you explain to the judge how the algorithm made its decision? People should not be able to evade responsibility by blaming an algorithm.

Credit rating agencies are a classic example of collecting data to make decisions about people. A bad credit score makes life difficult, but at least a credit score is normally based on relevant facts about a person’s actual borrowing history, and any errors in the record can be corrected (although the agencies normally do not make this easy). Scoring algorithms based on machine learning, however, typically use a much wider range of inputs and are much more opaque, making it harder to understand how a particular decision has come about and whether someone is being treated in an unfair or discriminatory way [19].

A credit score summarizes “how did you behave in the past?” whereas predictive analytics usually work on the basis of “who is similar to you, and how did people like you behave in the past?” Drawing parallels to others’ behavior implies stereotyping people; for example, based on where they live (a close proxy for race and socioeconomic class). What about people put in the wrong bucket? Furthermore, if a decision is incorrect because of erroneous data, recourse is almost impossible [17].

Much data is statistical in nature, which means that even if the probability distribution on the whole is correct, individual cases may well be wrong. For example, if the average life expectancy in your country is 80 years, that doesn’t mean you’re expected to drop dead on your 80th birthday. From the average and the probability distribution, you can’t say much about the age to which someone will live. Similarly, the output of a prediction system is probabilistic and may well be wrong in individual cases.

A blind belief in the supremacy of data for making decisions is not only delusional, but also positively dangerous. As data-driven decision making becomes more widespread, we will need to figure out how to avoid reinforcing existing biases, how to make algorithms accountable and transparent, and how to fix them when they inevitably make mistakes.

We will also need to figure out how to realize the positive potential of data and prevent it from being used to harm people. For example, analytics can reveal financial and social characteristics about personal lives. On the one hand, this power could be used to focus aid and support to help those who need it most. On the other hand, it is sometimes used by predatory businesses seeking to identify vulnerable people and sell them risky products such as high-cost loans or worthless college degrees [17, 20].

Feedback loops

Even with predictive applications with less immediately far-reaching effects on people, such as recommendation systems, there are difficult issues that we must confront. When services become good at predicting the content users want to see, they may end up showing them only opinions they already agree with, leading to echo chambers in which stereotypes, misinformation, and polarization can breed. We already know the impact that social media echo chambers can have on election campaigns.

When predictive analytics affect people’s lives, particularly pernicious problems arise because of self-reinforcing feedback loops. For example, consider the case of employers using credit scores to evaluate potential hires. You may be a good worker with a good credit score, but suddenly find yourself in financial difficulties due to a misfortune beyond your control. As you miss payments on your bills, your credit score suffers, and you will be less likely to find work. Joblessness pushes you toward poverty, which further worsens your score, making it even harder to find employment [17]. It’s a downward spiral due to poisonous assumptions, hidden behind a camouflage of mathematical rigor and data.

As another example of a feedback loop, economists found that when gas stations in Germany introduced algorithmic prices, competition was reduced and prices for consumers went up because the algorithms learned to collude [21].

We can’t always predict when such feedback loops may happen. However, many consequences can be predicted by thinking about an entire system (not just the computerized parts, but also the people interacting with it), in an approach known as “systems thinking” [22]. We can try to understand how a data analysis system responds to different behaviors, structures, or characteristics. Does the system reinforce and amplify existing differences between people (e.g., making the rich richer or the poor poorer), or does it try to combat injustice? Even with the best intentions, we must beware of the possibility of unintended consequences.

Surveillance

The excerpt below is from another section in Chapter 14, “Doing the Right Thing”

As a thought experiment, try replacing the word “data” with “surveillance”, and observe whether common phrases still sound so good [23]. How about this: “In our surveillance-driven organization we collect real-time surveillance streams and store them in our surveillance warehouse. Our surveillance scientists use advanced analytics and surveillance processing in order to derive new insights.”

This thought experiment is unusually polemical for this book, “Designing Surveillance-Intensive Applications”, but strong words are needed to emphasize this point. In our attempts to make software “eat the world” [24], we have built the greatest mass surveillance infrastructure ever seen. We are rapidly approaching a world in which every inhabited space contains at least one internet-connected microphone, in the form of smartphones, smart TVs, voice-controlled assistant devices, baby monitors, and even children’s toys that use cloud-based speech recognition. Many of these devices have terrible security track records [25].

What is new compared to the past is that digitization has made it easy to collect large amounts of data about people. Surveillance of our location and movements, our social relationships and communications, our purchases and payments, and our health data has become almost unavoidable. A surveillance organization may end up knowing more about a person than that person knows about themselves; for example, identifying illnesses or economic problems before that individual is aware of them.

Even the most totalitarian, repressive regimes of the past could only dream of putting a microphone in every room and forcing every person to constantly carry a device capable of tracking their location and movements. Yet the benefits that we get from digital technology are so great that we now voluntarily accept this state of total surveillance. The difference is just that the data is being collected by corporations to provide us with services, rather than government agencies seeking control [26].

Not all data collection necessarily qualifies as surveillance, but examining it as such can help us understand our relationship with the data collector. Why are we seemingly happy to accept surveillance by corporations? Perhaps you feel you have nothing to hide; in other words, you are totally in line with existing power structures, you are not a marginalized minority, and you needn’t fear persecution [27]. Not everyone is so fortunate. Or perhaps it’s because the purpose seems benign; it’s not overt coercion and conformance, merely better recommendations and more personalized marketing. However, combined with the discussion of predictive analytics from the last section, that distinction seems less clear.

We are already seeing behavioral data about driving, tracked by vehicles without drivers’ consent, affecting their insurance premiums [28], and health insurance coverage that depends on people wearing a fitness tracking device. When surveillance is used to make decisions that hold sway over important aspects of life, such as insurance coverage or employment, it starts to appear less benign. Data analysis can also reveal surprisingly intrusive things; for example, the movement sensor in a smartwatch or fitness tracker can be used to work out what you are typing (e.g., passwords) with fairly good accuracy [29]. Sensor accuracy and algorithms for analysis are only going to get better.

Takeaways

Thanks to Martin for writing this book, and to himself and Chris for doing a revamp for the second edition. The volume is now even more relevant to how we build systems in 2026 and beyond. You can purchase a hard copy from the publisher’s website or Amazon.

The first edition has a timeless quality because it focused on the fundamentals of large systems, and the new second edition follows the same approach, as laid out in its preface:

“Although the landscape of technologies for processing and storing data is diverse and fast-changing, the underlying principles endure. If you understand those principles, you’re in a position to see where each tool fits in, how to make good use of it, and how to avoid its pitfalls. This book focuses on those principles.”

Since the first edition appeared nine years ago, some things have changed in the tech industry:

• Much greater focus on the cloud. Building large systems on top of cloud infrastructure is more common. This brings lower complexity as cloud primitives hide a lot of implementation complexity, but it also means accepting more risk because when the cloud is down, so is your system.

• Systems which AI tools build upon are more relevant. Vector databases, DataFrames (for training datasets), and the processing of large amounts of training data with batch processing systems are relevant to anyone building production AI systems.

• Local-first software. Martin focuses on this area in his work, and with AI, we could see more demand for running models locally. Operating systems like Ubuntu are also focusing on this.

• Formal methods. The advent of AI-generated code means this topic is getting more attention industry-wide, and the second edition covers it.

• Regulation and legal context. Regulations like the EU’s General Data Protection Regulation (GDPR) are something software engineers increasingly need to know about, and the book now covers it.

If I had to summarize the evolution of the book in its second edition, it would be more focus on cloud and AI, and more on local-first software, testing, and how regulations affect engineers. Interestingly, this mirrors how the tech industry has developed over time, too.

I very much appreciate that the book closes with the final chapter focused on “doing the right thing” as a software engineer. Software systems have wide-ranging societal impact, and engineers working on these systems have a great say in what gets built, and how it gets built. As engineers, we owe it the very least to ourselves to consider the broader impact of our decisions — and doing so might also force us to make important ethical choices. There’s less discussion of the ethics angle on software engineering: and I’m glad that Martin and Chris did not shy away from going deeper into this topic.

If you’d like to get more background on the book – and on the hard parts of building large-scale systems – check out our podcast episode with Martin Kleppmann.

Today’s article focuses on Linux and Ubuntu, and we’ll cover Windows in a follow-up issue. Obviously, I reached out to Apple but heard nothing back, unsurprisingly. If you’re reading this and happen to work at Apple, it’d be great to learn more!

Jon Seager is VP of Engineering at Canonical – the company behind Ubuntu – and has provided new details about what the team there has built for AI support, and some new ideas that they’re brewing up. Today, we cover:

1. Hardware enablement: support for GPUs, NPUs and DPUs. When you turn on a machine with AI accelerators, Ubuntu aims for the hardware to perform at its full potential. This means having proper driver support for PCs and cloud data centers’ computing units.

2. Hardware partnerships. Working closely with NVIDIA, AMD, and Intel means Ubuntu can support those vendors’ new hardware from release day.

3. CPU architecture variants. New versions in a CPU family add to, or change, features. An operating system needs to support a new version of the CPU architecture variant in order to fully utilize it. Ubuntu does this for the x86‑64 family, making it a lot more performant on newer CPUs – while still supporting older CPUs.

4. Local-first bet & plans for agentic workflows. There’s a big focus on running local models and using “inference snaps” which help choose the right model with the right quantization. There is the intention to support agentic workflows at the OS level, one day, which is currently at the early exploration stage.

5. Developer ecosystem. There’s a plan to add more support for AI dev tools, a focus on sandboxing at the OS level, a push to support ARM64 laptops more, and we touch on the popularity of Windows Subsystem for Linux (WSL).

6. Engineering culture. A skeptical attitude to AI at Canonical has given way to one where experimentation is encouraged and devs lean into AI tools, but there are no targets for token usage or amounts of AI-generated code.

7. What other Linux distributions are doing. Arch Linux takes the “DIY your AI setup” approach, Omarchy makes it easy to install AI tools, while Red Hat Enterprise Linux ships with AI integrated into the command-line and support for AI accelerators & popular AI tools.

The bottom of this article could be cut off in some email clients. Read the full article uninterrupted, online.

Read the full article online

1. Hardware enablement: support for GPUs, NPUs & DPUs

Jon mentioned he detects a “Dotcom Boom”-era vibe in the industry, like around when “web 1.0” was created, and indeed, lots of startups today aim to be the Google-style success story of this “AI era”. At Canonical, the team asked: what does that mean for Ubuntu as an operating system?

For instance, should Ubuntu join the competition and try to position itself closer to AI, or keep focusing on what they’ve done for decades: build an operating system? Jon said:

“We need to make sure to remain a relatable and accessible system. I don’t think we should blur the line between application features and the OS itself. So, the most powerful thing we can do is hardware enablement.”

Hardware enablement means that if a computer (typically, a laptop) has AI-related hardware, Ubuntu should allow it to make full use of it. This involves adding support for GPUs, NPUs, DPUs and other types of accelerator cards. Let’s briefly go through each.

GPUs

As is likely widely known by readers, ‘GPU’ stands for Graphics Processing Unit. Originally built for graphics rendering, its #1 use case is no longer in video games but for AI training and inference. GPUs come in two forms:

• Integrated GPUs: located on the same die (integrated circuit) as the CPU, like GPUs on Apple’s M-series processors

• Discrete GPUs: separate chips on their own board; often for gaming, or in standalone GPU rigs for AI and ML workloads

NVIDIA leads the market in discrete GPUs for rigs with its Blackwell family, and in standalone GPU cards with the NVIDIA RTX series. Other vendors like AMD offer GPUs for data centers (like the Instinct MI300 Series) and for PCs with the AMD Radeon series.

NPUs

Neural Processing Units (NPUs) are also called “AI accelerators.” This is a dedicated block on the System-on-a-chip (SoC), on modern processors especially designed for running AI inference efficiently on‑device. Since 2022, many modern processors have had a dedicated NPU block, including all Apple’s M-series chips (from M1 and up), Intel’s Core Ultra and Core Ultra “Series 2”, AMD’s Ryzen AI 300 series, and also Qualcomm’s Snapdragon X Elite and Snapdragon X Plus.

A number shared for each NPU is TOPS. TOPS means Tera (trillions) of Operations Per Second, and the said operation is a “multiply-accumulate” (MAC) one, which Qualcomm describes as:

“A multiply-accumulate (MAC) operation executes the mathematical formulas at the core of AI workloads. A matrix multiply consists of a series of two fundamental operations: multiplication and addition to an accumulator. A MAC unit can, for example, run one of each per clock cycle, meaning it executes two operations per clock cycle. A given NPU has a set number of MAC units that can operate at varying levels of precision, depending on the NPU’s architecture.”

How TOPS is calculated: TOPS = 2 × MAC unit count × Frequency / 1 trillion.

“Frequency” refers to the clock speed (cycles per second) at which an NPU and its MAC units (as well as a CPU or GPU) operate, which directly influences overall performance. Processors at higher frequencies allow for more operations, but higher frequencies also mean more energy consumed, heat generated, and battery life decreased. The TOPS number that’s quoted for processors is generally the peak operating frequency.

NPUs are often ideal for low-power, local inference, and for running smaller, local models. They can be useful for things like Local speech‑to‑text (dictation, captions, meeting transcription), video background blur/replacement or auto‑framing, small local language summarization, etc. NPUs are more typical of laptop and PC processors, although some phone processors ship with them like the iPhone (A-series chips) and Google’s Tensor processor in Pixel phones. Basically, NPUs promise to bring efficiently-running local models on laptops one step closer.

DPUs

Data Processing Units (DPUs) are typically found in data centers, moving massive amounts of data fast. NVIDIA’s explanation:

“The CPU is for general-purpose computing, the GPU is for accelerated computing, and the DPU, which moves data around the data center, does data processing.

A DPU is a new class of programmable processor that combines three key elements. A DPU is a system on a chip, or SoC, that combines:

• An industry-standard, high-performance, software-programmable, multi-core CPU, typically based on the widely used Arm architecture, tightly coupled to the other SoC components.

• A high-performance network interface capable of parsing, processing and efficiently transferring data at line rate, or the speed of the rest of the network, to GPUs and CPUs.

• A rich set of flexible and programmable acceleration engines that offload and improve applications’ performance for AI and machine learning, zero-trust security, telecommunications, and storage, among others.”

Several major chipmakers manufacture DPUs, of which NVIDIA’s BlueField family is the most widespread. Others include AMD Pensando DPUs (Elba, Giglio), and Intel IPU / DPU cards (E2100, E2200 series).

DPUs are most commonly deployed inside Hyperscale cloud providers (AWS, Azure, GCP, OCI), or in AI and high-performance computing (HPC) data centers, or larger private clouds. DPUs make sense when GPU traffic is huge, or when the network telemetry overhead is so great that it could overwhelm the CPUs processing the data transfer.

2. Hardware partnerships

It’s easiest to add support to hardware by working with leading chip manufacturers, so Ubuntu has relationships with hardware vendors for that reason. As a result, the OS sometimes offers day-one support for cutting-edge AI supercomputers.

Partnership with NVIDIA

In September 2025, Canonical announced it would package and distribute the full NVIDIA CUDA toolkit directly within Ubuntu’s repositories. This deal collapsed into a single standard apt install, something that had previously been a multi-step manual installation process of downloading from NVIDIA’s site, importing GPG keys, pinning a separate APT repo – and praying nothing broke.

Packaging and distributing the CUDA toolkit makes developing with CUDA easier. From Jon:

“One of the trickiest things for developers who have to use this tech is the dance of matching the right version of Python, with the right version of CUDA, with the right driver. Projects end up with different versions of CUDA, and then machines end up breaking because the driver configuration gets inadvertently broken along the way.

The number one thing we can do as an operating system is to make this setup as easy as possible.”

Ubuntu’s strategy of working directly with chipmakers seems to be working. NVIDIA recently discontinued its custom NVIDIA DGX OS — a modified Ubuntu it maintained for years — and now ships plain Ubuntu. Jon:

“Previously, NVIDIA shipped NVIDIA DGX OS for which NVIDIA had an agreement with Canonical where they could take Ubuntu, modify it with the kernel modules and software they needed, do some product-specific optimization, and ship that as NVIDIA DGX OS.

This more recent development sees NVIDIA just shipping Ubuntu as it comes.

When NVIDIA released the DGX Spark, a $4,000 AI workstation with an ARM64 chipset, it shipped running vanilla Ubuntu as the only supported operating system.”

At CES 2026 in January, Canonical announced Ubuntu support for the NVIDIA Vera Rubin NVL72 rack-scale architecture, with day-one platform readiness in Ubuntu, version 26.04 LTS (Long-Term Support: at least 15 years for enterprise customers).

AMD and Intel

It’s clear Ubuntu and NVIDIA enjoy a strong partnership, but Canonical aims to remain neutral, Jon says:

“We have an amazing partnership with NVIDIA, but we do the same with Intel, the same with AMD, the same with Qualcomm, and the same with MediaTek because in reality there is hardware being released every day, and if we don’t maintain those partnerships, the ecosystem becomes even more fragmented than it already naturally is.”

Last December, Ubuntu announced native support for AMD ROCm, and also ships with Intel’s OpenVINO toolkit. Ubuntu 26.04 LTS will be the first major distribution to natively package all three GPU compute stacks — NVIDIA, AMD, and Intel — with long-term enterprise support. Under Ubuntu Pro, ROCm LTS releases receive up to 15 years of security maintenance.

Security maintenance means that if vulnerabilities or critical incompatibilities are discovered in an LTS version, Canonical will patch them even if the upstream vendor no longer supports those versions and no longer backports security patches.

AMD Instinct accelerators are gaining traction in HPCs and sovereign AI deployments, as enterprises look for alternatives to CUDA-locked hardware. AMD’s SVP and Chief Software Officer, Andrej Zdravkovic, said the partnership would make it “easier for developers and enterprises to deploy AMD solutions on supported systems.”

Chip vendors want to collaborate because it means less work for them to add operating system-level support. Jon:

“It’s a win-win on both ends. Silicon companies are in the business of building the best chips they can, and partnering with Canonical means they have to concentrate on fewer things which are not their core focus. My hope is that partnering with Canonical helps them to focus on what they’re best at, while enabling us to help with what we’re best at: integrating, shipping and maintaining a Linux distribution.”

3. Architecture variants

Modern x86 processors support multiple instruction set generations: x86_64 v1, v2, v3, v4, and v5. ARM has a similar hierarchy. Each generation adds capabilities, such as AVX-512 instructions that accelerate machine learning workloads.

Let’s take the x86_64 instruction set. The instruction set is versioned. These are the versions:

• For x86_64: v1, v2, v3, v4, v5…

• For ARM: ARM v8.2, v8.3, v9…

Until recently, Ubuntu ran slower on newer CPUs in order to keep supporting older ones. So, when installing Ubuntu compiled for AMD64, the OS supported architecture variants for AMD64 v1.

Supporting v1 has the advantage that the oldest of AMD64 processors can run this Ubuntu version. But if Ubuntu decided to support v2 instructions, then v1 processors could not run the OS! The OS did not use the new instructions; for example, a modern processor with hardware accelerators like AVX-512, didn’t use them.

Canonical has reworked its build infrastructure to produce binaries with specific architecture variant support. So, in the case of running an x86_64 v3 compatible processor, you can download an Ubuntu OS variant that’s compiled specifically for x86_64 v3.

One tradeoff the Ubuntu team had to make was building binaries several times, which takes up more processing time and storage at their end. Then again, the Ubuntu team doing this once means that users don’t need to do recompilation, which made it an easy tradeoff, Jon told me.

Now, Ubuntu supports x86_64 v3 as an architecture variant and plans to do more. Jon says:

“Today, we’ve released x86_64 v3 as a variant, but the capability in our build and delivery pipelines unlocks the ability to add variants for the next RISC-V RVA versions, for ARMv9, ARMv10, ARMv11 and so on.

We will start now onboarding variants to make sure that when you go and buy your latest Snapdragon laptop, your operating system and all of the parts of it are using the silicon to its fullest.”

Adding support for architecture variants was a significant undertaking. Jon explains:

“This work was especially complex because combined with having the hardware physically available in the build farm, Canonical also needed to make the build scheduler aware, and thread the capability through the build systems of Debian packages, Snaps, OCI images, virtual machine images, etc. As it stands, the capability exists for Debian packages, and support for further package types will land shortly.

In addition to the build infrastructure, work needed to be done on downstream package managers (apt, snap, …) and schedulers to ensure they pull the right version of packages, and consideration needs to be given to what happens if a VM containing x86_64 v3 code ends up trying to boot on v1 hardware, and so on.”

4. Betting on local-first & plans for agentic workflows

If you’ve tried to run an LLM locally on your machine, you’ll know it comes with friction. Jon:

Read more

Tech interviews have two parts: the technical interview – with a focus on things like coding, software architecture, problem solving – and the behavioral part – with a focus on past experience, and the situations that show you’d be a good fit at the company you’re interviewing with, along with things like attitude, motivation, culture fit. Technical interviews are going through a big change, thanks to AI tools: some companies are bringing in new, AI-assisted types of interviews, while others are trying to make “pre-AI” type interviews work.

What doesn’t seem to be changing is the second type of interviews: the behavioral ones. I’ve found the topic of behavioral interviews from a software engineer’s perspective somewhat under-discussed – even though this interview carries huge weight in securing an offer and what level you come in at. No matter how strong your technical skills are, especially at mid-sized and larger companies, you are unlikely to get an offer if you are deemed to not be a fit for what the company is looking for.

Steve Huynh was an engineer at Amazon for 17 years – I previously did a podcast episode with him on the reality of being a principal engineer at Amazon. During this time, Steve conducted nearly 1,000 interviews, of which around 600 were Bar Raiser ones. Bar Raiser interviews are unique to Amazon: it’s an interview conducted by someone outside of the hiring team, with the goal of ensuring that the new hire raises the company’s talent bar.

After leaving the e-commerce giant, Steve spent 2 years researching and writing the book Technical Behavioral Interview: An Insider’s Guide.

Today, we cover two topics on interviews and behavioral interviews:

1. Learnings from conducting ~1,000 behavioral interviews at Amazon. Steve reflects of major observations from his 17 years at Amazon, covering:

• You’re over-prepared for one interview and unprepared for the other

• How you deliver the story matters as much as the story itself

• The interview is an audition for what it’s like to work with you

2. What companies are looking for during behavioral interviews. An excerpt from Steve’s new book, Technical Behavioral Interview, covering ~75% of a full chapter of the book (out of the 14 total chapters.) We get into:

• Understanding fit: role and company

• The four dimensions that determine your level

• What each level looks like

• Reading and calibrating your own level

• Researching what companies really value

Longtime readers might remember Steve from my podcast with him a year back: What is a Principal Engineer at Amazon? With Steve Huynh

My usual disclaimer: as with all my recommendations, I was not paid for this article, and none of the links are affiliates. See my ethics statement for more.

With this, it’s over to Steve:

1. Learnings from conducting ~1,000 behavioral interviews at Amazon

A Bar Raiser is a specially trained interviewer whose job is to ensure that every hire raises the average talent level at Amazon. I had veto power over any candidate. I sat on nearly a thousand interview loops across every level from intern to Principal Engineer.

After 50 or so interviews as a Bar Raiser, the patterns became impossible to miss. And this was the biggest one:

The candidates who didn’t get offers seldom failed because they lacked technical skill. They failed because of how they presented themselves.

For sure, technical preparation is crucial, and I’m not telling you to skip it. But most candidates have massive blind spots when it comes to non-technical matters, which is a big problem. Why? Because that blind spot is where most hiring decisions are made.

The Bar Raiser who trained me put it this way:

Technical skills are the ante. They get you into the game. But they’re not what wins you the hand.

I didn’t fully appreciate what that meant until I’d seen candidates who were technically very strong get rejected because of everything else.

Think about it. By the time you’re sitting in a final round of interviews, you’ve already passed at least one technical screen or take-home assignment. The company already knows you could probably do the job. They already know you want to work with them.

But that’s not what the final round is for.

The final round is when the team figures out whether they want to work with you. Being technically proficient is part of it, but it’s not all of it. Can you explain your thinking clearly when you’re stumped? How do you handle it when things go wrong? Can they picture you in a design review or in a tough conversation with a partner team?

Fit.

Fit is what decides most hiring outcomes, yet it’s the thing most candidates spend the least time preparing for. After nearly a thousand interviews, I can tell you exactly where the gap is and how you can close it.

Learning #1: You’re over-prepared for one interview and unprepared for the other

The average candidate preparing for a tech interview probably spends 95% of their time on technical preparation and 5% on everything else. Some spend literally zero on everything else.

I get why. Technical preparation feels concrete. You can grind coding problems and measure your progress. You can study system design patterns and feel yourself getting sharper. There’s a clear input/output relationship. Do more problems, get better at problems.

For most technical interviews, even if you haven’t seen the exact problem before, you can still do a decent job. It’s simply not possible to prepare for every problem, so it’s expected that you can reason through an unfamiliar coding question and pick up on hints the interviewer gives you. You can work through a system design problem by applying fundamentals you already know. It’s expected that you will encounter new questions during an interview, so it isn’t fatal if you’re a competent engineer who can think on your feet.

However, the non-technical rounds are the opposite. You cannot wing them and expect to do well. When an interviewer says, “Tell me about a time something went wrong on a project and how you handled it” and you haven’t thought about that question before, there is no hint they can give you. There’s no reasoning your way through it in real time. You either have a prepared story ready to go, or you’re going to mumble your way through a word salad while the interviewer watches.

I’ve seen this play out hundreds of times. A candidate would crush the coding round, then I would ask them about a difficult decision they made, and they would fall apart. They would pick a half-remembered example, start rambling, backtrack to add context they forgot, in the process losing track of the question. Then, five minutes later, they would land on something like, “So, yeah, it worked out in the end.”

These candidates were often strong coders, but that didn’t matter. At the debriefs, the feedback was always some version of “I couldn’t get a concrete answer about their experience. Every story was vague and unconvincing.” We couldn’t extend an offer when a candidate couldn’t articulate how they worked.

The technical bar was met, but the hiring decision was made in the behavioral round.

Here’s what’s frustrating about this. Non-technical preparation takes a fraction of the time for technical.

If you’re going to spend 80 to 100 hours preparing for an interview cycle, spending a single weekend on your stories might be the highest-leverage investment you make.

Ten hours of story prep can completely change the outcome of your behavioral rounds. Meanwhile, your 80th hour of LeetCode will give you almost nothing you didn’t already have at 60.

The returns on technical prep diminish rapidly. The returns on story prep are exponential because almost nobody does it at all.

What to do: How are you currently splitting your interview prep time? If it’s 99% technical and 1% everything else, you’re over-indexed on the part with diminishing returns and under-indexed on the part where hiring decisions get made. You don’t need to cut your technical prep dramatically. Just reallocate. If you’re planning to spend 80 hours preparing, take 10 of those hours and move them to non-technical preparation. That reallocation will do more for your odds than 10 more hours working on practice problems.

Learning #2: How you deliver the story matters as much as the story itself

You can have the most impressive accomplishment of your career ready for your interview and completely waste it with bad delivery. The most common version of this is what I call the “ramble and stumble.”

The candidate starts talking, and you genuinely can’t tell if they’re figuring out the story as they go or if they’ve simply never said these words out loud before. Or they might give you five minutes of context and then still backtrack to add details they forgot. By the time they reach the outcome, you’ve lost track of how you got there.

Here’s something that’s always struck me as odd. If you had a big presentation at work, you’d spend hours preparing for it, right?. You’d think about the structure, the flow, the key points. You’d rehearse it. You might even do a couple of dry runs with a colleague. Nobody wants to walk into a presentation and wing it.

But in a job interview, where the stakes are arguably higher than any single presentation you’ll ever give? People wing those constantly. They walk in having never practiced their stories out loud. They might have thought about them, but they’ve never spoken the words, heard how they sound, or timed how long they take. Then they’re surprised when the words come out as a mess.

Think about any other high-stakes skill. You wouldn’t expect to be good at golf without practicing at the driving range. You wouldn’t expect to give a great keynote the first time you stepped on stage. Nobody calls a musician fake for rehearsing before a concert.

But for some reason, many people feel that preparing interview stories is inauthentic. As if it’s cheating somehow. As if the “real” version of you is the one that stumbles through an unrehearsed answer under pressure.

It’s not. The real you communicates clearly what you’ve done and what you’re capable of.

What to do: Good delivery doesn’t require a lot of charisma or natural presentation skills, but it does require practice. Start with the two questions that come up in virtually every interview: “Tell me about yourself” and “Why do you want to work here?” Write down your answers. Then record yourself delivering them. Watch the recording and take notes. Where did you ramble? Where did you fill space with filler words? Did you look nervous? Then do it again. And again. Keep going until you watch the recording back and think “That sounds like someone I’d like to work with.”

Once those two are solid, pick stories from your career and do the same thing. This process will be uncomfortable at first. Most people hate watching themselves on camera. Do it anyway. Thirty minutes of this will up-level your interview performance much more than 20 hours of coding exercises could ever do.

Learning #3: The interview is an audition for what it’s like to work with you

Most candidates think the interview is an exam. If you get the right answers, then you’ll pass the test and get the job. That’s simply not how it works. Yes, you are being evaluated, and what you say matters. But there is no answer key. The interviewer doesn’t have a rubric with the “correct” responses to which they compare your answers. They’re forming an impression of you as a person, and that impression is far more nuanced than “right” or “wrong.”

By the time you’re sitting across from the interviewer, you’ve already jumped through some technical hoops. The company already has evidence from your resume that you can code or design systems at the level they need. That bar has been cleared. The final round goes deeper on the technical side, but it’s also trying to answer a completely different question: Would we want this person on the team? Would we trust their judgment in a crisis? Would they make our team’s software better or worse?

As a Bar Raiser, my specific job was to determine whether a candidate would raise the bar, meaning that they would be better than at least 50% of the people already at the company in that role.

The thing most people don’t realize is that the type of coding we asked about in interviews wasn’t what we did on the job. Nobody was writing algorithms on a whiteboard during their workday. The questions we asked tested problem-solving ability in an artificial environment.

But the behavioral questions, the soft questions, those tested situations we dealt with every single day. Navigating disagreements, handling projects that were going sideways, influencing without authority, making tradeoffs with incomplete information. These weren’t hypothetical scenarios pulled out of a textbook. They were just another Tuesday.

So when I asked a candidate to tell me about a time they had to push back on a stakeholder, I wasn’t waiting to hear the right answer; I was picturing them in our next planning and prioritization meeting. When they described how they handled a conflict on their team, I was asking myself whether I’d want to be in that room with them. Every answer was a preview of what it would be like to work alongside that person day to day.

The candidates who treated it like a test tried to figure out what I wanted to hear and then gave me that answer. That’s exactly the wrong approach. They gave polished, rehearsed answers with no rough edges and perfect endings where everything worked out and every decision was the right one. I’d walk out thinking “I have no idea what it would actually be like to work with this person.” And when that uncertainty showed up across multiple interviews in the debrief, it almost always turned into a “No.”

What to do: For each story you’re preparing, stop thinking about what the interviewer wants to hear. Instead, think about what you’d want to hear from someone interviewing to join your team. You’d want to hear how they actually think. You’d want the real version of what happened, including the parts that were hard and the calls that were close. You’d want to walk away feeling like you understood what it would be like to work with them on a tough problem. Give your interviewer that same thing. Be honest and let them see how you think. That’s worth more than any polished answer.

What ~1,000 interviews taught me

After all those interviews, the lesson I keep coming back to is simple.

The people who get hired are the ones who can walk into a room and tell a clear story. This story is about their work and their capabilities, and makes the interviewer think, “I want to work with that person.”

Being able to tell this story is a skill. And like any skill, it gets better with practice. Most people never practice it because they don’t think of it as something you can prepare for, but you can. And a little preparation here goes further than almost anything else you can do for your career.

2. What companies are looking for during behavioral interviews

The below are excerpts from Chapter 2 from Technical Behavioral Interview: An Insider’s Guide. Some sections have been cut out and lightly edited for this article. Copyright © 2026 Steve Huynh. Used with permission.

Technical skills alone don’t determine your offer. Otherwise, those who can solve the coding and system design problems would get the same result. Instead, companies use behavioral interviews to answer two critical questions: Do you fit with both the role and the company? And if you do fit, at what level will you be most effective?

Get both right, and you will receive an offer at the appropriate level. Get the fit wrong, and you’ll be rejected regardless of your skills. Get the level wrong, and you’ll be either down-leveled or rejected for being underqualified.

This chapter explains how companies make their assessments of fit and level by analyzing the signals in your stories. Once you understand these dimensions, you’ll pick better stories and signal the right level.

Understanding Fit: Role and Company

The primary consideration for any tech role is whether you have the technical skills to do the job. Companies will assess this mostly through the technical parts of the interview, for example, coding challenges, system design, or whatever technical evaluation matches your role. If you can’t demonstrate the core technical capability, nothing else matters.

But technical skills alone don’t predict success. Companies learned this the hard way by hiring smart people who couldn’t work effectively in their environment. That’s why behavioral interviews focus on two additional types of fit:

Role Fit: Can you handle the specific challenges and working conditions of this position? A backend role at a fast-growing startup requires different capabilities than a backend role at an established enterprise. The technical skills might be similar, but the role demands will be different.

Company Fit: Will you thrive in the environment in which this organization operates? This goes beyond surface-level culture. They are assessing whether your working style, decision-making approach, and values match with how the company gets things done.

How Companies Detect Fit Through Signals

Companies can’t directly ask the question, “Would you fit here?” What candidate would torpedo their chance of success by answering with a “No”? Instead, companies look for signals in your stories that indicate alignment or misalignment.

Role Fit Signals emerge from how you describe handling situations similar to what the role requires:

• If the role requires working with ambiguous requirements, do your stories show comfort with uncertainty?

• If the position involves cross-team coordination, do you show an ability to cope with organizational complexity?

• If the job needs rapid iteration, do your examples show shipping quickly and adjusting based on feedback?

Company Fit Signals come from the choices you made and how you describe them:

• A company that values “bias for action” looks for stories that show you moving quickly despite incomplete information.

• An organization that prizes “customer obsession” wants to hear examples of you going deep to understand user needs.

• A place that emphasizes “radical transparency” seeks stories that show you sharing information openly, even when you’re uncomfortable.

The same story can send different signals to different companies. You spending three weeks perfecting a solution might demonstrate attention to quality at one company but analysis paralysis at another. Moving fast and fixing issues later demonstrates good judgment at a growth startup but recklessness at an established healthcare company.

Common “Mis-Fits”

Even a talented candidate will get rejected sometimes if they are not a good fit. The same behaviors that are positive at one company can signal poor fit at another.

Independence vs. Collaboration: This covers both how you work and how you make decisions. Some companies need people who pick up a problem, run with it, and come back with a solution. Others expect you to bring the team along at every step. These often go together: companies that want you to work solo also tend to want you to make calls on your own, and companies that want collaborative work also want group buy-in on decisions.

If every story you tell involves going off and building something alone, consensus-driven companies will worry you’ll steamroll people or make choices that won’t stick. Flip it around: if every story involves checking with the group before you act, companies that prize individual ownership will wonder whether you can make a decision without a meeting.

Speed vs. Thoroughness: Startups often need rapid experimentation, where you ship MVPs and iterate based on feedback, while companies in healthcare or finance require careful validation before any release. This tension also shows up in how teams think about code quality: some organizations will happily spend extra weeks on clean architecture, while others want a working solution on deadline even if the code needs cleanup later. Whereas stories about methodical testing might bore a startup, your “ship it and fix it” examples could terrify a medical device company.

Excellence vs. Pragmatism: Some organizations value technical excellence and clean architecture above all else. Others need pragmatic solutions that ship on deadline even if imperfect. Focusing on perfect code fails at deadline-driven companies, just as accepting technical debt everywhere fails at companies maintaining critical infrastructure.

Innovation vs. Stability: Some roles require creating new solutions and challenging existing approaches, while others need you to maintain and optimize proven systems. If you say that you’re constantly reinventing established processes, teams that value stability will not consider you a good fit. Conversely, stories that show you only follow existing patterns will disappoint teams that are looking for creative problem-solving

Direct vs. Diplomatic: Some cultures prize radical candor and want you to say exactly what you think. Others value maintaining harmony and face-saving communication. If you are too blunt, you will not fit in well at a relationship-focused company. If you are not direct enough, you will not like working at a company that values “disagree and commit.”

Data vs. Intuition: Some companies require data to justify every decision (”data-driven” cultures), while others trust experienced judgment and move on gut feel. Showing that you make decisions based on instinct does not impress analytical companies, and telling a company that values experienced judgment that you conduct three A/B tests to choose a button color will get you struck off their list.

Specialist vs. Generalist: Large companies often want deep experts who master one domain, while smaller companies need people who are comfortable wearing multiple hats. Know which sort of company you are walking into.

Once you understand fit, you can pick stories that match the company and the role.

The Four Dimensions That Determine Your Level

Companies assess your level through four dimensions that appear in every story you tell. Each dimension reveals different aspects of your capability. Together, they show the company where you operate most effectively.

Scope (Dimension #1)

Scope provides a measure of the number of people on your team and, extending outward as you advance, whose work was affected by your actions. The greater the number affected, the higher your level for this dimension.

Entry Level: Your work affects your own productivity and starts to help other team members. For example, you might improve how you handle assigned tasks or fix issues that were slowing down a few teammates.

Mid Level: Your work affects aspects of the team and shapes how it operates. You might redesign a process that changes a significant part of how your team works or solve problems that affect most of the team’s effectiveness.

Senior Level: Your work directly impacts your entire team and is beginning to influence at least one other team. Perhaps you create solutions that change how your whole team operates and affect workflows in adjacent teams, or you solve problems that require coordination with other groups. You may also start collaborating more closely with product or design partners on your immediate team’s work.

Staff Level: Your work directly impacts at least two teams and is beginning to have an influence on the broader division or organization. Examples of this include developing technical strategies that change how multiple teams make decisions and solving problems that require buy-in across several parts of engineering. Your influence extends beyond engineering into product, design, and program management as you shape solutions that affect how cross-functional partners work.

Principal Level: Your work affects many teams or changes how large parts of the organization operate. Perhaps you have created technical strategies that have influenced how dozens of teams make decisions. Or you have solved problems that cut across a large engineering organization. At this level, your influence regularly extends into business strategy, shaping decisions alongside product, design, program, and business leadership.

Contribution (Dimension #2)

Contribution captures what you did, not what happened around you. It is important to be precise about the line between “I” and “we.” Companies will expect to see evidence of increasing leadership and ownership as you advance in your career.

Entry Level: You execute assigned work and are beginning to take ownership of small pieces. Examples: implementing solutions designed by others; fixing bugs in existing systems; taking full responsibility for well-defined features within larger projects.

Mid Level: You own complete solutions from problem to implementation while also guiding others. Perhaps you have identified issues, designed the approaches, implemented them, and you have verified that they work, and you have helped your teammates understand the reasons for your decisions.

Senior Level: You lead initiatives requiring coordination. You’re expected to make progress even when the requirements are unclear or the path forward is uncertain. Examples of this include driving technical decisions for your team; mentoring others through complex problems; architecting solutions to be implemented by others; and ensuring quality work outcomes for many people.

Staff Level: You lead cross-team initiatives and establish technical direction, often in situations where the right approach isn’t obvious and stakeholders have competing priorities. This could look like defining technical approaches that are adopted by multiple teams, creating systems that enable other teams to solve problems on their own, or driving agreement on complex technical decisions across several teams.

Principal Level: You create organizational capabilities and establish new ways of working. At this level, you’re frequently operating in highly ambiguous environments where you must define the problem before you can solve it. You might define technical standards that guide dozens of teams, build systems that enable others to solve entire classes of problems, or transform how the organization approaches its hardest challenges.

Impact (Dimension #3)

Impact shows what changed for the better as a result of your work. Companies want to see that your work produced results worth the investment. Strong stories put numbers on the impact and connect technical wins to business or user outcomes.

Entry Level: You improve your personal productivity and are starting to help the team work better. Examples include reducing the time you spend on repetitive tasks, fixing issues that were slowing down teammates, or improving the quality of code in the areas you touch. Even simple measures matter at this level: time saved or bugs prevented.

Mid Level: You improve team effectiveness in specific areas and influence team-wide practices. Perhaps you reduced deployment times for specific workflows, eliminated categories of bugs in your domain, or you created tools that have made the team more productive in particular areas. You can quantify these improvements and connect them to broader outcomes like feature velocity or reliability.

Senior Level: You transform how your entire team works and are starting to have an impact beyond your team. For example, you might have introduced new workflows that changed your team’s capabilities. Or perhaps you eliminated major sources of operational problems, or the improvements that you have created have been adopted by adjacent teams. Your impact extends beyond just engineering metrics to product outcomes, user experience, or operational costs.

Staff Level: You improve how multiple teams operate and drive organizational improvements. These sorts of impact come from achievements such as establishing practices that several teams adopt, solving infrastructure problems that were impeding multiple teams, or creating new capabilities that open up new types of work across teams. Your measurable impact can be tied to business metrics like revenue, customer retention, or time-to-market.

Principal Level: You create organizational capabilities and drive strategic changes. Impact at this level could come from establishing technical foundations that dozens of teams use to build upon, solving problems that were blocking major business initiatives, or creating leverage that compounds benefits across the company. Your impact is measured in business outcomes and strategic capability, not just technical improvements.

Difficulty (Dimension #4)

Difficulty reflects the complexity of problems you’ve tackled, the constraints you have faced, and the trade-offs you have managed. Under this category, solving easy problems with big impacts is less impressive than hard problems solved well.

Entry Level: You work on straightforward problems within established patterns. For example, you might face challenges learning new technologies or debugging unfamiliar code, but the path forward becomes clearer once you understand the problem or ask for help.

Mid Level: You work through challenges and obstacles in your work. The problems you tackle have more moving parts and less obvious solutions. These could be competing requirements or having to work through technical complexity you haven’t seen before. Or perhaps you have had to manage dependencies within your team that affected your timeline or figure out solutions when the approach wasn’t immediately obvious.

Senior Level: You manage constraints and make technical decisions with team-level architectural implications. The problems you solve involve multiple interacting systems and competing concerns. You might have to balance needs across multiple stakeholders with different priorities. Maybe you make architectural decisions that affect how your whole team works, or you have to work around technical limitations that require creative solutions, or solve problems that require you to address both technical and business factors.

Staff Level: You manage competing trade-offs across multiple teams while handling problems with significant technical and organizational complexity. Examples of difficulty at staff level include:

• Balancing different technical approaches when teams have genuinely conflicting needs.

• Creating solutions that affect how several teams work together.

• Making architectural decisions that have to work across diverse contexts.

• Getting teams to agree when the technically optimal solution differs for each team.

Principal Level: You handle fundamental trade-offs between competing organizational needs or solve problems where no clear solution exists. The complexity at this level often involves novel problems that lack established patterns or precedents. You might balance technical excellence against delivery speed at organizational scale; work within organizational constraints while maintaining technical integrity; create approaches for entire classes of problems the company hasn’t solved before; or make decisions that affect company strategy and require executive buy-in.

What Each Level Looks Like

Here’s how the same types of accomplishments look across each level. These aren’t templates. They’re meant to help you develop a sense for the difference between a mid-level story and a senior one. Compare adjacent levels and notice what actually changes as you move up and down.

Researching What Companies Really Value

You’ll never have perfect information about what a specific company values, but a little focused research will often reveal surprising insights that most other candidates will miss. The difference between having even partial intelligence and going in blind can be whether or not you emphasize the right things in your stories.

Start With Your Recruiter

Most candidates treat recruiters as gatekeepers to avoid, but if you do this, you will waste your best source of insider information. Recruiters want you to succeed, because their performance is based on the number of accepted offers received by the candidates they put forward. They have prep materials, they know the interviewers’ focus areas, and they understand what they are looking for.

Ask your recruiter directly: “What should I know about this company’s current challenges?” Or “What competencies matter most for this role?” Or “Can you share any interview prep materials?” Many recruiters have documents about interview format, team priorities, or even the specific behavioral competencies they evaluate. The questions that are used as examples in the prep materials have a high likelihood of being asked in the interviews.

Mine Publicly Available Information

When companies repeat certain words when describing job opportunities, they’re telling you what matters. For example, a job posting that mentions “fast-paced” several times signals something different than one emphasizing compliance. Those words are there for a reason.

Where to dig:

• Engineering blogs: How do they describe their wins? What problems do they celebrate solving?

• Tech talks and conferences: What topics do their engineers present? Speed of delivery? Scale? Innovation?

• Open source contributions: What they choose to open source reveals their priorities. If they open source developer tools, this suggests they value community. If they are happy to make internal tools public, this shows transparency.

• Technical documentation: The existence of public API docs or technical guides (and the quality thereof) shows how they support both users and their own teams.

• Status pages and postmortems: Companies that publish detailed postmortems demonstrate that they value learning from failure. A company that shares their incident response processes likely has a strong operational culture.

Even companies without engineering blogs will leave traces. Product release patterns tell you about their development pace. Technology choices show their priorities: newer frameworks suggest a focus on innovation, whereas relying on proven technologies indicates they prefer stability.

Look for Patterns in Discussions

Glassdoor, Blind, and Reddit contain gold buried amongst rubble. Ignore the rubble (e.g., individual rants). Instead, look for patterns across multiple posts. If five different people mention “lots of process” or “no work-life balance” or “amazing learning culture,” that’s a pattern you will want to know about.

Pay attention to what people complain about and what they praise. Complaints about “too many meetings” may suggest the company has a collaborative, consensus-driven culture, or, alternatively, that productivity within the company is inhibited by an excessive number of meetings. Praise for “autonomy” indicates they trust their people to make decisions without checking in. Both types of comments reveal what behaviors the companies will reward.

Talk to Current Employees

If you know someone at the company, ask them directly what behaviors get rewarded and, conversely, what behaviors will cause people to struggle. Skip surface-level queries about culture, and ask specific questions:

• “When someone gets promoted here, what do they do to earn it?”

• “What behaviors get negative feedback?”

• “How does the team make decisions when there’s disagreement?”

• “What surprised you most about working here?”

Current employees will tell you truths the company website never would. Perhaps they’ll tell you that at their company, “customer obsession” really means checking usage data before writing code, or that “ownership” means being available to resolve production issues at two o’clock in the morning.

What You’re Really Looking For

All this research serves one purpose: understanding what stories will resonate at your interview. Think of it as finding the real intersection between your experience and what they care about.

If research reveals they prize speed over perfection, then emphasize stories that tell how you shipped quickly and iterated. If they value technical depth, highlight examples of diving deep to understand root causes. If they care about collaboration, make sure your story focuses on cross-team work rather than solo accomplishments.

The research will also help you decide whether this company is the right place for you. If everything you learn suggests they value the kinds of behaviors you don’t naturally demonstrate or don’t want to develop, then perhaps you don’t need to pursue that particular role.

Putting It All Together

Companies aren’t just evaluating whether you can do the job. They’re also assessing whether you’ll thrive in their specific environment and at what level you’ll be most effective. These two dimensions determine not just whether you will get an offer, but also whether that offer will position you for success.

Understanding fit helps you know which of your experiences will connect most with what the company values. This small company needs someone who ships fast and figures things out alone. That enterprise needs someone who navigates processes and builds consensus. Neither is inherently better than the other. They’re simply different environments that reward different approaches.

Understanding levels helps you position your stories appropriately. The same project can demonstrate entry-level execution, mid-level ownership, or senior-level leadership depending on your actual contribution and how you frame it. Get this wrong and you will either get rejected for overreaching or down-leveled for not properly communicating your capabilities.

The payoff is immediate. You’ll pick better stories, focus on the right details, and make it easier for interviewers to see what you can do. You’ll make better decisions about which roles actually match who you are and what you want to do. The goal isn’t to get any offer. The goal is to get the right offer at the right level at the right company to ensure your success.

Takeaways

Gergely, again. Thanks to Steve for both sharing his learnings, as an interviewer, and for sharing nearly a full chapter from his whole book. The book goes a lot deeper than the above sample chapter. A few of the ones I found helpful:

• High-signal storytelling (Chapter 3): a framework for explaining your work in a way that “sticks” with the interviewer

• 9 competencies with many examples and stories throughout the book: ones like “delivery” (Chapter 6), “earning trust and dealing with conflict” (Chapter 8) and “Strategic leadership and thinking big (Chapter 13)”

• Examples of what interviewers typically see as key signals, yellow flags and red flags

If you would like to have a fresh resource to prepare for behavioural interviews at tech companies, the full book offers far more explanations, tactics and exercises to do so:

Get the full book on Amazon

Steve also writes a newsletter titled A Life Engineered: you can sign up to it here.

It’s helpful to understand how and why companies hire, and what they look for. To us engineers, hiring processes often look illogical from the outside. We’ll ask things like:

• “Why does the interview process not resemble day-to-day work?”

• “I already have open source code I wrote: why does the company need to do a coding interview to confirm what is clear: that I need to code?”

• “Why did I get a rejection, even though I did well on all of the interviews?”

It feels to me that there are similarities between hiring and dating: both parties show up with goals and expectations in their head, which are often not communicated. Sometimes there’s a match; sometimes there is not. This phase of a relationship is often about “selling:” as a candidate on the job market, it’s about selling yourself, and convincing the company that you would be a fit for what they are looking for.

Doing your research on the company is underrated, and not all that many candidates do so, in my observation. When I was a hiring manager at Uber, roughly half of the people who got on the call with me did not do any research about the company, and perhaps 1 out of 10 candidates did any research on the team they interviewed for – when we had public blog posts about our work, on the company blog! So those showing up prepared helped them stand out in the “motivation” dimension, from the get go.

It all starts with being able to pass the “technical” interviews – but it’s a mistake to sleep on the “behavioural parts.” To state the obvious: candidates who do not do well on the technical interview rounds will not get offers. But I’ve personally had to say to several candidates who did great on the technical side of things, but turned out to be misaligned with what we were looking for, as confirmed on the behavioral rounds.

And I do believe you can uplevel in doing better on these behavioral rounds: starting with researching what the company’s culture is like, practicing how to present yourself better, and putting yourself in the shoes of the interviewers, understanding what they are looking for.

I know plenty of software engineers who refuse to do any preparation for interviews, staying “if the company doesn’t want me as I am, they don’t deserve me anyway.” This is a valid strategy, and can work for highly in-demand professionals, the same way as showing up to a first date in sweatpants and slippers can still work out for highly attractive and desirable people. For the rest of us not as incredibly in-demand for a position we’re applying for: it’s probably worth putting in additional effort, in hopes for better outcomes during interviews.

For today’s issue, we’ve dug into your 900+ responses to look for trends in AI tool usage among software engineers and engineering leaders. This article surfaces insights that are less about specific tools, and more about the effect these tools have on tech professionals. We cover:

1. Costs. Unsurprisingly, companies pay for most tool usage, and those responsible for budgets are increasingly nervous that AI-related costs are headed only one way: up.

2. Usage limits. Around 30% of respondents say they have hit limits. Switching tools, upgrading plans, or moving over to API pricing are common responses.

3. Impact on “Builders.” Folks who make larger code changes and do “quality-of-life” work are builders, and they’re also dealing with more AI slop. Some also grapple with a loss of professional identity.

4. AI tools speed up “Shippers.” Engineers who focus more on getting things done are the most positive about AI tools. But they also add tech debt faster and might build the wrong things.

5. “Coasters:” learning faster while generating AI slop. Less adept engineers can uplevel faster with AI, but they generate a lot of “AI slop” while doing so, which frustrates builders.

6. Changing software engineer & engineering manager (EM) roles. Engineers have to orchestrate and context switch more often, while engineering managers can be more hands on. It’s interesting to see the engineer and manager roles becoming more similar.

7. Other impacts on the craft. We’re going from “how” to build to “what” to build, solo devs are seeing improved results, workloads are increasing with AI tools, and more.

See also two other articles in this mini-series:

• AI tooling for software engineers in 2026: a detailed summary of survey responses, covering the most-used AI tools, trends, AI agent usage, company size and usage, and tools engineers love.

• The impact of AI on software engineers in 2026: key trends. Part 2. Tradeoffs of AI tooling, why adopting AI at company-level is hard, what’s changed in two years, and more.

Full subscribers also have access to a more detailed, 65-page report on our findings.

1. Costs

Concern about the cost of AI tools is a trend throughout the survey, with around 15% of respondents mentioning it in some way.

Tech companies foot the bill for the majority of spending on AI tools. More respondents say their employers pay for AI coding tools than those who say they pay themselves, and predictably, employers fund more expensive packages than what individuals buy personally.

Companies commonly pay for “max” plans with the likes of Claude Code, Cursor, and Codex (around $100-200/month per engineer), although some companies’ budgets only stretch to $20/month per engineer – around the price point of GitHub Copilot, and the cheapest Claude or ChatGPT subscriptions.

The most-mentioned AI tool spending patterns:

• When companies pay: ~$200/month plans. Many have enterprise subscriptions, sometimes with subsidies and vendor lock-in. Some companies allow usage-based coverage on top of monthly plans.

• When personally paying for tools: ~$20/month or free tiers. This can stack up across different tools. Around 5% of respondents have separate work and personal subscriptions, and free tier usage is widespread for personal use.

For now, companies seem to be in the experimentation phase with AI tools, and several respondents say that they believe their companies have unsustainable AI-tooling budgets. This is likely because businesses are figuring out the best way of leveraging the tools, and the message to engineers at such places is to not worry about price and usage while that unfolds. A CTO at a small, US-based company shares:

“Right now, we’re not sweating the costs because we’re trying to evolve best practices for the tools, but that has resulted in some devs really blowing through budget, so we may start instituting caps on spending.”

Breaking the budget

At small and mid-sized companies, leadership teams seem more comfortable about going over budget, than engineers running out of budget. There are more accounts from C-level folks and founders about racking up large bills than there are from engineers. A CPTO (Chief Product and Technology Officer) at a mid-sized company:

“I ran up several monthly bills of $600 with Cursor. We have the dev team subscribed to ~$100/month plans. We’re now in the process of moving the rest of the team to Claude Code, as we can get more resources for around $100/month in cost.”

Top spenders can be allocated higher budgets. A number of tech businesses have separate, larger budgets for their heaviest AI users. A senior C++ engineer working in the video game industry says:

“I’ve become my team’s AI champion. In theory, my limits are higher than normal, but I keep myself limited to what others can use, so I can show them useful things they can do.”

UK and EU companies worry more about budgets than US-based ones. Most responses which mention finance teams pushing back against spending even $30-50/month per engineer on AI tools are based in the UK and EU. One amusing example is a 10-person, seed-stage startup where the CEO questioned why they were paying as much as £25/month per engineer for one of the cheapest AI tools around.

In general, it feels like European companies want to see clear value-add in order to justify an increase in tooling spend, whereas US companies are more comfortable with investing first and measuring impact later. At present, the impact of these tools is hard to quantify.

A niche approach is that of AI teams educating devs to use cheaper models. Some European companies go as far as offering education to new joiners on using cheaper models. From an AI Enablement Lead at a 1,000+ person, digital transformation company:

“Within our organisation, we’ve had incidents where our Claude users have overshot their limits. We’re now attempting to educate devs in knowing the difference between different models (knowing when to use Claude Sonnet versus Claude Opus).”

Cost trajectory worries

The cost trajectory of AI tools is generally considered unsustainable in our survey. Devs using the tools heavily tend to hit usage limits, and their employers then have to pay more. At places with API-based pricing, usage is increasing. Those in leadership positions who are responsible for budgets are generally concerned about the direction of costs.

Subsidies are keeping things at bay – for now. A common enough pattern in our survey is of heavily-subsidized enterprise plans that come with vendor lock-in. Several responses raise concerns about what will happen when the subsidies run dry. Experienced engineering leaders recall that cloud providers also played the same game of subsidizing for a few years, then raising prices when a customer was fully “locked in.”

The AI hype cycle is dampening awkward conversations about budgets at some places. A principal engineer at a Fintech tells us:

“The AI hype has created a special, generous budget for AI tools, and there’s no effective budget – yet!”

But some finance teams are getting grumpy. A CTO at a sports-tech company says:

“It’s hard to keep our CFO supportive about investing in these tools because the productivity benefits have proven difficult to conclusively prove. The point that resonated the most was the loss of value when people hit daily limits: having to stop work immediately! Surprisingly, our CFO is still pushing back, despite having experience of getting a lot of value through their own AI usage with their spreadsheets.”

Most survey respondents think the price of AI tools will have to rise. If that happens, it would cause problems at several companies – particularly those in Europe:

“I cannot see how the spend on AI tools is fiscally sustainable in its current form; Max 100 with Claude Code is $100 a month. A single small task powered by Kimi K2.5 using OpenCode is $5, mostly in input cost. If we assume that the third party inference providers are doing so at a sustainable price, the much more expensive Opus model cannot be sustainable, never mind profitable at these plan costs.” — Founder at a seed-stage company, Europe.

“From the economic perspective, at some point, these companies will need more funding or profit, I’m curious how much it costs them to have a proper agent, and still become profitable. It feels slow when you run out of credits when working on repetitive tasks.” — Principal Software Engineer at a seed-stage company, Europe

2. Usage limits

Another major trend in our survey results is the topic of usage limits:

• Hitting limits: ~30% of respondents. Running out of tokens or hitting reset limits is frustrating and disruptive, especially when you’re working on a task or are in a flow state. The majority of respondents who complain about hitting limits are on cheaper plans (typically $20/month.) But this issue is also mentioned about higher subscription levels.

• Under the limit: ~20% of respondents. Avoiding usage caps generally correlates with being on more expensive plans with higher limits, or in roles with enough non-coding work for it not to matter, or when devs do enough work “manually” for AI usage to not be an issue.

Why users of AI tools hit limits

Common reasons cited in the survey:

Being a new AI user or a power user. These are two distinct groups, but an engineering manager at a mid-sized company in Canada says that each one similarly blows through token limits for different reasons:

“We’re mindful of trying to manage costs by setting AI spend limits across the org. We have two subsets of users at odds with each other:

1. Individuals who are still learning and blow through their credits at an inordinate rate, forcing us to keep limits low.

2. Power users who hit the limit through regular use and apply pressure to raise the limit.

It’s a tough balance.”

Using Opus for all work. A few engineers mention being careful about how they use Opus because it previously ate up their token budgets. Here’s a software engineer at a mid-sized company in Europe:

“I made the mistake of using Opus in the past and burning through budgets quickly. Now, my routine is to start in ‘plan’ mode with Opus. I then paste the acceptance criteria and description of the issue and let the plan mode figure it out. I then switch to Composer or Sonnet and have the agent take over from there.”

Mistakes that eat up tokens are easy to make. These include starting on work or a problem from the wrong end, using AI directly for a task rather than opting for a simple script, trying some new tool or technique that ends up consuming tokens (OpenClaw and Ralph Loops are cited), and others.

What happens when the limit is hit?

Hitting the limit with an AI tool is inconvenient and happens to many developers, who take a variety of next steps:

Switch the model or tool. Around a quarter of respondents who hit limits mentioned switching. From a software engineer at working at Atlassian:

“In my company, for Cursor and Windsurf we have monthly limits. Our internal coding tool (called codelassian) also has daily prompt and hourly token limits. When I hit a limit in one tool, I switch to the other.”

Upgrade to a pricier plan. When it’s an option, this is a no-brainer at most places, especially as the alternative would be devs twiddling their thumbs waiting for the limit to be reset. A senior engineering manager at a mid-sized company says:

“In my team, we are regularly hitting session limits with Claude. We upgraded some teammates to the Max 20x plan – and on this plan we have not been hitting limits, so far.”

Adopt API-based pricing. This is the easiest way to keep working without abandoning a task you’re knees-deep in. A senior engineer at a large company says:

“The company provides both the Claude and Copilot corporate offerings. When the limits are reached, I tend to use API keys that my teammates give me.”

3. Impact on “builders”

We identified three different types of professional in the survey:

• Builders: those who care about quality, good architecture, following good coding practices, and who talk about the craft of software engineering, etc.

• Shippers: those who primarily focus on outcomes for a product, features, testing, and experimenting with users. A fair number of leaders, managers, and engineers who were more hands-off with coding before AI tools are in this category, as are product engineers.

• Coasters: engineers who are not considered particularly good or great engineers, but they get the work done. They often do this without much taste or concern for quality, and seem to be mostly coasting along and doing what they’re told.

The overall consensus in our survey results is that AI will amplify and multiply tendencies and patterns that existed before, and the impact of the tools varies accordingly among users. Let’s start with the impact we’ve observed upon builders in the responses:

Builders say they get value from AI tools in the following areas:

Larger code changes. Builders generally find AI helpful for work like:

• Refactoring

• Migrations

• Improving test coverage

• Carrying out large codebase changes

All these are changes that are laborious, but not very challenging technically. They also require experience in knowing what you want to do and how to do it.

Accomplishing “quality of life” tasks. Builders mention that with AI tools, they get to fix and improve things like nagging bugs that otherwise wouldn’t be “worth it” in time invested, but the barrier to entry is lower with AI.

A good example of this is in last week’s podcast with David Heinemeier Hansson (DHH), the creator of Ruby on Rails, in which he revealed how one of their engineers optimized P1 – the fastest 1% of web requests:

“One of our most agent accelerated people asked: “What about P1? What about the floor? Can we fix the floor?” He found that the floor [of request speed] was 4 milliseconds.

Well, 4 milliseconds can add up if you have a bunch of fast requests. So, he just said: “We’re going to optimize P1. The fastest 1% of requests, we’re going to make them even faster.” He took it from 4 milliseconds to less than half a millisecond. He did this P1 project over a couple of days, like a side project.

He had an intuition that there was something here. He let agents run with it. The work ended up being 12 pull requests, and about 2,500 lines of code changed.

This is exactly why the explosion of the pie suddenly lets us look at problems we would never have contemplated looking at before.”

Typing is no longer a bottleneck. Some builders report falling even more in love with coding with the help of AI and agents, since physically typing out code is no longer a bottleneck for them. They enjoy being able to prompt. From one “builder”:

“For someone who loves to build – but also values code quality, performance, reliability, and security – I ship a lot more quality code faster, if for no other reason than because the AI can read and write 100x faster than me. I get to stay at the conceptual level of shipping a product, and I can dive into debugging with the agent as needed. But if the agent has a good handle on the situation I can give it as much of the tedious parts as I wish.” – Staff Engineer, at a large tech company, US

The negative sides of AI tools, as experienced by builders:

• More AI slop. Builders seem to be the most overwhelmed and derailed by reviewing a lot more AI-generated code. They can get frustrated with low-quality code shipped by colleagues which could be categorized as “AI slop.”

• More debugging. AI-generated code introduces bugs and issues, and builders tend to spend the most time debugging and fixing those issues.

• Identity loss. Some builder-types report a sense of identity loss and even some grief. Much of this relates to no longer doing hands-on coding because they cannot justify it, since AI agents generate pretty decent code faster than someone can type it.

4. AI tools speed up “Shippers”

The “shipper” archetype thrives when they get things to production quickly. This group is by far the most enthusiastic about AI tools in survey responses. They are also the ones who praise – or hype up – the tools because of their personal experiences of shipping much faster with them.

The biggest upsides mentioned by shippers:

Read more

Martin and Kent go back decades, and Martin jokes that his career is “mostly about writing down Kent Beck’s ideas.” They first collaborated in the 1990s, and each has published influential books – ‘Extreme Programming Explained’ and ‘Test-Driven Development’ by Kent, and ‘Refactoring’ and ‘Patterns of Enterprise Application Architecture’ by Martin.

At the Pragmatic Summit, they each shared a wealth of hard-earned learnings and decades-worth of perspective, along with a healthy dose of skepticism. Needless to say, the conversation did not disappoint, and this article summarizes what we discussed in their own words. You can also check out the full recording.

We cover:

1. Technology shifts similar to AI. The arrival of the microprocessor, introduction of object-oriented languages, the Internet, and agile software development principles were all major changes – but one big difference was that it took time for these technologies to be adopted. Not so with AI.

2. Agile and AI similarities. With Agile, company incentives were often misaligned, “snake oil” vendors were everywhere, and a “mid pack” of developers who resisted the change saw their career prospects hit. These trends look likely to repeat with AI.

3. What’s happening inside companies. There’s some confusion – and even panic – at large companies, while AI tools don’t work nearly as well on large and complex codebases as on greenfield projects. Also, a “re-soloing” of software development is inbound.

4. Avoiding burnout with AI agents. Set and maintain boundaries, and pay attention. Martin suggests to catch when you start producing “negative value”: that’s when to take a break.

5. Unhealthy performance metrics. Companies are starting to measure things like frequency of pull requests – when they should be looking to quantify outcomes and results.

6. Lower quality on purpose? It seems every business is optimizing for speed with AI, but quality can get dropped. Also: building features is more obvious with AI, than investing in “futures.”

7. Test-Driven Development (TDD): tests no longer optional? Kent pioneered TDD, and today it’s more relevant than ever for working with AI.

8. Thriving in an AI-native industry. Focus on working with agents to express your craft, try to get more enjoyment in understanding your domain, and take on more ambitious work.

Before we start, a programming note: this week, there will be no The Pulse on Thursday — I’ll be attending AI Engineer Europe in London on Thursday and Friday, including doing a fireside chat, and hosting one with Linear CTO Tuomas Artman.

1. Technology shifts similar to AI

Do you recall a tech change as similarly promising and unpredictable as AI?

Martin: “Nothing has hit with the magnitude of AI. This is a whole size different from anything we’ve faced before. On a smaller scale, we were very much involved in the growth of object oriented languages, which scared a lot of people. It didn’t scare us so much because we were part of it.

Looking back, the internet had a huge impact on us all, and of course, Agile software development, too. Agile had a very big impact on a lot of organizations: you could tell by how hard they resisted it. We had to persuade people of the importance of these technological changes; yes, even the internet! It may sound surprising but there were people who didn’t think it was important.

The thing about AI is that today there is no argument about how important it is.”

Kent: “The other analogy I have is the introduction of the microprocessor. Before that, computers were big boxes; you couldn’t move them around. If you wanted another computer, you’d mortgage your house for it. Having a computer was a big deal.

I was a kid in Silicon Valley with my dad as a programmer when the Intel 4004 hit the market [in 1971]. We went: “Wait a minute, that chip is a computer? Oh my goodness!” The possibilities of computing suddenly expanded thanks to it. If you could figure out how to write software on this chip and figure out how to design hardware around this thing, you could suddenly do things you hadn’t even imagined.

And so I think part of AI is this expansion of imagination. I’m writing projects that are ridiculously ambitious: I’m working on a persistent Smalltalk. I’m writing library-quality code for Rust.”

Balancing skepticism and curiosity

What was the feeling in the industry during those revolutions, and the differences between professionals who thrived back then and those who didn’t?

Martin: “There was a mix of people chasing the hype and those saying, “this new thing is nothing special.” I think you’ve always got to have that balance of skepticism and curiosity, and to be selective about it. I mean, I have been completely skeptical about some big changes: Blockchain was one I was extremely skeptical about.

My skepticism is well-rooted because I’ve seen so much “snake oil” over the years. In fact, my skepticism has to be absolute and total, which means I have to be skeptical about my skepticism! To be that skeptical also requires curiosity: you’ve got to be curious enough to say “how do I probe in order to detect signs of something useful?”

You also need to be aware that your early interactions may not actually be a true signal. When I started playing around with AI, it was with GitHub Copilot a year and a half ago. I was pretty unimpressed; it would give you something wonderful, but most of the time it gave you such garbage that you would just delete it right away. If that had been my only impression of AI, I would’ve immediately flipped the “bozo bit” on it, like I did with blockchain.”

Kent: “Here’s the thing, the capabilities of AI can change week to week. I’ll try something with Gemini one week and it fails miserably. Then Claude Code works pretty well, and then it doesn’t. And then I try Gemini for the same thing and it works, when it hadn’t worked last week!

People want an answer, but the answer’s always changing. In this environment, you can’t possibly have the answer. That’s the bad news, but the good news is that nobody else has the answer either. So, you’re just as smart as everybody else because we’re all equally ignorant.”

2. Agile and AI similarities

In 2001, the ‘Agile Manifesto’ came out, of which you were both co-authors. I think many companies are expecting the same thing with AI as Agile promised: better, faster, cheaper software. But how did Agile adoption really play out?

Kent: “It turns out people don’t want faster, cheaper, better! Inside some companies, the incentives are misaligned with actually achieving that. And so as geeks trying to achieve these improvements and saying: “it’s 40% better, 12% cheaper and less fattening,” people will punish you if that doesn’t align with their incentives inside organizations.

In the ideal organization, everybody would care about the same things, but that’s just not the way it works! So, if AI is coming along to promise the same things, we’re going to see the same reaction as before.”

Martin: “An obvious difference is the sheer magnitude and speed there is with AI. Also, I think there will be a big difference between people who use it well and people who use it badly. The trick is figuring out how to use it well and putting the effort in to learn. There will be a big distinction between those two groups.

But I suspect there will still be some similarities with Agile. The core notions behind Agile and extreme programming are solid and good, but a huge snake-oil industry appeared around it – the “Agile industrial complex”, as I refer to it. This is also happening with AI right now, and it’s often hard to see the difference between snake oil and the real stuff.”

AI as an amplifier

Kent: “AI is an amplifier. If you’re young and learning quickly, AI can amplify your learning. I personally think this is the golden age of the junior programmer. I get people coming to me all the time saying things like “my son started his second year in CS and wants to go into something more commercial like art history.” And I’d say, “this is like if you’re a carpenter and they just introduced the circular saw and you think, ‘oh, well, carpentry is over. Anybody can build a house now.’ Well, no! Now, you have more powerful tools. You have less of the crummy work to do.

I think that young people are going to learn faster, and experienced folks who are working effectively are going to work quicker and more effectively.”

Developers stuck in the middle

Kent: My concern is that there’s a “middle” of people who got into programming as a way to make money. If we look back at the Dotcom crash, there was a “mid pack” of such people who ended up going into real estate, more or less. But today, I don’t know where that “middle” will go, and it’s also much bigger now than 25 years ago.”

Martin: “But that middle has also been “flushed out” to some degree by retrenchment in the software industry at the end of the zero interest rate period. So, that’s an interesting difference because we’ve had these things occurring at once: the AI boom, and the economic headwinds of the past 2-3 years.

This is an interesting mix that wasn’t present in the ‘90s with the Dotcom Boom. Back then, it was pretty much all a solid boom.”

Return of “let’s get rid of programmers!”

Kent: “Another interesting confluence of factors is the periodic, “we can get rid of all the programmers, woo-hoo” trend, which started with Cobol in the 1970s. With Cobol, business analysts were supposedly going to be able to write the programs, and the logic was that we wouldn’t need programmers anymore. That comes back repeatedly.

Agile, however, was definitely not a “let’s get rid of programmers” trend. With Agile, we wanted programmers to be more effective in their jobs. And since we started it, and were programmers, we were able to push that agenda pretty effectively.

However, today the “get rid of programmers” trend is repeating. As programmers, it behooves us to think about why they keep wanting to get rid of us. Some of that’s about us as programmers, and some of it not. Still, we should think about why people periodically want to axe us. In the end, this trend amps up the fear factor that everybody’s experiencing.”

“Re-soloing” of programming

Kent: “A big trend is the “re-soloing” [reduced in-person collaboration] of programming.

A big part of extreme programming (XP) was creating a safe social environment for basically antisocial people. On an XP team, people are talking to each other for hours a day, and are happy to do so because it’s set up to be a positive experience.

Now, I see programmers saying, “I’ve got six agents, so really I’m managing a team.” No, you’re not: you’re using six tools at once, which is fine, but it’s very different from having a conversation with somebody who sees things slightly differently, or has a different energy level from you on the day.

We used to have programmers in individual offices with doors, and you’d shut the door and slide the pizza underneath. That was easy to manage, but then along came this messy, social, complicated, chaotic process of software development, which just happened to produce really good results.

But now, instead of 50 people on my team, I can have five and they don’t have to talk to each other, and each can have 10 agents. Is that the same? No, it’s not.”

More effective two-pizza teams & the future of pairing

Martin: “Are we seeing two-pizza teams [of 5-10 people] becoming one-pizza teams because agents don’t eat pizza, or do we see two-pizza teams staying and becoming much more effective and capable? My bet is on more effective two-pizza teams.

We’re beginning to see some interesting feedback in terms of pair programming. With pair programming, is it one human and the genie (AI) programming, or is it two humans and one genie? If it’s two of us, perhaps we can control the genie a bit better, and we also have interaction.

I’ll be very interested in reports of people trying to control genies in pairs, possibly even beyond pairs. There’s also the whole ‘mob programming’ thing, and how that will go with genies. I don’t necessarily think that one person and many genies is the right answer.”

Kent: “My experience of pairing with two humans, plus one or more genies, has been very positive. And the fact the AI is slow is really nice. Every time models come out and are faster, I’m like, “Oh, there’s less time to talk.” When the AI goes away for three minutes, we can talk about our philosophy of naming, or how we express conditionals, or about what we should be doing next. But if it pops back in 15 seconds, you don’t have time for that conversation.”

4. Avoiding burnout with AI agents

Do you find yourself getting close to burnout, especially when spinning up multiple threads? Do you have strategies for managing the mental impact?

Read more

And now, inference engineering is becoming more widespread, too, as open LLM models grow more capable. This is because with closed models, inference engineering is done only by the AI engineers who build the model, whose number might add up to a few thousand globally. In contrast, with the open models which tech companies are adopting, it’s possible to tweak them to perform better at inference. For example, Cursor built its new Composer 2.0 model on top of the open Kimi 2.5 model, and successfully used plenty of inference engineering approaches to make it even faster.

So, based on this industry-wide prevalence and the related need for superior technical performance, it’s worth understanding a bit about what inference engineering actually is, and some interesting approaches worth knowing about, as a software engineer.

For some answers, I turned to Philip Kiely, a software engineer who has been working for four years at the inference startup, Baseten. With his hard-earned experience, Philip has written an excellent, in-depth book about precisely this topic, “Inference Engineering.”

In today’s issue, we cover:

1. Setting the stage: why is inference so important? More capable, widespread, open models are driving demand for inference engineering.

2. What is inference? As the phase that comes after training a model, the inference layer introduces new engineering challenges like batching, caching, and quantization.

3. When is inference engineering needed? Investing in this area is typically worth it when your product and usage scales up, and there are product requirements which a current, off-the-shelf solution lacks.

4. What hardware does inference use? Datacenter GPUs are the most common, while on-premises, air-gapped GPUs are also employed.

5. What software does inference use? Commonly-used software includes NVIDIA’s CUDA and Dynamo, as well as hardware-agnostic projects like PyTorch, vLLM, and others, which are growing in popularity.

6. What infrastructure does inference need? Autoscaling is a baseline requirement. Kubernetes is a popular choice for autoscaling inside a cluster, while multi-cloud inference might be necessary for high-scale use cases.

7. Five approaches to make inference faster. Quantization (reducing the numerical precision of a model’s weights), speculative decoding (taking advantage of spare compute to generate “draft tokens”), caching, parallelism (tensor parallelism and expert parallelism) and disaggregation (separating the prefill and decode phases to run on separate workers, not the same GPU).

This deepdive uses a few abbreviations and concepts that are everyday lingo for inference engineers, but maybe are not for those less versed in the domain:

• CUDA: Compute Unified Device Architecture. NVIDIA’s proprietary API to program NVIDIA GPUs for high-performance computing, including LLM-related use cases.

• TTFT: time to first token. Think of this as the “time to process the prompt.” This metric determines the perceived responsiveness of models and GenAI systems.

• TPS: tokens per second. Akin to a model’s “typing speed.”

• ITL: intertoken latency. The time between generating one token and the next.

• KV cache: key-value cache. The cached results of the attention algorithm, reused between requests to speed up inference. We cover more on KV cache in the Scaling ChatGPT deepdive.

• Prefill / decode: the two phases of inference. Prefill is when the model takes the full input and processes tokens, outputting the KV cache. Decode is the phase in which the model generates one token at a time.

• MoE: Mixture of Experts. An architecture that enables models to be pretrained with far less compute. More details on this approach.

Below is an introduction to inference adapted from Philip’s book, “Inference Engineering,” which is free to download as an e-book. Physical copies are currently sold out, but Philip is printing more as fast as possible.

My usual disclaimer: as with all my recommendations, I was not paid to mention this book, and no links in this article are affiliates. See my ethics statement for more.

With that, it’s over to Philip:

1. Setting the stage: why is inference so important?

Inference is the most valuable category in the AI industry, but inference engineering, on the other hand, is still in its infancy. In their work, inference engineers work across the stack from CUDA to Kubernetes in pursuit of faster, less expensive, and more reliable serving of generative AI models in production.

When ChatGPT launched in late 2022, there were perhaps a few hundred inference engineers in the world, and they didn’t call themselves that. These specialists mostly worked at frontier labs like OpenAI, Midjourney, and Anthropic, or at big tech companies like Google and NVIDIA.

Back then, it looked like this might be the way of the AI industry: that training generative AI models would be so hard and expensive that only a handful of companies would develop closed models and thereby require inference engineering for production serving. In that alternate future, the rest of the world would be mere consumers of AI via APIs, renting intelligence a token at a time.

Three years later, it turns out that training generative AI models is indeed both hard and expensive – but it’s not so hard and expensive to be limited to a handful of players. Instead, a proliferation of open models – more than two million and counting on Hugging Face (the “GitHub for AI”) – means that today every engineer can now deploy their own intelligence to power AI products.

Research labs around the world, from OpenAI and NVIDIA Nemotron in America, to Mistral AI and Black Forest Labs in Europe, to Alibaba Qwen, DeepSeek AI, Z AI, and Moonshot AI in China, regularly release open models of all modalities.

Despite closed models getting smarter and cheaper, the movement into open models is accelerating, which differ by the availability of their weights:

• Closed model: A proprietary model whose weights are unavailable to the public, like GPT-5 and Claude Sonnet.

• Open model: A model whose weights are publicly available, like Llama or DeepSeek, and which is usually released under the MIT license, or a similar permissive license (some models restrict commercial use, so always double-check license terms).

Before December 2024, there was a meaningful gap in intelligence between closed and open models, but when DeepSeek V3 and R1 were released, that gap disappeared. Note from Gergely: we previously covered how DeepSeek’s release rocked the AI industry.

Today, new closed models are matched by open models within months if not weeks, and occasionally, open models like Kimi K2 Thinking even exceed closed models’ capabilities for brief periods.

Despite the fact that open models are constantly chasing closed models on benchmarks, they nonetheless change the equation for AI product builders. And as both types improve, closed and open models cross capability thresholds.

In 2022, it was impossible to build the kinds of AI-native products that define the industry today. But over time, closed models got smarter and new categories like customer service voice agents and AI-powered IDEs became possible. The early models were slow, expensive, and unreliable, but the capabilities were there and AI engineers began building companies around them.

As open models crossed the same capability thresholds, these folks began using them to replace closed models. Many also began fine-tuning open models to cross capability thresholds faster, and even exceed closed model quality in their specific product and domain.

Switching to open models means the opportunity to use inference engineering to make the models powering AI products better in new ways:

• Latency: Closed model APIs are built for throughput, but open models can be optimized for real-time applications.

• Availability: While APIs for GPT and Claude are stuck at two nines of uptime, it’s possible to achieve four nines or better with dedicated deployments of open models.

• Cost: Open models are often at least 80 percent less expensive at scale.

So, whereas three years ago it looked like inference engineering was a niche field, the fact is that today, every company aiming to build truly differentiated and competitive AI products needs an inference strategy.

AI-native startups like Cursor, Clay, Gamma, and Mercor are redefining hypergrowth by building products that rely on open and in-house models. Leading digital native companies like Notion and Superhuman succeed by deeply integrating AI capabilities into their category-defining products.

Elsewhere, a new generation of blended research and engineering teams – World Labs, Writer, Mirage, and dozens more – are building businesses by training and productizing their own foundation models.

Adoption is even strong in enterprise and regulated industries, which historically were slow to adopt new technologies. Companies like OpenEvidence, Abridge, and Ambience are making generative AI ubiquitous in healthcare, while at the world’s largest companies, AI initiatives are moving past the pilot stage into massive user adoption. Market-wide demand for inference means that everyone from developers to executives has the opportunity to learn inference engineering and use it to advance their career and business.

I’ve been incredibly fortunate to have a front-row seat in the fastest-moving market in history over the last four years at Baseten, where we power mission-critical inference for the best AI products, including every company listed in the previous paragraphs.

The good news is that you are early. There are still relatively few professionals working on inference, and newcomers can become experts quickly. Also, the potential and impact of inference is becoming ever clearer, but the domain is still in its infancy. That means there are enormous opportunities to solve novel, interesting, and deeply technical problems at all levels of the stack.

2. What is inference?

Inference is the second phase of a generative AI model’s lifecycle:

• Training: The process of learning model weights from data.

• Inference: Serving generative AI models in production.

During the past decade’s machine learning (ML) boom, hundreds of thousands of data scientists and ML engineers became familiar with the full lifecycle of training and inference for ML models.

Inference for classic ML models is relatively straightforward. In the early days of Baseten, we ran inference for models built with tools like XGBoost on lightweight CPUs with a simple software stack.

In contrast, inference for generative AI models is complex. You can’t simply take model weights, get some GPUs, and expect inference to be fast and reliable enough for large-scale production use. Doing inference well requires three layers:

• Runtime: Optimizing the performance of a single model on a single GPU-backed instance.

• Infrastructure: Scaling across clusters, regions, and clouds without creating silos, while maintaining excellent uptime.

• Tooling: Providing engineers working on inference with the right level of abstraction to balance control with productivity.

These three layers must work together to create a system that can handle mission-critical inference at scale.

The runtime layer is responsible for ensuring an individual model running on a GPU (or across several GPUs in a single instance) runs as performantly and efficiently as possible. This layer depends on a sophisticated software stack, from CUDA, to PyTorch, to inference engines like vLLM, SGLang, and TensorRT-LLM. Low-level optimization is important, with kernels like FlashAttention delivering significant performance gains.

The runtime layer relies on a number of model performance techniques that apply new research to the challenges of inference on generative AI models:

• Batching: Run incoming requests in parallel, weaving them together on a token-by-token basis to increase throughput.

• Caching: Reuse the KV cache – the cached results of the attention algorithm – between requests that share prefixes.

• Quantization: Lower the precision of select pieces of the model to access more compute and reduce memory burden.

• Speculation: Generate and validate draft tokens to produce more than one token per forward pass during decode.

• Parallelism: Efficiently leverage more than one GPU to accelerate large models without introducing new bottlenecks.

• Disaggregation: Separate the two phases of LLM inference, prefill and decode, onto independently scaling workers.

These model performance techniques are used for all modalities and not just LLMs, such as vision language models, embedding models, automatic speech recognition, speech synthesis, image generation, and video generation, which extend the capabilities of AI systems and require their own inference optimizations. But these runtime optimizations are not enough: no matter how performant a single instance of a model server is, it will eventually receive more traffic than it can handle. This is not a CUDA problem or a PyTorch problem, it’s a systems problem that needs to be solved at the infrastructure layer.

The nature of infrastructure problems changes at each level of scale. At first, the problems are around autoscaling: knowing when to add and remove replicas, and figuring out how to do so quickly.

Past a certain scale – generally a few hundred GPUs – infrastructure problems are defined by capacity. To get access to enough GPUs, inference engineers begin spreading workloads across multiple regions and cloud providers. This quickly leads to silos, where models in one cluster may be starved for resources while other clusters have unused capacity. The final level of scale in infrastructure is a global system that treats all available resources as a single unified pool of compute.

Thoughtful multi-cloud infrastructure also improves reliability, protecting against downtime in any individual region or cloud provider. And for global applications, running inference near to end users improves end-to-end latency.

Once these runtime and infrastructure capabilities are built, they need to be presented at the appropriate level of abstraction. Inference providers like Baseten and internal teams building inference need to consider what tooling and developer experience to provide as the critical third layer in a complete inference platform.

Of course, developer experience is subjective. For inference, one extreme is the black box: give a platform model weights, and get back an API. At the other extreme is providing only basic constructs for compute, network, disk, and so forth.

The right developer experience is somewhere in the middle, where inference engineers have enough control to run mission-critical inference confidently, and enough abstraction to work productively.

This article – which is an excerpt of Inference Engineering – presents an overview of the technologies and techniques that power inference across all three layers of runtime, infrastructure, and tooling.

3. When is inference engineering needed?

Inference engineering adds speed and scale to AI products by optimizing production serving of generative models. Optimization means identifying the best solution from a range of options.

Before optimizing model performance and building robust infrastructure, you need to know what “best” means for your product; many performance improvements come from making tradeoffs in latency, throughput, and quality. In practice, optimization is often about finding the right balance, rather than maximizing a single factor.

For example, NFL players are big, fast, and strong. But they’re not as big as sumo wrestlers, fast as Olympic sprinters, or strong as champion powerlifters. Their bodies and skills are optimized to fulfill the specific demands of their position over the course of a full season.

Similarly, your inference system must be optimized to fulfill the specific demands of your model, product, and traffic. The more constraints you can introduce, the better the outcomes that can be achieved.

You should know:

• Model requirements: Which model(s) do you need to run inference on?

• Application interface: How will inputs be delivered to the model, and how is the output expected to be formatted?

• Latency budget: How fast does your product need to respond to a user action, end-to-end?

• Unit economics: How much sense does it make to spend on a per-request, per-user, or per-month basis?

• Usage patterns: How many concurrent users are you serving, and is there any pattern to their usage (e.g., more activity during business hours)?

Early in building an AI product, the answers to these questions may not be clear. At this point, it’s often better to use off-the-shelf APIs whenever possible, rather than investing in dedicated inference. But as a product scales, the requirements become clear and inference engineering becomes a worthwhile pursuit.

4. What hardware does inference use?

Inference engineering relies on accelerators: powerful hardware designed to load terabytes of data and perform trillions of operations per second.

The most common type of accelerator for inference is the GPU, and the market leader in GPUs for inference is NVIDIA. My book focuses on inference engineering for NVIDIA GPUs in the datacenter, and also covers other vendors of datacenter accelerators and local inference.

Across vendors, there are three types of GPUs on the market:

• Datacenter GPUs: Racked servers with interconnected high-performance GPUs. Example: NVIDIA B200.

• Workstation GPUs: Individual desktop GPUs for professional workflows. Example: NVIDIA RTX Pro 6000.

• Personal computing GPUs: Individual desktop GPUs for everyday use. Example: NVIDIA GeForce RTX 5090.

Inference at scale uses datacenter GPUs mounted on racks: refrigerator-sized chassis with standardized power, networking, and cooling.

Datacenter GPUs like the NVIDIA B200 offer the highest individual performance, and more importantly, include high-bandwidth GPU-to-GPU interconnects, are installed in highly standardized configurations, and are available by the millions in datacenters worldwide.

I doubt there is a B200 GPU running under your desk, but if there is, please send me a picture! Instead, inference on datacenter GPUs runs in one of three modes:

• Cloud: GPUs are rented in someone else’s datacenter, usually hyperscalers like AWS and GCP, or neoclouds like CoreWeave and Nebius.

• On-premises: GPUs are purchased and installed in a datacenter that you control directly.

• Air-gapped: GPUs are installed on-prem and you need to physically access the GPUs to run inference.

Most inference engineers use cloud GPUs. Large enterprises and governments run on-prem and air-gapped deployments, but cloud-based GPUs offer the flexibility and access that fast-growing AI products need to scale.

Even with constraints, navigating the hardware landscape is complex. From variations between cloud providers, to NVIDIA’s personal naming conventions, there are many nuances in selecting the right accelerator.

5. What software does inference use?

NVIDIA’s market dominance in the inference space is in no small part due to the robust, mature software ecosystem around its hardware. Hardware iteration cycles are slow. Best-in-class hardware companies like Apple and NVIDIA release new architectures and generations at most annually, with two-year release cycles being more common. But software iteration is fast. Often, to run a newly released open model on day zero, you need to install a nightly build or other pre-release version of each software dependency just to get support for the new model.

Software’s fast iteration cycle and lower barrier to entry dramatically expands the landscape of inference engineering. There are countless companies building software at various levels of the inference stack, in contrast to hardware, which centers on NVIDIA and a few competitors.

For inference engineers, these are some of the key software players:

• NVIDIA: Invests heavily in its own sometimes-proprietary software ecosystem, from CUDA up to Dynamo.

• Hugging Face: Maintains a model registry for all open models plus transformers (models built on the transformer architecture), and diffusers (models built on the diffusion-based generative ones).

• The Linux Foundation: Maintains hardware-agnostic projects like PyTorch and vLLM.

• LMSYS Org: Develops essential tools for inference and evaluation, most notably SGLang.

There are thousands more companies, universities, and research institutions making essential open-source contributions to inference. Over time, technologies have been built at increasing levels of abstraction:

• CUDA: Direct communication to the GPU for explicit control over computations and memory.

• Deep learning frameworks: Abstractions over CUDA for training, exporting, and running neural networks in Python.

• Inference engines: Highly configurable PyTorch-backed inference for common architectures.

• NVIDIA Dynamo: Sits on top of inference engines to power large-scale deployments.

Most inference engineering today happens at the higher levels of abstraction, configuring and deploying inference engines and orchestrating inference across multiple GPUs. No matter which level of the stack you work at, it’s essential to have a strong mental model for the adjacent levels of abstraction to guide your work.

6. What infrastructure does inference need?

When you scale production traffic, your assumptions are rigorously tested. Everything from input and output sequence lengths, to traffic patterns, to what topic a user decides to chat about; they all impact your observed performance in production. And maintaining secure, robust infrastructure is an entirely different skillset from optimizing model inference on the GPU.

No matter how fast and efficiently a single instance can serve a model, the service will be overwhelmed if traffic gets high enough. It’s an infrastructure problem, not with PyTorch or CUDA, and it requires a different mindset and different technologies.

Scaling in production introduces new complexities about where and how to get GPUs, balance traffic across them, and prevent downtime. The goal of autoscaling is to ensure you always have enough resources to serve all incoming requests, while maintaining latency SLAs and without wasting money on idle GPUs.

Autoscaling systems use Kubernetes, an open-source container orchestration system, along with a cluster-level system for provisioning and deallocating compute. Kubernetes can run one or more replicas of a model container, each on its own instance. An instance includes the GPUs and other hardware resources that the container requires.

Unless your traffic is unusually consistent, there probably isn’t a specific number of replicas that perfectly matches your needs.

Autoscaling is the practice of dynamically adjusting the number of replicas allocated to a given model within a cluster. There are two ways to make autoscaling decisions:

• Utilization: Scale up or down based on GPU utilization signals like memory usage or compute usage.

• Traffic: Scale up and down based on the number of requests being processed in the system.

Utilization and traffic don’t always match. For example, in LLM prefill, a few requests with hundreds of thousands of uncached input tokens could cause much higher utilization than many small requests with high cache hit rates.

Traffic-based scaling decisions can be made proactively, while utilization is a lagging indicator. Use both in combination to keep system resources matched with demand.

When designing a traffic-based autoscaling system, you want to configure five factors:

• Min replicas: What is the minimum number of replicas that stay running, regardless of traffic?

• Max replicas: What is the maximum number of replicas you can allocate when traffic is high?

• Autoscaling window: How long is the sliding timeframe used to measure traffic and make autoscaling decisions?

• Scale down delay: For how long after a scale-down is suggested do you wait, in case there’s another traffic spike?

• Concurrency target: How many requests can each replica handle at once?

The exact configuration determines how well the autoscaling system achieves its goals of maintaining latency SLAs without wasting resources. For example, increasing the scale-down delay prevents premature scaledowns for spikey traffic, but could result in unnecessary spend after traffic has properly cooled down.

Autoscaling within a single cluster works up to a certain point, but high-volume deployments serving a global user base need thousands of GPUs distributed around the world.

It’s straightforward to build multi-cloud inference as a collection of siloed compute across different cloud providers. But in these setups, there’s no way to use inter-cloud compute fluidly, and moving workloads across clouds is a tedious, error-prone process.

True multi-cloud inference requires building a multi-region, multi-provider bin packing tool, which treats distinct pools of compute as fungible with each other. Like Kubernetes within a single cluster, multi-cloud capacity management must take a global view, enabling global scheduling.

Running true multi-cloud inference unlocks:

• Capacity: Pool capacity from multiple providers for greater, more flexible GPU access.

• Redundancy: Split inference across providers for resiliency against outages.

• Latency: Run inference close to end users to reduce network latency overhead.

• Compliance: Run inference in compliance with data sovereignty and other regulatory requirements.

Scaling from one cluster in one cloud to many clusters in many clouds requires a new coordination layer. A multi-cloud architecture contains:

• Control plane: Handles model deployment and global scaling decisions, receives real-time event streams.

• Workload planes: Handles direct inference traffic and in-cluster scaling decisions, reports utilization and demand.

This separation of responsibilities ensures that individual workload planes can serve traffic independently. If something happens to the control plane or any given workload plane, other workloads should be unaffected.

7. Five approaches to make inference faster

One of the coolest things about working in inference engineering is that, unlike many industries where new academic research takes years or decades to be adopted, techniques from new papers are live in production within months or even weeks.

But there is a gap to bridge between research and production, and some of the most visible inference engineering work of all comes from doing so.

Real-world traffic defies constraints. But with volume, you can adapt systems over time to match the changing nature of usage. Tuning the parameters of inference engines, speculation algorithms, and model servers isn’t a one-time task. Instead, either through iterative deployments or dynamic runtime adjustments, you can continuously improve the performance of an inference system.

Finding the right combination of techniques and configurations takes patient experimentation. I remember an internal hackathon during which one of Baseten’s inference engineers worked on an autocomplete model for code, and ended up trying 77 different configurations via a handwritten script before finding a non-obvious solution that doubled TPS (tokens per second) for a customer’s model.

Sometimes, techniques are symbiotic or incompatible, which makes inference optimization even more complex. For example, quantizing the KV cache alleviates a bottleneck in disaggregation, but increasing batch sizing reduces the compute available for speculation. An inference engineer’s challenge is always to create a balanced set of optimizations that delivers more than the sum of its parts.

Let’s look into the key categories of applied research for inference acceleration: quantization, speculation, caching, parallelism, and disaggregation.

Approach #1: Quantization

Quantization means reducing the numerical precision of a model’s weights. It improves latency (both TTFT [time to first token] and TPS, increases system throughput, and opens up headroom for other optimizations like disaggregation, speculation, and prefix caching to be even more effective. But when it goes wrong, quantization can materially reduce a model’s output quality.

Models are trained with weights, activations, and other components represented in a certain native number format. Usually, this is BF16 or FP16, although 8-bit and 4-bit native precisions are becoming more popular for training.

Post-training quantization works by changing those model weights and other values from their native number format, to a lower-precision format. Cutting precision in half improves performance in both phases of inference:

• Prefill: Compute-bound prefill now runs on lower-precision Tensor Cores with twice the FLOPS.

• Decode: Memory-bound decode now loads half as much data per value, effectively doubling memory bandwidth.

Working with quantized data introduces overheads, so it’s not linearly twice as fast to go from 16 to 8 bits. In practice, quantization down a single level of precision generally offers 30%-50% better performance for LLMs. The catch with quantization is that it runs the risk of reducing a model’s output quality, and has the potential to introduce precision errors throughout the calculations that power inference.

Precision errors compound over time. Consider what happens when you square and cube different precisions of Pi:

Most of the work in quantization is in preventing precision errors and minimizing their impact on the final model output.

Sixteen-bit, 8-bit, and 4-bit precisions are the primary formats for inference. Number formats contain:

• Precision: The number of bits used to express a single value in the format. For example, FP16 uses 16 bits.

• Type: Whether these bits are interpreted to represent an integer (non decimal) or a floating-point number (decimal).

• Scale factor: A multiplier used to map values from a low-precision format back to the higher-precision format.

Combined, these attributes determine the two factors behind how well a number format represents the values used in inference:

• Dynamic range: The difference between the lowest and highest value that can be represented in the format.

• Granularity: The number of parameters or other values that are quantized along a single scale factor.

Dynamic range is essential to low-precision inference without quality loss. Sixteen bits can represent 65,536 distinct values, while 8 bits can only represent 256 different values. The dynamic range is the distribution of these values – the difference between the smallest and largest available value.

Dynamic range explains why floating-point formats are better than integer formats for inference. Floating-point formats have three properties:

• Sign: A single bit that represents whether a number is positive or negative.

• Exponent: A set of bits that, taken together, represent an exponent factor.

• Mantissa: A set of bits that together represent the base value multiplied by two to the exponent.

An FP8 number in an E4M3 data format means it has a 4-bit exponent and a 3-bit mantissa, with the remaining bit for the sign. Integer formats only have sign and value bits.

The exponent in floating-point numbers gives it a higher dynamic range, meaning it can better express very large and very small numbers. This is important because outlier values are significant in inference, and floating-point number formats better represent outliers after quantization.

Within floating-point formats, there are multiple options at each precision, like FP4, MXFP4, and NVFP4. These formats differ in granularity, or in the number of values quantized by a single scale factor.

Quantization can be applied at three levels:

• Tensor level: Calculate a single scale factor for the entire QKV tensor.

• Channel level: Calculate a different scale factor for each feature vector within the tensor.

• Block level: Within each feature vector, divide the vector into blocks of N values and calculate a scale factor for each block.

More granular quantization has a lower chance of smoothing over outliers, which preserves quality. However, more granularity also introduces extra overhead for storing and applying scale factors.

The components of a model have varying sensitivities to quantization. Reducing the precision of more sensitive components runs a higher risk of quality degradation. From the least to most sensitive components:

1. Weights: the linear layers are least sensitive to quantization.

2. Activations: The intermediate output of activation functions are only somewhat sensitive to quantization. They are rarely quantized as they are such a tiny fraction of the model’s weights.

3. KV cache: The cached values from the attention calculation are moderately sensitive to quantization.

4. Attention: The attention layers of a model are highly sensitive to quantization, especially equations like softmax.

Within each component, you can get more selective about quantization.

Even in linear layers and activations – generally the least sensitive to quantization due to their size – early and late layers, like the input and output layers of the neural network, may be left in their original precision as these layers are more sensitive.

While quantizing weights and activations helps performance, KV cache quantization gives an additional boost to techniques like prefix caching and disaggregation. The KV cache is a valuable resource and quantizing it allows inference engines to store more of it in memory and read it more quickly.

However, the KV cache for each token is used by each subsequent token. This means precision errors introduced by quantization can compound from token to token. Compounding errors are exactly why attention layers are the riskiest to quantize: not only is attention very sensitive to dynamic range, but each attention calculation relies on the results of each previous attention calculation. Therefore, over a sequence of thousands of tokens, errors accumulate quickly.

All but the most aggressive quantization schemes run functions like softmax in their original precision.

A moderate approach to low-precision inference uses a format like FP8 with high dynamic range – if possible, a microscaling format like MXFP8 – to carefully quantize select linear layers, activations, and often KV cache values. Even with these high dynamic range formats, components of the attention layer are rarely quantized.

Approach #2: Speculative decoding

The decode phase of LLM inference is an autoregressive process in which tokens are generated one at a time. The bottleneck on decode is memory bandwidth, with compute sitting idle at low-to-moderate batch sizes as weights are read from memory.

Speculative decoding takes advantage of that spare compute to try and generate multiple tokens per forward pass through the target model. If an inference engine could generate two, three, or even more tokens for each round-trip of weights through memory, it would generate far more tokens per second. Note, speculative decoding only improves TPS / ITL (inter-token latency), not TTFT (time to first token.)

There are multiple algorithms for speculative decoding and they share a common mechanism:

1. The speculator generates one or more draft tokens.

2. The target model – or the underlying model that you’re trying to accelerate – performs validation on these tokens to check if they match what the model would generate.

3. The target model accepts any valid draft tokens and generates an additional token itself, completing the forward pass.

This generates N+1 tokens per forward pass, or iteration through the decode loop, where N is the number of accepted draft tokens.

Generating draft tokens is not free, it takes both compute and memory. However, it is much faster for a target model to validate a draft token than to generate an original token. If you imagine a sudoku puzzle, solving it is hard, but checking if the solution is correct is very easy. For the target model, generating a token is like solving a sudoku, while validating a draft token is like checking a finished sudoku.

The performance uplift from any speculative decoding strategy depends on three factors:

1. Draft token cost: Time taken to generate a draft token.

2. Draft sequence length: The number of draft tokens generated per forward pass.

3. Token acceptance rate: The percentage of draft tokens accepted by the target model.

Token acceptance rate is high early in the draft sequence, but draft tokens get less reliable deeper in the sequence.

Aim for short, high-percentage sequences because while generating and validating tokens is inexpensive relative to generating tokens in the original model, it still comes with meaningful overhead. Additionally, once a single draft token is rejected as wrong, all subsequent tokens in the sequence are also rejected.

Working with speculation is interesting because so many factors affect token acceptance rate. The big one is the temperature parameter – higher temperatures yield token distributions that are harder to predict, reducing the effectiveness of speculative decoding. But even factors as simple as subject matter can make a difference on acceptance rate if the draft model or additional decoder head used for speculation is better versed in, say, math than history.

Another limitation on speculative decoding is that it’s most useful at low batch sizes where there are spare compute cycles. At higher batch sizes, speculative decoding must be dynamically disabled as compute is too saturated to afford verification.

Each speculation algorithm navigates these tradeoffs differently, and careful implementation of the right algorithm for the situation can lead to major improvements in TPS.

Approach #3: Caching

During prefill, the inference engine builds a KV cache (a store of keys and values for each token) on the input sequence. It then updates the KV cache for each token during decode. As inference is autoregressive, the value for each new token depends on the value of every previous token in the sequence.

Every inference engine uses KV caching by default on a request-by-request basis. Without KV caching, LLM inference would be unbearably slow since each previous value in the entire sequence would need to be recalculated for each subsequent token.

However, engineers can get more utility from the KV cache by reusing it between requests rather than solely within each inference sequence.

Consider the following two prompts, each with four tokens on most tokenizers:

By default, the inference engine has to run prefill on all four tokens of each prompt. But the first tokens of each prompt – “Weather in” – form a shared prefix between the pair.

With prefix caching, you can reuse the KV cache from the first request to improve TTFT on the second request by skipping prefill on the first two tokens and reading in the existing KV cache instead.

When you see pay-per-token APIs charge less for “cache hit” input tokens than “cache miss” tokens, this is why: reusing cached tokens takes very little compute power and time. As an inference engineer, you can apply the same principle to reduce latency, improve throughput and therefore save money on your own deployments.

Saving two tokens won’t make a big impact on TTFT, but prefix caching can skip prefill on thousands of tokens in certain domains:

• Complex system prompts: Agents, customer-facing chatbots, RAG scaffolds, and tool calls often feature long, complex system prompts on every call.

• Code completion: Code completion, code generation, and other coding functions require passing the same thousands of lines of code as shared context.

• Documents and retrieval: Document summarization, question answering, and retrieval all add repeated context ahead of user prompts.

• Multi-turn conversations: Ordinary conversations repeat back every message in a chat template, increasing the savings from prefix caching with every turn.

Prefix caching works from the start of the input sequence until the first non-repeated token. The fourth token in the weather example, a question mark, is shared between the two input sequences. However, the prefix ends at the first non-repeated token, so the fourth token isn’t read from cache.

Since prefixes end at the first unique token, your context engineering determines TTFT savings. Consider a different approach to the same prompt:

Here, there is no savings from prefix caching, as the very first token differs between the two sequences, even though every subsequent token is the same. To take advantage of prefix caching, ensure that novel tokens are as late in your prompt as possible.

Approach #4: Parallelism

Tensor Parallelism (TP) should be your default strategy for multi-GPU model inference. It supports dense models like Llama 405B, and the MoE (mixture of experts) models that currently dominate the open model landscape.

TP works by splitting apart each layer of the model (as opposed to Pipeline Parallelism, which keeps layers intact) and distributing the layer fragments across the allocated GPUs. For each layer, the expense of reading from weights’ memory and executing matrix multiplication is shared across the GPUs.

However, the results of each layer need to be communicated in an all-reduce fashion (across all eight GPUs) into a single output before the next layer can be computed. In nodes with high-bandwidth intra-node NVLink and NVSwitch, this communication overhead is minimized.

Increasing Tensor Parallelism improves TPS on a per-user basis, assuming the model is large enough and the sequences are long enough that the communication overhead doesn’t outweigh the faster forward pass – which is the case for most frontier models.

Expert Parallelism (EP) neatly divides experts across GPUs, so that in a model with 128 experts served in EP8 across eight GPUs, each GPU hosts 16 full experts.

EP improves total system throughput, making inference more scalable and less expensive. With individual experts processing tokens separately, each token takes just as long, but the system as a whole can handle more simultaneous tokens.

Many deployments use a mix of TP and EP to achieve both benefits.

EP requires less inter-GPU communication than Tensor Parallelism. The Expert Router, which determines which experts each token activates, is replicated onto each GPU as it is a relatively small component of the model. Inter-GPU communication is necessary for passing tokens from expert to expert, but unlike TP, it is not required to collect the results of each layer.

Thanks to this lower communication overhead, EP scales well to multi-node deployments and systems with limited interconnect bandwidth.

Approach #5: Disaggregation

Disaggregation combines three important ideas in inference engineering:

1. Prefill is a compute-bound process that determines the time to first token (TTFT), while decode is a memory-bound process that determines TPS.

2. Specialization improves performance in everything from kernel selection to inference engine parameter tuning.

3. You can effectively parallelize model serving over multiple GPUs, or even multiple nodes, if you can avoid bottlenecks from lower-bandwidth interconnects.

When prefill and decode run on the same node under heavy traffic, they have a higher chance of interfering with one another. Ideally, prefill uses more compute resources, while decode uses more memory, and the two can co-exist efficiently. However, with larger batches and more compute-intensive optimizations, prefill and decode start competing for resources.

Disaggregation, or disaggregated serving, is the idea of separating prefill and decode into separate engines on separate GPUs or nodes.

Disaggregation turns LLM inference into a three-step process:

1. The prefill engine takes the input sequence and generates a KV cache while computing the first token.

2. The prefill engine sends the KV cache over the hardware interconnect to the decode engine.

3. The decode engine computes all subsequent tokens.

In conditional disaggregation, the request is first sent to the decode engine, which checks if the input sequence is already cached, or is short enough to handle locally:

1. If so, the decode engine handles prefill locally, skipping disaggregation.

2. If not, the decode engine transfers the request to the prefill engine for disaggregated serving.

Conditional disaggregation is better for real-world traffic.

Another benefit of disaggregation is that with separate prefill and decode engines, you can optimize each engine individually and the system as a whole. For example, the compute-bound prefill engine requires a lower TP than the memory-bound decode engine.

Takeaways

This is Gergely again. Thanks to Philip for this deepdive into inference engineering, which is around 10% of the contents of his new book, ”Inference Engineering.” If you’d like to go deeper into this topic, you can download the full book for free:

Get the full e-book, for free

This title will also be available in physical, printed form: sign up to the waitlist to be notified when it’s available.

It’s encouraging that inference engineering is no longer a “monopoly” belonging to a few leading AI labs. Top AI model makers like OpenAI and Anthropic control all aspects of their AI models – from training to inference – so there’s no inference engineering to be done with them.

However, thanks to increasingly capable open models, engineering teams have the opportunity to tweak how they use models, and this is where the theory and practice of inference engineering becomes invaluable.

Even so, the discipline of inference engineering still seems to only make sense for a subset of tech companies. To justify investment in inference engineering, you need to be spending big money on inference from vendors. This is the point at which it can make sense to invest time and money to see if you can set up your own inference stack on top of open models, and swap out some existing usage.

I wonder if inference engineering is the AI version of the “build vs buy” dilemma. For software-as-a-service (SaaS), the question for every company is whether to build it in-house, or buy from a vendor. For example, should you build a project management software (it’s possible!), or just buy an existing one? And what about feature flagging, not to mention observability?

Experienced engineers all understand the pros and cons of building it yourself (time and maintenance, which is a constant drag.) Tuning and operating your own LLM stack is a much newer field, and inference engineering is at the heart of building better inference stacks than what comes “out of the box” with open models.

Picking up the basics of inference engineering feels like a valuable skill – and it’s also new and interesting. If you become well-versed in inference engineering, you could create optionality for your own team and company in LLM usage. Running your own inference stack on top of an open model gives control of what you’re running and of pricing. Inference engineering helps create options for achieving better performance from an open model by using the approaches covered in the extract above from Philip’s book.

After I left Uber, we stayed in touch, and a few months ago I heard he was exploring the next opportunity, and found out from him that Sam’s job search looked nothing like most people’s: he didn’t apply for a single role. Instead, there were reachouts from former colleagues desperate to hire them.

When we talked for this article, Sam had three warm leads which wanted to interview him ASAP. One startup was not even hiring, but the founder was ready to create a new position just for him.

I posted a message on LinkedIn about Sam:

“I hate the term “10x engineer” but this engineer is a role model for what a standout engineer is - in fact, some of my writing of standout engineers reference my interactions with folks like them (e.g. my article on the product-minded engineer, this one: https://lnkd.in/et7nWBgW)

And still, from the outside, this engineer is nearly completely invisible.

No social media footprint. The LinkedIn profile lists his companies worked at, and nothing else: no technologies, no projects, nothing. Their GitHub is empty for the last 5 years, and has perhaps a dozen commits throughout the last 10 years.”

This is Sam’s GitHub contributions for the last several years: absolutely nothing.

One of the most upvoted comments on my post was by cloud technologist Olivier Frolovs, who requested an article on Sam for others to learn how he operates. Now, Sam has generously agreed to an interview and has asked to remain anonymous, hence the nom de plume (pseudonym).

He doesn’t seek public attention and has a strong professional reputation. During our chat, he offered pointers for engineers looking to excel, and also as proof that an empty GitHub profile and zero social media presence don’t mean you can’t be a truly standout developer.

I also interviewed one of his former managers from Uber for that perspective. Today, we cover:

1. Getting things done. High-level task breakdowns, combined with communicating delays as tradeoffs to stakeholders.

2. Setting boundaries. Saying “no,” prioritizing family or work – and being clear about it, and treating prioritization as a daily practice.

3. Office politics. Participate selectively and cautiously, build relationships with influential colleagues, pre-sell ideas, direct communication.

4. Negotiation and conflict. Approach engineers before their managers, build bottom-up consensus, and start with relationship-building.

5. Promotions, keeping up-to-date, and finding the next job. A personal take on The Big Tech promotion processes, keeping up with the industry, and relying on referrals more.

6. Becoming a manager. Ownership and independence separate “good” engineers from “great” ones.

7. Feedback from Sam’s ex-manager. A former manager reveals what made Sam stand out to them – and shares some potential growth opportunities.

Background

In this article, my questions are in italic and Sam’s voice is in normal text.

Sam, how did you get into tech?

I was always intrigued by computers, software, and – as it came around – the internet. My dad had a black-and-white screen laptop for work with Windows 3.1 and some games. We got our first personal computer with Windows 95 and I remember it vividly.

I started developing websites for my primary school and the company my mom worked at when I was 12. I got paid a small sum, so that was my first-ever paid programming project! Around then, I taught myself Visual Basic 6 and started building 2D and 3D mini-games. I found the website DirectXVB (which is still live today), emailed the website’s owner with issues I ran into, and they helped me with pointers. Later, I taught myself PHP and built more dynamic websites.

From the age of 14, I stopped all coding – I just got tired of it! – and focused on my studies, and getting into college. I chose a non-computer science major for university, but picked up coding on the side, and I rediscovered that spark, so, in the first year of my Master’s, I decided to switch and do a Computer Science Bachelor’s. It was during college that I started to build apps and websites, and it’s when I got truly hooked on software development.

From agency, to large company, then Uber

I joined an agency as their first hire, building apps for local companies. It was a small team, we learned with trial-and-error, and getting done at 2-3 o’clock in the morning was common enough. I stayed 18 months and learned a lot about ownership, the importance of an eye for detail, and collaborating with others.

My favorite part of the job was the few times I worked directly with a designer: our agency employed freelance designers who were not involved in most of the projects because the company was trying to save money by having them work less, and not be involved in planning and rollouts. But during the implementation phase, I’d find myself talking with the designer and bouncing implementation and design ideas around.

I then joined a small startup where we built our own product. A highlight was having two designers on the team fulltime, whom I could work with and learn from. Engineering also felt like a level up: everyone cared about software quality and UX details.

Our startup got acquired by a larger company and most of us moved to the Bay Area. We stayed together as a team and were told we would maintain a “startup culture.” The founders tried their best to stay true to their word, but they couldn’t shield us from the reality of working in a corporation.

I learned a lot about corporate processes, and it was more interesting than I’d expected. As I was getting closer to the senior engineer level, I had to understand how internal politics worked, how to “massage” peer teams to help support our proposals, and how to talk with engineering leaders like senior managers and directors. Our company was also hugely focused on the annual company event: it was eye-opening for me to see just how much effort went into preparation. It consisted of several rehearsals and dedicated engineering work to showcase our stuff in a way that was near-flawless on the day.

After a few years, I felt ready for a change and joined Uber. I took a “title cut,” something akin to the “seniority rollercoaster.” At Uber, I worked in a new area and got promoted several times. After Uber, I worked at another Big Tech, and now – very recently – I’ve begun at a startup.

1. Getting things done

Feedback at Uber about you during performance calibrations was that you’re excellent at getting things done. What’s your process?

When I started out as a junior dev, I pulled long hours so I could deliver on time – regardless of how much effort it took. I don’t know what it was, but I always felt that failing to deliver on time was never an option.

I still vividly remember one project where I worked incredibly hard but still failed to deliver with the quality I expected from myself. As embarrassing as it is, I was so exhausted that I almost started crying on the spot. One of my coworkers comforted me and told me:

“Man, you’re crying about the wrong thing. No one died, no one got hurt, and no one will even care that we’re a few days late, save for the project manager. But even he’s used to everything being late. Go home, have some sleep, come back tomorrow and take it easy.”

They were right, of course. Still, I’m pretty sure this inner pressure to be unsatisfied with “good enough” explains a lot about how I work.

Having a high-level breakdown of the work, and communicating to stakeholders has been my “secret”, later in my career. After a few years as a dev, my estimation skills got better and I had to pull fewer late nights. I also found a hack that greatly helped was doing a high-level breakdown as early as possible, in all cases. As soon as I understand what the work is, I break it all down, ideally on a whiteboard or paper.

Importance of communication

You were also seen as a strong communicator, whether it was with engineers, engineering managers, or product managers. How do you get your point across?

Communicating delays as “tradeoffs” works extremely well. As soon as I start a project which I’m the lead on, I establish communication channels with key stakeholders: product managers, my engineering leadership, and business stakeholders, via email or Slack, I keep them in the loop at least weekly, and about anything that could be a roadblock.

In my experience, delays are not an issue as long as they are communicated upfront with an explanation and potential alternatives. When we hit a roadblock that slows down our work, I would never communicate that we’re “behind”. I would offer alternatives like:

• We can still ship on time, but we’d need to cut X and Y features for this release

• If we are not comfortable cutting X and Y features, then we will need to push out the target date by 2 weeks. If we are comfortable, we can push it out by 1 week

The trick, I’ve found, is to make it clear to stakeholders that we have a choice: choose more features to ship, or choose a lower-priority feature to drop.

I learned most of my hacks from people who are good at getting things done, and they have a few attributes:

• Task breakdown: early in my career, there was a senior engineer who was methodical about breaking down tasks and making estimates, even for seemingly trivial projects – and it worked!

• Communication tools: I observed the few really organized product managers, engineering managers, and tech lead, and made their communication styles into a “package” that worked for me; things like email updates, facilitating kickoff meetings, launch announcements (including how to communicate a failed/sunset project as a successful launch), and more.

Being good at communication means you have a solid foundation, and then develop a feel for how to best utilize the tools you have. There’s no “one-size-fits-all” approach: people react better or worse to different things. Try to get to know folks around you and put yourself in their shoes.

Doing great work

What does “standout” work look like to you?

I think about the quality of my work similarly to the quality of work I do at home. I have moved houses and renovated them several times and I greatly care about the quality of that work. And I’ve seen plenty of contractors come to my place, perform their work, and then leave without actually caring about the quality. They just want to “get s*** done” and be out of there. I never understood how someone can keep doing their job without feeling a lot of love for it!

I need to get energy from everything I do, not just in my job. Whether it’s playing games with my kids, helping my wife with her website, or building a new website feature for the company I work for: I approach it all with the same attitude.

Equally, if I no longer get energy from the work I do, then I basically stop enjoying it and this can be a nudge to start to look for something else. If it continues for a long time, this urge can become more persistent, and that’s the point when I have switched companies or teams. I can go on for some time without getting energy from my work, but it drains me. I try to catch myself before it gets too bad, and I’ve managed to do so, up to now. This is why I quit my last job without having anything lined up: I stopped getting energy from it for many months and talked with my management chain about it, but they were unable and unwilling to change anything. I needed a change, so it was me who made it.

Stepping outside of domain expertise

You frequently went outside of your domain, working with engineering teams on other platforms and contributing to codebases you’re not expert in. You seemed to have a great relationship with most engineers, in contrast to some devs. How did you do this?

I am pretty curious and prefer to talk directly with engineers. So, when I’d work on a project with engineers on a different stack, I would ask them to explain their high-level architecture approaches, and roll up my sleeves to make small code changes in a stack I was unfamiliar with.

Once you understand the high-level structure of a different codebase, and you also know how to make a few small changes, suddenly, it’s so much easier to figure things out on your own!

An approach that consistently worked for me is approaching problems from the customer’s perspective, and being genuinely curious. For example, I might ping an engineer working on a different system and ask:

“I noticed a customer has this problem, and to fix it, we probably need to touch the system you own. I don’t know much about this system: can you explain how it works, and what we could perhaps do to solve this issue that causes frustration for the customer?”

By making it clear that my goal is to solve a customer problem, I’m not coming across as just digging around for nothing. And by making it clear that I’d like to learn from them, it avoids being seen as someone trying to confirm what they are doing, which could come across as arrogant – especially when the other engineer is the expert on their own system. I’ve found fellow engineers are happy to explain their understanding and decisions.

2. Setting boundaries

At Uber, I recall you were very good at setting boundaries and saying “no.” How do you do that?

Honestly, I find it tough to say no – but I learned that it’s worse when I don’t. I found that saying ‘yes’ to everything usually results in an unmanageable, unbalanced pile of work. Prioritizing is key: I always remind myself to focus on what matters most. For me, the “most important” thing for any given topic could be:

• A shipping deadline that needs to be hit and is non-negotiable

• Family needs

• An urgent task that needs to be done on the same day

Family is very important to me. When I worked at Uber, I had a reasonably long commute to the office. I blocked out my calendar so I could leave on time in order to be home for dinner with my family. This did not mean I stopped work immediately; I would sometimes work during my commute and, when necessary, I logged back on to continue working after my kids were in bed.

When we had important deadline agreements at work, I made an agreement with my partner that I stayed longer in the office because I knew it was important to put in extra effort and deliver standout work then.

It goes back to prioritizing and focusing on the most important thing. Looking back, I’d say most of the time, the most important thing for me was family, and that work overrode this every now and then.

My approach to prioritizing keeps changing, though. Demands at home keep changing and expectations at work also change; after Uber, other jobs increasingly focused on async and remote work. This meant more flexibility to accommodate family time – but work could spill over into evening hours if I did not finish everything.

If I can give one piece of advice, it’s to understand what is important for you. Know your number one, number two, number three priorities: and arrange your workday so you do your top priorities. Don’t compromise on the most important one!

3. Office politics

At work, how plugged in were you to office politics?

I was aware of politics and tried to build relationships with “influential” people. I try to stay away from “cocky” types, and to find what I want to achieve through different folks.

The importance of politics is something I really started to understand when working at Uber. Initially, I was ignorant, but the more experience I got in Big Tech, the more it became obvious. It took a while before I was able to participate in it. I never liked it; I tend to be direct and transparent, but that does not work in every situation.

Did you take part in it to get stuff done?

Yes, sometimes by being direct and transparent, and communicating the right amount of information, you can get a lot done. Occasionally, it required me to “massage” an idea on multiple people before going to the person who called the shots.

What is your view of engineers who are seen as“political”?

It’s part of the game and sometimes it’s useful to have a good relationship with those people, as you can use that for your own benefit, as well. I personally would never invest much time in understanding and practising politics, as I prefer to focus on building and product.

4. Negotiation & conflict

Negotiating with teams

You were perceived as being good with other teams, and at removing roadblocks for your own. How did you approach this?

Read more

Last week, we took an inside look into how Uber is adopting AI, internally. The rideshare giant has built close to a dozen internal systems to deal with code generated by AI agents. However, when quantifying the impact of AI, the focus was on how much output has increased, and how devs who use more AI also generate more pull requests; these are the “power user” devs who generate 52% more PRs than devs who use AI less. There was no mention of product quality – at all!

And there are signs that product quality is dropping overall. Today, we dig into this under-discussed topic, covering:

1. Anthropic: degraded flagship website. An annoying UX issue irritated paying Claude customers – and no one at Anthropic noticed. The company moves very fast, generates 80%+ of production code with Claude, but quality and user experience seem to be taking a backseat.

2. Amazon: AI-agent reliance triggers SEVs. Amazon’s retail org has a leap in outages caused by its own AI agents. Now, senior sign off is needed for junior engineers’ AI-assisted changes.

3. Big Tech: “use AI or you’re unproductive.” Companies like Meta and Uber are tracking AI token usage in performance reviews, putting pressure on engineers to use it heavily — irrespective of the tools’ quality impact.

4. OpenCode: more time spent cleaning up. Dax Raad, OpenCode’s creator, warns that AI agents are lowering the bar for what ships, discouraging refactoring, and don’t speed teams up.

5. 5. Startups: founders see LLMs slowing down long-term velocity. Sentry’s CTO and others observe that while AI removes the barrier to getting started, it also produces bloated, hard-to-maintain code that slows long-term development.

6. Research: AI agents underperform claims. Some studies show AI coding tools produce short-lived velocity gains followed by significant tech debt increases.

7. How do we solve it? Engineers with strong architectural sense become more critical than ever, proposed solutions include formal validation methods, and perhaps reviving some old school QA ideas.

1. Anthropic: degraded flagship website

This article’s genesis was last week, when I’d finally had enough of a persistent UX bug on Claude’s flagship website: the prompt I typed in regularly got lost. Below is a video of me typing “How can I…” – and “losing” the first two words when the page loaded:

It’s pretty straightforward:

1. The page starts to render and the textbox is displayed

2. The user starts to type their prompt, but the page has not finished loading subscription data

3. The subscription information loads around a second later

4. The textbox is reset and the typing is lost

This is a pretty basic bug you might expect in a prototype, except that this is the landing page of Claude.ai, and it’s a bug that impacts every paying customer – easily millions – every day. Even worse, the bug happens every time you visit the site.

Somehow, nobody at Anthropic tested the site to catch a plainly obvious bug which impacted 100% of paying customers. At the same time, no company uses AI coding tools more than Anthropic: around 80% of the company’s code is now generated by Claude Code, so we can assume a good part of the website is also created that way.

My complaint about Anthropic’s website being broken went a bit viral and got the attention of the developer team:

To their credit, three days later the bug was gone. There’s no longer a “double load” of the textbox: it takes a bit longer to load but only does so once.

Still, it makes me wonder how much longer this issue would’ve continued had nobody complained. Also, how many more bugs are present on the Claude website that nobody highlighted on social media? How many more features could be shipped in a state that is subpar for production-grade software with millions of paying customers?

Anthropic seems to be prioritizing moving very fast over doing so with high quality. There is no denying that the company is moving at incredible speed and running laps around competitors. A good example is how they built Claude Cowork in just 10 days. Claude Cowork handled work with Microsoft Word and Excel documents surprisingly well, to the point that it set off a “code red” inside Microsoft’s Office division, I understand.

Microsoft responded as fast as possible, but it still took 2-3 months to launch their (cloned) response, called Copilot Cowork earlier this month, with full access still to follow soon.

In the case of Anthropic, moving fast with okay quality seems to make good business sense: they build a better product than what already exists, so no matter if it’s a bit rough around the edges; they can fix quality issues post-launch and still be months ahead of the competition.

2. Amazon: reliance on AI agents causes SEVs

Anthropic can afford to move fast while it’s growing at an extremely high rate and expanding its market share rapidly. At the same time, established players like Amazon have extreme focus on reliability: AWS has become the top cloud provider not least by being extremely reliable (as well as aggressive on pricing).

Well, reliability at the online retailer seems to be getting worse, too, and the company’s AI agent, Kiro, could be causing SEVs (Amazon’s phrase for “outage”), according to The Financial Times (emphasis mine):

Amazon’s ecommerce business has summoned a large group of engineers to a meeting on Tuesday for a “deep dive” into a spate of outages, including incidents tied to the use of AI coding tools.

The online retail giant said there had been a “trend of incidents” in recent months, characterised by a “high blast radius” and “Gen-AI assisted changes” among other factors, according to a briefing note for the meeting seen by the FT.

Under “contributing factors” the note included “novel GenAI usage for which best practices and safeguards are not yet fully established”.

“Folks, as you likely know, the availability of the site and related infrastructure has not been good recently,” Dave Treadwell, a senior vice-president at the group, told employees in an email, also seen by the FT. (...)

He asked staff to attend the meeting, which is normally optional.

Junior and mid-level engineers require more senior engineers to sign off any AI-assisted changes, Treadwell added in the briefing note.”

This meeting was the regular “This Week in Stores Tech” operational one, but what was new was the note telling staff to attend this “optional” meeting, and the mandate for senior engineers to sign off code changes from juniors. The outages may have been caused by less experienced engineers over-trusting GenAI’s output. Also, there were incidents caused by AI changes, said the FT:

“Separately, the company’s cloud computing arm — Amazon Web Services — has suffered at least two incidents linked to the use of AI coding assistants, which the company has been actively rolling out to its staff.

AWS suffered a 13-hour interruption to a cost calculator used by customers in mid-December after engineers allowed the group’s Kiro AI coding tool to make certain changes, and the AI tool opted to “delete and recreate the environment”, the FT previously reported.”

Again, a tool causing an outage is not its own fault: it’s on the engineer who lets the tool run wild. If I delete two lines of code, then push it to production, and the server crashes, the fault is not with the text editor or the Git client, but with me who made the change. Similarly, if you prompt an AI agent to do something, and the AI agent goes off and does its stuff which causes an outage, then responsibility lies with the engineer who didn’t set up guardrails for the agent.

However, there is the issue that AI agents can wreak havoc in ways devs don’t quite understand or expect, until learning the hard way. This was what took down a lesser-used AWS service, according to the report:

“Amazon Web Services experienced a 13-hour interruption to one system used by its customers in mid-December [2025] after engineers allowed its Kiro AI coding tool to make certain changes, according to four people familiar with the matter.

The people said the agentic tool, which can take autonomous actions on behalf of users, determined that the best course of action was to “delete and recreate the environment”.

It sounds like an engineer gave overly broad permissions to the coding agent which then used its scope to delete a service. As mentioned, the engineer is responsible, but there is also a learning curve with these AI agents to consider: it’s not like this type of outage happened in the past. Plus, companies like Amazon are heavily incentivizing using AI agents for as much work as possible, which naturally leads to overuse.

3. Big Tech: “use AI or you’re unproductive”

Something happens at places that measure devs’ AI usage: pressure builds for all devs to use more AI or else be seen unproductive and at risk of poor performance reviews, potentially leading to a PIP or worse.

Meta is taking token usage into account during perf reviews. A current engineering manager at the social media giant told me that the token usage of each engineer is now a data point — one of many! — for performance calibrations. By itself, it is not a positive or negative signal, but someone perceived as having low impact and with low token usage is now seen as a blatant low performer. For high performers with outstanding impact, very high token usage is seen as a good thing as it conveys to the manager group that they’re personally invested in AI and are improving their workflow – as proved by results.

We previously covered how performance calibrations work at places like Meta.

Big Tech CEOs are starting to see AI “power user devs” as superior to their coworkers. Uber is a good example: the Dev Platform team started to analyze the output of engineers by whether or not they’re in the “power user” category, meaning they use AI agents at least 20 days per month. They found more PR output by engineers who are power users. So far, that’s useful data, but it’s just one piece of information, and doesn’t reveal the quality of the PRs, the impact of the engineer, or any other business outcome.

By the time this data reaches CEO level, it has turned into something else. Here’s Uber CEO, Dara Khosrowshahi, interpreting the same data points on the Diary of a CEO podcast (emphasis mine:)

“While 90% of our engineers are using AI tools of some sort, there’s about 30% of them that are using them at a completely accelerated pace. And it [using AI tools heavily] really is changing their productivity in a way that I’ve never ever seen before.”

There’s a step from observing more PRs per engineer, to judging power users as being more productive for that reason. Dara continues:

“I can imagine maybe 5 years from now, as the engineers get more and more productive, that I may not decide to add engineering headcount because at that point instead of adding an engineer, I should add agents and buy some more GPUs from Nvidia. That may be the investment in the future.”

Unsaid in the above is that by that time, only engineers “using AI at a completely accelerated pace” would be employed. Would it also mean that engineers not on the bandwagon are on the way out? I appreciate Dara speaking his mind and shedding light on the thought process of a Big Tech CEO.

Inside large tech companies, it’s becoming a career risk to not use AI at an accelerated pace, regardless of output quality. These large companies are the ones likely to be mulling layoffs, like Meta reportedly preparing to cut up to 20% of staff. And when it comes to identifying redundancies, it’s a fair assumption that things like “AI usage” and “pull requests per engineer” will be taken into account, especially as one theme of such layoffs will almost certainly be that the employer wants to focus more on AI.

So, it’s common sense (and self-preservation) to use more AI, if only not to be seen as unproductive. Their perceived output will rise and engineering leadership will share more reports about productivity being up, and interpreting more code generated and more pull requests as the proof.

4. OpenCode: “more time spent cleaning up”

Dax Raad is founder and CEO of OpenCode, an open source AI coding agent, into which you can plug in models like Claude, ChatGPT, Gemini, and others. It’s an increasingly popular alternative to the likes of Claude Code and Codex. In our recent AI tooling survey, it came up as a tool used nearly as much as Google’s Gemini CLI and Antigravity. A small team works on this increasingly influential tool, and is seeing problems with AI overuse. Dax wrote this note to the OpenCode team (emphasis mine):

Read more

Update on 11 March: the Uber team shared updated numbers as of March 2026:

• 84% of devs at Uber are agentic coding users (either using CLI-based agents or making more agentic requests than tab-completion in their IDE)

• 65-72% of code is AI-generated inside IDE-based tools. This number is, naturally, 100% for AI command line tools like Claude Code.

• Claude Code usage nearly doubled in 3 months — from 32% in Dec to 63% in Feb, while IDE-based tools (Cursor, IntelliJ) have plateaued.

The article is updated with these new and correct numbers.

I spent four years working at Uber until 2020 and experienced firsthand the company’s standout engineering culture. Uber is a company that did the speed run of going from a small startup, through hypergrowth, to being a large company facing major risk during the pandemic, when the rideshare business briefly collapsed. Today, it’s maturing as a publicly traded, profitable company, and employs almost 3,000 people in the tech function.

At the recent Pragmatic Summit in San Francisco, one of the most interesting behind-the-scenes sessions came from the ridesharing company’s principal engineer, Ty Smith, and director of engineering Anshu Chada, who pulled back the curtain on what Uber has been doing with AI tools, internally. They were candid about the amount of work it took to build up Uber’s internal “AI stack,” why all that work was necessary, and also discussed the drawbacks as well as benefits of this rapidly spreading technology. You can watch their presentation at The Pragmatic Summit here.

In today’s issue, we cover:

1. Agentic layers & systems. Four layers spanning an internal AI platform, context sources, industry tools, and specialized agents for testing and code review.

2. Internal tooling: MCP Gateway, Uber Agent Builder, and the AIFX CLI. Uber built several internal tools to make it easier for devs to use AI tools, and to make internal AI agents more effective.

3. How AI changes developer workflows. A move away from single-threaded coding in an IDE, to orchestrating multiple parallel agents. Engineers naturally gravitate toward kicking off new agents, which starts to create resource and cost challenges.

4. Minion: running background agents at scale. Uber built Minion, an internal background agent platform with monorepo access and optimized defaults. It’s a clever abstraction layer that works well in practice.

5. New internal dev tools. More AI-generated code means more code reviews and more noise, so Uber built Code Inbox for smart PR routing, uReview for high-signal AI code review comments, Autocover for generating 5,000+ unit tests per month, and Shepherd for managing large-scale migrations end to end.

6. Challenges. AI adoption is slower than expected, even at a forward-thinking company like Uber. Top-down mandates are less efficient than engineers sharing their wins with peers.

7. Impact in numbers. 92% of Uber devs use agents monthly, 65-72% of code is AI-generated inside IDEs, and 11% of pull requests opened by agents. At the same time, AI-related costs are up 6x since 2024, and token cost optimization is a growing priority.

Longtime readers might recall we’ve covered Uber’s engineering culture over time:

• Developer Experience at Uber – with Uber’s founding engineer on Developer Platform, Gautam Korlam (2025)

• How Uber built its observability platform (2023)

• Inside Uber’s move to the Cloud (2023)

• How Uber is measuring engineering productivity (2022)

• The Program and Platform split at Uber (2021)

Let’s get into it:

AI is not new at Uber, but rolling it out companywide is. The company has used machine learning and AI technologies in many systems, including its Marketplace platform, which are responsible for routing and matching drivers with riders, forecasting demand, etc. What is relatively new at nearly all tech companies is the process of integrating AI across engineering and beyond.

The official strategy at the ridesharing giant is to become a “GenAI-powered” company:

I appreciate Uber sharing this approach openly because while most companies say that they want to be “AI-powered” – however cliche that claim might be – not all provide as much transparency.

It’s worthwhile engineers internalizing how leadership views AI. These folks, in general, see a tool that can bring efficiency everywhere. My take is that in some ways, AI is seen similarly to the cloud, which has been perceived as a means to reduce costs and improve the flexibility and elasticity of hardware resources. Today, AI is seen as the way to increase efficiency and lower costs, such as customer support, software development, the finance function – or any function.

Uber is focusing not on automating everything possible in engineering. Instead, it wants to:

1. Eliminate toil: helping AI do “boring” work like upgrades, migrations, trivial bug fixes, etc.

2. Free up engineers to focus more on creative work.

As Anshu Chada, Engineering Director on Uber’s Dev Platform, puts it:

“What we found is when we push some of the boring stuff to AI – upgrades, migrations, bug fixes – not only does it result in much higher satisfaction from our engineers, but they’re able to push our product and create features for end users in ways that we didn’t even think were possible.”

1. Agentic layers & systems

Uber’s “agentic system” for software engineering is actually made up of several systems:

Categories of systems:

• Internal AI platform. Built on top of Michelangelo, Uber’s ML/AI platform. This layer provides things like a model gateway to proxy to frontier models or internally hosted models.

• Internal Uber context: accessing Uber’s source code, engineering documentation, Slack information, JIRA tickets, etc. These all serve as “memory” for agents to use.

• Industry agents: Uber’s approach is to enable the “latest & greatest” AI agents for engineers, so they support several tools like Claude Code, GitHub Copilot, Codex, and other clients.

• Specialized agents: Uber’s background agent platform, the test generation platform, code review agents, and more.

• Engineering enablement: measuring the efficiency of agents, controlling costs, and educating engineers about which tools to use.

2. Internal tooling: MCP Gateway, Uber Agent Builder, and the AIFX CLI

MCP – the Model Context Protocol – has quickly become the standard way to connect agents and data sources with one another. A frequent analogy is that MCP is like the “USB-C interface for AI agents.” We published a deepdive on the MCP protocol and covered real-world MCP server use cases.

Uber put together a “tiger team” (a temporary unit that gets things done fast) to design the MCP strategy and build the central MCP gateway, which looks like this:

This MCP gateway allows:

• Proxy internal endpoints to MCPs: any internal Thrift, Protobuffer, or HTTP endpoint can be exposed as an MCP server with a simple configuration change. Uber uses the Apache Thrift protocol and Protobuffer protocols extensively for backend service communications

• First-party MCPs: these are exposed as a single, consistent interface

• Third-party MCPs: external MCP servers are also exposed via the gateway, which handles all authentication and authorization tasks.

• Platform concerns: the gateway takes care of authorization, telemetry, and logging in one central place. Plus, it conveniently offers a unified interface to interact with any MCP.

The MCP gateway also provides:

• A registry: to look up MCP servers, and for devs to be able to register their own.

• A sandbox: for devs to experiment with MCP servers without long-winded setups.

Uber Agent Builder

Uber’s Agent Builder product is a no-code solution to build agents that can access Uber’s internal data sources (both MCP servers and Uber data sets), and hand off work to other agents:

The platform includes a tool called Agent Studio, where multi-agent workflows can be visualized, debugged, traced, versioned, and evaluated. This is how it looks:

The agents built in Agent Builder become discoverable through a registry:

Accessing agents via the AIFX CLI

Uber’s Developer Experience platform team had a few issues with deploying AI agent tooling at scale:

• How does the company update all clients when a new version comes out? For example, if a new version of Cursor is released, how can they ensure all devs use the latest one?

• How can the clients be configured with helpful defaults? Uber’s Dev Platform team might have found better defaults for tools, like more helpful models. How are these rolled out to all devs?

• How can devs easily discover MCP servers and configure them for agents?

• How can agents connect to Uber’s background task infrastructure?

The Dev Experience team built the AIFX CLI, which is the AI tooling command line used by all engineers there. Here’s what it looks like:

This tool supports:

• Provisioning AI agents (client tools like Claude Code, Codex, Cursor, and others)

• Finding and using MCP servers

• Running background agent tasks

• Updating AI agents and clients to the latest versions

3. How AI changes developer workflows

The traditional way of building software:

Devs spent some time planning, most time writing code (code authorship), and then some time in code review.

The first AI agent-based workflows were single-threaded: devs worked with a single agent in the command line or inside their IDE:

At Uber, the latest workflows which many software engineers follow are pretty different, involving parallel agents, each kicked off with their own tasks:

As Ty explained, running multiple agents comes naturally to most devs:

“[Once you start using agents] as an engineer, you’re giving a prompt and waiting for something. While it’s running and you’re waiting you’re thinking: ‘What am I going to do? Have a coffee or browse Reddit? Might as well kick off another background agent.’

And so, engineers get into this mode of running several agents at once, right? Both us [at Uber] and a lot of the industry is trying to push towards this.”

4. Minion: running background agents at scale

Read more