Three weeks ago, at Craft Conference, in Budapest, Hungary, I opened the event with a keynote titled ‘Slow Down to Speed Up’.

As with most of my talks, it came together in stages, including with some input from full subscribers to the Pragmatic Engineer, with whom I shared my thinking in advance, in ‘Ideas: slow down to speed up when working with AI agents’. Thank you for all the comments!

As fate would have it, just two days beforehand, social media giant Meta appositely provided a real-world case study for my talk, with its most embarrassing outage of all time: users could simply ask the Meta AI to change the email of any account, and the bot happily complied – even if the account belonged to someone else entirely – including a former US president. It was a timely example to kick off the talk with. Check out the full keynote that’s available to view on YouTube:

Watch the keynote video

In this article, I summarize the key parts of my Craft Conference keynote in detail, and some responses received at the event. Full subscribers also have access to the slides, here, and at the foot of this article.

We cover:

1. Meta: “AI psychosis” in effect? Meta has been destroying its engineering org, and an obsessive focus on AI seems to be one reason for it. For more on this question, check out this deep dive.

2. Everything’s changed in six months. From around November last year, things changed with a more capable generation of AI agents like Opus 4.5 and GPT-5.4.

3. How are tech companies changing how they work? Anthropic, OpenAI, Google, Uber, startups, and traditional companies.

4. Trends. Individual productivity is up, but team productivity’s flat, tokenmaxxing and tooling adoption, vanishing middle management, CEOs and CTOs back to coding, and more.

5. Trends across software. Falling software quality, GitHub’s constant reliability woes, AI slop overwhelming devs who care about quality, and more.

6. Advice for software engineers and engineering leaders. Suggestions to help future-proof a career.

7. Feedback. “It’s happening here too!” is a common theme, and relief for some that it’s not unique to their own workplace.

1. Meta: “AI psychosis” in effect?

I thought it was a made-up story when I read that Meta had enabled account takeovers via a “zero auth” policy; i.e., simply asking the Meta AI bot was sufficient to change any account’s email address. After all, shipping such a regression would fly in the face of security measures, code reviews, automated testing, and metrics. Plus, the company has dedicated Integrity teams whose mission statement is to ensure something like this never happens… And yet, this bug shipped.

It went undetected by anyone at Meta, and high-profile accounts like that of former US president, Barack Obama, were taken over as a result. Instagram’s dedicated Integrity team seems to have discovered the embarrassing issue via the news.

As mentioned, it was two days before the Craft keynote, so there was enough time to ask around at Instagram and Meta. Engineers at the company there told me this disaster was caused by AI-generated, AI-reviewed code, along with layoffs, and by forced reassignments from Integrity teams and elsewhere onto AI labeling and related duties.

The problem at Meta seems to be that leadership is aggressively pushing AI, while withdrawing resources and headcount from areas responsible for security, quality, and reliability. Since last week’s deepdive into what’s been happening behind the scenes was published, I’ve learned further details:

• Integrity teams at WhatsApp have been hit hard by layoffs and enforced data-labeling reassignments

• Instagram’s design team suffered a 44% cut in headcount during layoffs

• The Developer Documentation and Support team had a full 95% headcount reduction during layoffs

• Data labeling at the ADO group goes beyond “just” labeling; there are many AI training tasks to do. But these are repetitive, unless you get really creative.

Based on everything I heard from talking with Meta folks, AI-induced behavior was indeed at the heart of this outage. AI-generated, AI-reviewed code, and security teams being gutted, were also factors in the beyond-embarrassing incident. As reported in last week’s deepdive:

• Instagram’s Trust and Safety Team lost around 50% of its staff to data labeling and layoffs. Some of the most senior folks were drafted onto AI training tasks.

• AI-generated changes with zero human input, with just an additional AI code review, have been very common in recent months across the codebase. The change that caused this outage looked like one of these

• Normally, the Trust and Safety team would be on top of monitoring and alerting of security breaches, but it is currently in full disarray due to rapid, internal disorganization”.

If major changes like data labeling assignments and staff tracking are undone, then perhaps things at Meta could return to normal. But so far, the most being done is that leadership has boosted budgets for snacks, travel, and events. Hardly the change needed to restore morale and the former culture!

The comparison to the Lumon corporation in the hit show, Severance, was duly made:

Meta’s worst-ever outage can be interpreted as a warning about what happens when there’s so much focus on AI that the basic health of a company’s main – money-spinning – products is neglected. Instagram, WhatsApp, and Facebook generate the bulk of revenue for Meta, but the company is reallocating more engineers to training the coding model, and aggressively cutting the headcounts of vital orgs to do so – up to the point of not having oncall coverage for key services, and security teams being too stretched to do their jobs.

Am I missing some insight about why it’s more important to build a state-of-the-art, likely-closed AI model that’s good at coding, than it is to keep operating revenue-generating businesses with stable infra?

2. Everything’s changed in six months

Independent, experienced software engineers with zero affiliation to AI labs have been saying for a few months that how we do software engineering has been transformed.

David Heinemeier Hansson (DHH), creator of Ruby on Rails in January:

”Just [in] summer 2025, I spoke with Lex Fridman about not letting AI write any code directly, but it turns out part of this resistance was simply based on the models not being good enough at the time! I spent more time rewriting what it wrote, than if I’d done it from scratch. That has now flipped.”

Simon Willison, creator of Django, in May pinpointed the start of the change to late last year:

“The models released in November 2025 elevated agents to being genuinely useful. We’ve had six months to get used to that idea now; it’s no wonder companies are beginning to spend real money on this technology.”

Teams using agents now ship 5x as many pull requests as two years ago. Here’s data from Linear:

Devs using AI harnesses are producing 2.5x as much code versus 18 months ago. Data from Cursor shows that their users, on average, went from adding 3,500 lines of code in January 2025 to 8,600 today:

The size of pull requests is up 3x versus 18 months ago. Also from Cursor:

More AI changes are accepted without human review. Data from Cursor shows a big jump in changes being accepted without human review from around February this year, when Opus 4.7 and GPT 5.5 launched:

We’re seeing a lot more code generated, and less of it than ever being reviewed by devs. In the relatively short time since AI agents became really good last November, there are more pull requests generated by devs, those pull requests are getting better, and code reviews are harder to keep up with. And so, reviews are less stringent and more changes are shipped to production sans human review! As per my discussions with Meta engineers, these kinds of AI-generated, AI-reviewed pull requests [at Meta, they’re called diffs] are what caused the most recent, embarrassing outage at Instagram.

3. How are tech companies changing how they work?

Details from a few larger tech companies:

Anthropic: all-in on AI agents. In March, Boris Cherny, creator of Claude Code, was on the Pragmatic Engineer podcast and shared some details:

• He personally runs ~5x agents parallel, and ships 20–30 PRs/day

• Product requirement documents (PRDs) are dead & prototypes have replaced them inside Anthropic

• ~100% of Claude Code was generated by Claude in March

• ~70-90% of code inside Anthropic was generated by Claude

• Claude Cowork – another billion-dollar product in terms of revenue potential – was built in just 10 days

Since then, Boris has shared that his workflow has changed to setting up loops to run agents.

OpenAI: moving much faster with AI agents. OpenAI’s Codex team was on the main stage at The Pragmatic Summit in February. Tibo Sottiaux (head of engineering, Codex, OpenAI) shared interesting details on how software development is done in the Codex team:

• There’s a “fix this” button integrated into the internal OpenAI mobile app. It makes one-shot fixes to bug reports, which devs review and can merge

• AI code review for all code changes. With a tiered approach, some changes can be merged with just AI review, and more important ones need an extra human review

• Most devs run several agents in parallel, often walking around with their laptop lids open, so the machine doesn’t enter sleep mode and suspend agents

• Code isn’t really written by hand anymore on the Codex team, and is also less common on other teams too

• “Taste” is becoming a core skill for working at the company

• Codex improves itself: it runs its own test suite, runs improvement tasks overnight, and during team meetings it takes actions on topics discussed

Google: AI widespread. Gemini is not as capable at coding as Claude or Codex, as acknowledged by Google’s CEO, but it’s widely used companywide. The less capable coding model could be hurting AI adoption compared to other companies.

Uber: in-house AI infra. We covered in-depth how Uber uses AI for development, touching on internal systems like:

Uber’s MCP Gateway:

Uber Agent Builder:

The AIFX command line interface:

Minion: background agents

Code Inbox:

Smart Assignments as a neat feature of Code Inbox:

Risk Profiles: another smart feature inside Code Inbox:

uReview, Uber’s AI code review tool:

Autocover and Shepherd for large-scale migrations:

Uber is a good case for learning how much of internal developer infra needs to be rebuilt in order to work well with AI agents. Uber built all the tools above because they needed new, better ways to integrate AI agents into the developer workflow, but couldn’t find anything that worked up to requirements. I’d also point out how much time and effort Uber invested in making code review more efficient. Devs are, indeed, getting overloaded with AI code reviews and Uber’s Code Inbox tries to separate the important pieces of code to review from unimportant ones.

Startups are jumping into using AI agents, although their integrations are more basic. In preparation for the keynote, I talked with several startups about their AI usage. Harnesses like Claude Code, Codex, Cursor, OpenCode and others are popular, and I also noticed most startups are heavily integrating AI agents into Slack, so devs can kick off bugfixes or small feature requests straight from the chat tool.

I observed startups being the most likely to experiment with new AI dev tools; from code review, all the way to AI incident management tools.

“Traditional” companies are also heavily investing in AI dev tools. At the recent Pragmatic Summit in San Francisco, Laura Tacho shared interesting details:

• In February, 18,000 Cisco developers used Codex for complex migrations, code review, and refactoring. This was very early – Codex was just starting to gain industry-wide adoption!

• JP Morgan Chase built a multi-agent framework for annotation, using multiple specialized agents to label customer interaction data, and judge agents to aggregate and rank results. These are pretty advanced use cases!

In general, “traditional” companies do not seem to be lagging behind in using, paying for, and adopting AI agents and AI developer tools.

4. Industry trends

There are trends I’ve observed around the adoption of AI dev tools:

Read more

Subscribe now

Many subscribers expense this newsletter to their learning and development budget. If you have such a budget, here’s an email you could send to your manager.

For two decades, Meta had a unique, high-performance engineering org; right up until around April of this year. For the first 20 years of the company’s existence, it had a “move-fast-and-break-things” culture, and in the early 2020s this shifted to a “move-fast-with-stable-infra” one. Engineers I know at the company were empowered to do good work, focus on impact, and to balance business interests with solid engineering.

But in the past few weeks, all that has changed, as if the leadership has been following detailed blueprints on how to demolish a proven, successful engineering culture in the most ruthlessly efficient way possible.

For the past few weeks, I’ve been sharing how bad things are inside the social media company for engineers in one of Silicon Valley’s most prestigious workplaces. In this article, we walk through what’s happened, and ask what’s going through the minds of leadership who are reducing software engineering there from the profit center that it was between 2004 until very recently, to the disdained cost center that it has become in just a few weeks.

We cover:

1. Meta’s pre-AI engineering culture

2. Investing in AI and pressing engineers to always use it

3. Core engineering folks feel treated like trash

4. Most embarrassing-ever outage

5. Internal mess

6. Self-inflicted wounds

7. Is “AI psychosis” just a Meta issue?

1. Meta’s pre-AI engineering culture

I’d split Meta’s engineering culture into two eras: “move fast and break things”, and then “move fast with stable infra.”

“Move fast and break things”

In the 2010s, Facebook’s unconventional engineering culture had grown somewhat legendary in the tech industry, as the company went against conventional best practices and succeeded massively.

In 2012, when Facebook hit the billion-users landmark, the company produced a small physical book about its culture which was placed on employees’ desks. Presented with retro propaganda design, it was dubbed the “little red book”, co-opting the name of a famous volume of the thoughts of Chairman Mao, (1964).

At around 70 pages long, Facebook’s version codified its engineering culture: speed, fearlessness, taking ownership, and thinking outside of the box.

Back then, mantras in Facebook’s little red book were also in print across campus, and included:

• Move Fast and Break Things

• Done is Better Than Perfect

• Fail Harder

• What Would You Do If You Weren’t Afraid?

• Every Day Feels Like a Week

• The Wright Brothers Did Not Have Pilot Licenses

• The Foolish Wait

• Fortune Favors the Bold

There was genuine focus on building good products. Also from the book:

“Move fast with stable infra” culture

In 2022, I did what is one of the longest deepdives we’ve published on the topic of Meta’s engineering culture. By then, things had evolved, and much of any former recklessness was gone, replaced by the principle of moving fast, but with stable infra. Here’s how I described Meta’s engineering culture then:

“The culture is incredibly engineering-centric: much more than most of Big Tech. This might come from Mark Zuckerberg being an engineer himself, or because much of the innovation in the early days of Facebook came from engineers.

Focus on individual impact. Impact has been the bread and butter of the focus at Facebook. This is very true since the early days, and the focus on generating impact remains.

One detail in common with most Big Tech firms is that both the engineering culture and general culture focus so much on individual impact. This results in some people focusing on short-term, measurable wins and assuming that teamwork and split wins between groups might be less rewarded.

The lack of rigid processes. Facebook seems to have the least amount of processes or standardization across all of Big Tech. Don’t even try to compare it to Amazon’s engineering culture and the countless formal processes there. But even compared to companies like Google, Microsoft or Uber, Facebook’s processes are much looser. Most of this comes from the engineering-centric nature of the company and engineers disliking processes.

Surprisingly little emphasis on testing, documentation or code comments. You’ll find shockingly little automated testing and documentation at Facebook, compared to the rest of Big Tech. Inline code comments are also very rare.

A founder-engineer driven company. Facebook is one of the few Big Tech firms whose founder is an engineer, and still is the CEO. Netflix is the other one where founder and co-CEO Reed Hastings was also a software engineer before starting the company. Amazon was the other example of this until recently, but it’s not the case at Google or Apple. There are good examples of smaller companies like Cloudflare, but they’re all younger than Facebook.

Bootcamp. A unique onboarding process, unlike what any other Big Tech firms offer. We cover this more in the Bootcamp & onboarding section.”

Also, Facebook, as a product, has one of the most sophisticated auto rollout systems in the industry. Instagram has a battle-tested infrastructure where it was almost trivial to launch a new social network (Threads) with 100 million users served in its first week.

Engineers whom I knew inside the company are capable, motivated, and product-minded, and their work was appreciated. CEO, Mark Zuckerberg, was influential: he personally coded the first version of Facebook, had stayed close to engineering, and valued software engineers very much. Engineers there felt they were working inside a profit center.

2. Investing in AI and pressing engineers to always use it

Meta has been the only company among the big five of Apple, Microsoft, Amazon, Google, and itself not to own a hardware platform or operating system. Apple has the iPhone, iPad and Macs, Google has Android, ChromeOS and Pixel phones, Microsoft has Windows, and Amazon has the Kindle.

Stepping back, it looks as though the Mark Zuckerberg of today has resolved not to miss a platform opportunity, after the company failed to build its own mobile OS or mobile phone during the 2010s.

This is one reason for investing so much in virtual reality (VR) with Oculus, and in augmented reality with the Meta Glasses. Facebook changed its name to Meta in 2021, back when it looked like VR – and the metaverse – could be massive. Billions was spent on ensuring Meta would be the market leader in this space. But once again, VR didn’t go mainstream; since the end of the pandemic, popular interest in the segment has died down considerably.

When it became clear that AI would become a mega-trend in 2022, Zuckerberg didn’t miss it: he assembled the internal FAIR group (Fundamental AI Research team) as well as a GenAI product organization and released a series of open-weight AI models:

• Llama 1: released in Feb 2023, three months after ChatGPT, built by FAIR

• Llama 2: in June 2023, built by the GenAI product organization (as well as all subsequent Llama models)

• Llama 3: in April 2024. This model was Meta’s most competitive LLM of all, and gained momentum in adoption across the industry

• Llama 4: in April 2025. This model was deeply disappointing

In June that year, Meta acquired a 49% stake in Scale AI to reboot its AI efforts for a whopping $14.8B, and brought in Scale AI’s CEO, Alexandr Wang to take over Meta’s AI strategy. The acquisition of Chinese startup Manus AI for $2B is currently in question after China blocked the deal from being completed.

Based on the investment made into Scale AI and Wang, it’s pretty clear that Meta – and Zuckerberg – is determined to build a state-of-the-art LLM that can be competitive with the latest versions of Claude and ChatGPT. But Meta has to start pretty much from scratch, and it’s up to Alexandr Wang to deliver.

Scale AI brings in a very specific kind of expertise to Meta, as one of the best in the industry in:

• Training data and labeling: Scale started, and is still best known, as a provider of high-quality labeled datasets for machine learning and AI training, including code, text, image, video, etc.

• RLHF and fine-tuning: A RLHF (reinforcement learning from human feedback) flow which Scale runs, where people give feedback for foundation models, as a “human in the loop” data engine that many leading AI labs use to create better LLMs.

Wang seems to have a very broad reign to do what he has been an expert in: creating training data, doing data labeling and RLHF. This is being pulled off with the labor of Meta’s engineering workforce, and by surveilling it.

Problem #1: Tracking keystrokes and mouse clicks, with no option to opt out. In late April, Meta told engineers they were being enrolled into a system that tracks every keystroke and click, to produce training data for Meta’s new AI. There’s no way to opt out.

Needless to say, this is invasive and raises privacy questions: If you log into your personal bank account, does the tool track you? What about when you’re writing a personal email, or responding to a personal call? Meta held no consultation and there are no workarounds; just a top-down decision being pushed through.

This month, Reuters reported that people’s concerns there are finally being heard:

“Meta is dialing back elements of its plan to collect employee mouse movements, keystrokes and other actions for use as AI ​training data, it said in an internal memo on Tuesday, following weeks ​of angry pushback from staffers.

New controls will allow employees to pause ⁠the data collection for up to 30 minutes at a time and ​request exemptions from the initiative, according to the memo, authored by Stephane Kasriel, ​a vice president in Meta’s AI model-building Superintelligence Labs unit.”

From talking with current Meta engineers, I understand the logging system has not been rolled out in the UK due to data protection regulation.

Problem #2: 30-50% of engineers on core teams have been forcefully reassigned to data labeling and RLHF, upsetting folks even more. Also starting in late April, product engineering teams received a mandate from above, whereby 30-50% of engineers were to leave the team and join the ADO org (Agent Data Optimisation).

“Forceful” reassignment is very relevant here because of Meta’s traditional engineering culture. Between its founding in 2004 and until last year, Meta gave engineers autonomy to choose where they work and what they work on. This was structural to how the company worked:

• Engineers were not hired for a specific team (save for at the Staff+, levels, in some cases). They were hired to the company

• During a 6-week bootcamp, new hires got familiar with Meta’s engineering culture and chose a team

• Team matching meant talking with multiple teams who had headcount, doing small work with them, and finding a match

• Internal transfers were easy, and often initiated by engineers

Team selection via bootcamp started to die down in around 2024, but any Meta engineer with at least two years’ tenure knows that previously they chose what to work on, and of course, could pick the most impactful thing to work on. And then, out of the blue, they’re assigned to a division where the impact is not clear, the work is menial, and doing it too long will surely hurt their career prospects.

“Data labeling” is more involved work, even though a bit repetitive. There are labeling tasks, where you create a website, then look at it and decide if it looks good or not, and give this feedback: this is the “typical” data labeling. But this is the less frequent type of work. But then there are more involved AI training tasks, which looks like this:

• Come up with a task that the AI should do

• Then write the tests that confirm the result

• Package all of this up into a Docker container, using Harbor framework

• Then read the code that the AI writes - often doing this based on feedback from several models, and give it feedback

This work is not easy to do — and you can see why you need good software engineers to do it! — but it gets repetitive really quickly. Most engineers who I talked to said that they found it hard to do this work motivated day-in, day-out. Then again, I did talk with an engineer inside the org who said that by varying the technologies they use, and challenging themselves, they found this work motivating and interesting! This engineer expects there to be more software engineering work coming up, after the phase of training passes. There’s a lesson here: when life gives you lemons, you can either complain about it, or you can make lemonade, like this dev has done so.

This training work is secretive across the AI industry, and vendors are offering $100+ per hour for devs to do this type of work. There are rumors across the industry that OpenAI and Anthropic spends more on these training environments (to make the models great at coding) than the model training run itself!

Infrastructure and security teams were hit especially hard by reassignments. I talked with several engineers in infra orgs, who had 30-50% of their teams drafted into the ADO org. And in some cases, it was the best engineers who left.

One engineer told me that the whole situation feels like the movie, The Hunger Games, when tributes are randomly selected and then removed from their environment, to something completely different. Except, at Meta, many more folks are being affected, with between three and five from a 10-person team going from building products used by hundreds of millions, to giving human feedback on AI-generated GitHub repos, over and over. So, a wider impact than in the Hunger Games, but with less drastic consequences.

Around 6,500 people are in the ADO org, more than at OpenAI and Anthropic. Roughly four to five thousand of these are software engineers. Meta has around 25,000 engineers, meaning that one in every 5-6 software engineers may now find themselves doing data labeling full time.

As you can imagine, people are actively open to new positions, and nobody is updating their job title on LinkedIn and elsewhere to “data labeling at Meta.”

I’ve spoken with people in this role and they don’t like doing it, and feel upset about the top-down decision making. The silver lining is that they still have a job, have retained their salary, and were not part of layoffs. They still have time to leave Meta for something that pays comparably and is not a data labeling job.

3. Core engineering folks feel treated like trash

Problem #3: a month-long waiting game, stoking fear across the company. On 20 April, Reuters reported that Meta planned to lay off 10% of staff in a month’s time, and Meta confirmed the news, meaning there was a period of four weeks when everyone knew that they could be unemployed very soon.

Forced reassignments to data labeling started to happen. As I covered at the time:

“Understandably, there are mixed feelings about this redeployment [to data labeling], with layoffs coming soon. On Wednesday, 20 May, Meta will announce layoffs. Perhaps those moved to do data labeling could actually be “safer” than colleagues on product teams. Of course, this is speculation, but it would be cruel if Meta cut devs reassigned to data labeling.”

Problem #4: Performance review is hyper-aggressive at Meta, so devs optimize all metrics. The internal performance review process, PSC (Performance Summary Cycle), is very stringent, compared to Google and Apple, I’ve learned. Managers inside Meta “fight” over the pay packets of their employees, which involves “knocking down” the packet of engineers on other teams, so their direct reports are ranked higher. It’s common to weaponize metrics in this process – be that business impact, the number of code reviews, number of lines of code written, pre-AI (see our Coding Machine archetype podcast on this.)

Quotas are handed down to managers for the splits of the workforce to be put in each ‘bucket’, and the internal politics gets heated as managers try to get their reports into higher buckets.

After a few years, engineers at Meta learn that the best way to not get a bad PSC rating is to have all metrics – impact, code committed, and other numbers – higher than their peers’ are. Learn more about the internal politics of performance calibrations.

Problem #5: tokens are measured as part of perf, so devs aggressively optimize for it. When layoffs were confirmed, engineers also learned that managers shall inspect token count during perf reviews. This raised worries that those with low token counts might be marked as underperformers and dismissed.

So, what is the natural reaction to this as an engineer at Meta? They started using AI tools for the sake of generating more tokens. This happened while Meta had an internal token leaderboard, encouraging tokenmaxxing. As I wrote on 16 April:

“As per The Information, Meta employees used a total of 60.2 trillion AI tokens (!!) in 30 days. If this was charged at Anthropic’s API prices, it would cost $900M. Of course, Meta is likely purchasing tokens at a discount, but that could still come in at $100M+ – in large part from senseless “tokenmaxxing”.”

The biggest problem: people stop caring about real work and focus on performative work. Let’s check the four ingredients that Meta’s leadership has decided to introduce to their workplace:

1. Tracking the keyboards and mouse clicks of all engineers, where legally possible

2. Reassign a good chunk of engineers to fulltime data labeling

3. Let staff know that 10% of them will be laid off

4. Have a culture where devs optimize for any and all metrics measured during PSC

5. Measure token usage as part of PSC

Shake this mix up well, and what do you get? Two things:

1. Everyone overuses AI to boost their personal stats. An engineering workforce that pretends to work with as much AI, and as little human input, as possible. It’s a strange incentive where an outage caused by a failure to review code properly is not grounds for dismissal, but writing code by hand – instead of having an AI agent write it – could cost you your job

2. Every longer-tenured engineer is seeking a new job, or at least considering it. Those who have been around at Meta longer term have seen enough. Let me describe this visually:

Fresh data seems to confirm that starting in May, a lot more engineers at Meta are looking for an “out.” Here’s how signups to leading Big Tech interview preparation service interviewing.io from Meta have changed over the last year and a half: we can see a massive jump in May vs the year before, as shared by founder and CEO Aliner Lerner with us:

(We covered more data from interviewing.io and other interesting data sources in the deepdive State of the software engineering job market in 2026)

To its credit, Meta has given out generous retention equity packages to several engineers considered key on the remaining teams. These packages make it harder to get matching compensation elsewhere. Still, I talked to one engineer who got an equity top-up, and said that this approach helped him decide to leave as soon as possible because he feels bitter about the lack of autonomy and having no control over things.

4. Most embarrassing-ever outage

Meta’s core infra and security teams have suddenly found themselves severely understaffed. Most folks are pushing AI-generated code merged with AI-only reviews, without paying much attention to quality. After all, they’re dealing with the possibility of unemployment, while firefighting to operate a team without its best engineers whose headcount has been cut in half, all with the knowledge that AI usage could affect their own job security.

Two weeks ago, on 30 May, the most embarrassing outage in Meta’s history happened. Here’s software engineer Siddharth Sundharam’s summary (emphasis mine):

“Yesterday, a slew of Instagram accounts, including some high profile ones like the Obama White House account, seemingly got hacked.

Look, I’m no spring chicken. I’ve spent almost a decade and a half identifying vulnerabilities and exploits at unicorn scale, but this is hands down the most unserious, “almost too stupid to be true” of them all.

The Takeover Flow:

Step 01: Faking the Location & Initiating Support. All the attacker needs to kick this off is your account username. Then, they hop on a VPN or proxy close to your city so Instagram’s security algorithms don’t suspect a thing. (You can quite easily get this from your public profile or “About” section or a hundred other ways.) Once it looks like the request is coming from the correct region, they tell the Meta support AI that the account is hacked and ask it to send the verification codes to an arbitrary email address they control.

Step 02: That’s It. Really, that’s it.

The first proper zero auth password reset I’ve seen in production. There appears to be no additional check as to whether the email being given is actually something the user has used before. Once the AI sends the security code to the attacker’s email, the attacker passes it right back to complete the verification. The platform hands over a fresh password reset link, granting full ownership to the attacker.”

This is a security breach in which Meta left its extra-secure, reinforced front door unlocked, so that anyone could come in, and there was no alarm to notify anybody when it happened! It seemed Meta only noticed when users started reporting it on social media!

From talking with folks inside Meta, I’ve learned that AI was at the heart of this outage. AI-generated, AI-reviewed code, and security teams being gutted were together the cause of this beyond-embarrassing incident. I poked around, and here’s what I gathered:

• Instagram’s Trust and Safety Team lost around 50% of its staff to data labeling and layoffs. Some of the most senior folks were drafted onto AI training tasks.

• AI-generated changes that saw no human input, just another AI code review, were very common during the last two months, across the codebase. The change that caused this outage looked like one of these

• Normally, the Trust and Safety team would be on top of monitoring and alerting of security breaches, but it is currently in full disarray due to rapid, internal disorganization.

Meta’s Chief Security Officer resigned the very next day. The outage was resolved on Monday, 1 June, and an investigation started as part of the SEV process. On Tuesday, Meta’s Chief Information and Security Officer (CISO), Guy Rosen, announced his departure.

Coincidence? I suspect not: the CISO might have stepped down if they warned against the Security org being gutted but were then ignored, and so no longer trusts leadership. I also imagine the CISO didn’t have the idea to move a good half of Instagram’s security team over to data labeling.

As a note, the previously worst outage was all Meta services going down for seven hours in 2021, due to a DNS / BGP configuration issue. It was a bad outage, but Meta handled the follow up well, in my opinion. After that 2021 outage, Meta shared a postmortem and apology. It has not done so for the latest Instagram account takeover outage.

5. Internal mess

Wired shares more details on just how bad the situation is inside Meta, right now:

“Someone interrupted a livestreamed, employee-only presentation at Meta earlier this week with an expletive-filled outburst about “being the company’s bitch,” according to a recording heard by WIRED. The individual then asked the people leading the call to write to a specific Meta AI executive and “tell him that he’s a piece of shit.”

The incident, which took place on a call open to thousands of employees, reflects growing frustration inside the company’s Applied AI team, which was formed in March to support the work of AI researchers at Meta Superintelligence Labs. Three current employees tell WIRED there is widespread dissatisfaction with how Meta assembled the unit of about 6,500 engineers and product managers and the drudgework they allege they have been assigned to improve AI models.

“It’s literally the gulag,” one of the employees claims. “You have zero purpose in life all of a sudden, you barely interact with anyone, you just have these tasks every week.”

There’s more: Meta’s Chief Product Officer, Chris Cox, reportedly admitted to staff that Meta’s upper leadership (the folks above him, meaning the C-level at Meta) created the mess. Also from Wired:

During a meeting this week open to all employees at Instagram, Meta chief product officer Chris Cox addressed the “difficult” and “brutal” environment created by the “insanity of this company” in the past few months, according to a recording heard by WIRED. Cox applauded Instagram employees for launching features and serving around 2 billion users amid what he compared to “running a marathon in the middle of a hailstorm and then, like, your teammate gets replaced and then we’re recording you.”

“It’s like what the fuck,” he said, drawing laughs, before repeating himself. “It is like what the fuck.”

6. Self-inflicted wounds

So, is there an ultimate source of the “insanity of this company”, as CPO Chris Cox put it? Engineers whom I talked to point the finger at two individuals: Mark Zuckerberg and Alexandr Wang. Zuckerberg has full control over the business, and has made the decisions to reallocate a good part of engineering folks to data labeling, to roll out tracking software, and to lay off 10% of staff when Meta achieved record revenue and profits. As the CEO, the buck clearly stops with him.

But it’s hard to unsee that – outside of layoffs – everything that Meta is doing is taken from the Scale AI playbook, and that surely comes from Wang:

• Mandatory keystroke and mouse tracking to generate training data

• Forced data labeling with 4,500+ engineers is to generate high-quality RLHF, surely for Meta’s under-construction coding LLM

• Taking away the best engineers from the heart of the business is surely signed off by Mark Zuckerberg, believing that it is more important for Meta to train a coding AI than it is to operate its core business like Instagram, Facebook or Messenger reliably. Oh, did I mention that on Saturday (12 June) Facebook and Instagram had another SEV0, that is, a full-on outage?

Before all this happened, Meta was on track to overtake Google as the world’s #1 ads business by the end of the year. But for some reason, Mark Zuckerberg decided that building a coding LLM is more important.

Meta’s leadership is now trying to undo all the damage they have done. Wired reports that Meta CTO, Andrew Bosworth, admitted to staff that the AI reorg was atrocious and committed to better communication in the future.

To me, it looks obvious that Zuckerberg doesn’t care how engineers feel about the massive changes, and that Bosworth likely ignored the chaos, all while engineers know for a fact that the next AI model matters more than they do to the business. Bosworth also said that employees will have access to AI coaching tools. Very considerate, given the situation!

Based on all I’ve learned, Meta’s engineering culture is dead because leadership has made it clear that engineering at the company is a cost center.

Needless to say, I hope my assessment is way off, but I’ve seen nothing yet from Mark Zuckerberg and Alexandr Wang – the two executives creating this current mess – to suggest it is. There may be a short time period where, if major changes like data labeling assignments and staff tracking are undone, then things at Meta could return to normal. The longer the current conditions persist, the more tenured engineers will surely leave.

7. Is “AI psychosis” just a Meta issue?

It’s tragic to see a technical founder at Meta so focused on AI that he neglects the engineers who built the heart of his company. But is Meta a one-off exception?

Mitchell Hashimoto (creator of Ghostty, founder of HashiCorp) says he is seeing similar behavior by other founders (emphasis mine:)

“I strongly believe there are entire companies right now under heavy “AI psychosis” and it’s impossible to have rational conversations about it with them. I can’t name any specific people because they include personal friends I deeply respect, but I worry about how this plays out.

I lived through the great MTBF vs MTTR (mean-time-between-failure vs. mean-time-to-recovery) reckoning of infrastructure during the transition to cloud and cloud automation. All those arguments are rearing their ugly heads again but now it’s... the whole software development industry (maybe the whole world, really).

It’s frightening, because ‘psychosis folks’ operate under an almost absolute “MTTR is all you need” mentality: “it’s fine to ship bugs because the agents will fix them so quickly and at a scale humans can’t do!” We learned in infrastructure that MTTR is great but you can’t yeet resilient systems entirely.

The main issue is I don’t even know how to bring this up to people I know personally, because bringing this topic up leads to immediate dismissals like “no no, it has full test coverage”, or “bug reports are going down” or something, which just don’t paint the whole picture.

We already learned this lesson once in infrastructure: you can automate yourself into a very resilient catastrophe machine. Systems can appear healthy by local metrics while globally becoming incomprehensible. Bug reports can go down while latent risk explodes. Test coverage can rise while semantic understanding falls. Changes happen so fast that nobody notices the underlying architecture decaying.

I worry.”

The takeover outage at Instagram was exactly like this: the engineering team dropped the quality bar for AI-generated and AI-reviewed code, probably expecting that they could recover quickly from failures. And they did indeed recover… after the damage was done, high-profile Instagram accounts were hacked, and the system was compromised, all very publicly.

Mitchell highlights the specific concern of founders over-estimating the capabilities of AI, and consequently casting aside sensible safeguards when shipping code to production.

Takeaways

Most of us probably have something to learn from the disastrous events at Meta caused by hyper-focus on AI to the exclusion of people who are the lifeblood of that company. In some good news, I’m hearing that in the UK, some of the 10% layoffs have suddenly been cancelled: at the end of the mandatory consultation period, several infra and security teams are learning that no engineers on their team will be let go, as originally expected.

Meta has a booming business, and is already a beneficiary of AI via increased ads revenue. Meanwhile, my Facebook feed is filled with fake, AI-generated videos, with hundreds of comments from bots and people who seemingly don’t realize it’s AI. It all seems like just more content for Meta to show ads next to.

And yet, despite business booming, Meta’s leadership has gone on a crusade to inflict the most damage possible on its engineering org. Apparently, they’re now learning that most of it was pointless.

If you’re in a leadership position and feeling the temptation to make drastic org changes for AI-related reasons, take a deep breath and see where it left Meta. Meanwhile, If you’re an engineer at a company whose leadership is over-indexing on AI, consider forwarding this article as additional context.

If you’re hiring standout engineers who are extremely hands-on with AI, then it’s never been easier to get talent from Meta, than right now. Every engineer I know at the company is an extremely early adopter of AI, and knows how to build products and AI infra. These folks have soured on the company and its leadership. Meta’s loss of talent will be the gain of other startups and the rest of Big Tech; it’s one benefit of AI that’s probably a bit unexpected – not least of all by Meta!

It seems like the old mantra of “move fast and break things” has now extended to Meta’s engineering org itself, with the company’s rush to over-invest in AI, so it will avoid missing the latest mega-trend in the tech industry.

First of all, a big thank you to partner teams for sharing exclusive details for this deepdive:

• Interviewing.io: anonymous mock interviews with engineers from top companies. Thanks to founder and CEO, Aline Lerner.

• Workforce.ai, built by Live Data Technologies, which monitors 1M+ job changes and 300M+ employment validations monthly, across companies, roles, levels, functions, industries, and locations. Special thanks to Alex Hamilton for his input.

• SignalFire: a VC firm with a standout data analysis team. Ordinarily, their data is used to give their portfolio companies a major commercial advantage, but they made an exception to share some for this article. Special thanks to Asher Bantock.

• TrueUp: a platform that scans every open job in Big Tech, top startups, and scaleups, typically paying in the top two tiers of the trimodal software engineering compensation model. Thanks, Amit Taylor.

Today, we cover:

1. Top AI labs are now more attractive than Big Tech. Anthropic is most in demand among job candidates for interview preparation services. Along with OpenAI, it’s almost certainly the place with the most competition for jobs in tech.

2. Harder for new grads & interns to get hired. Data shows that intern intakes have fallen, even as software engineering recruitment recovers. Large tech companies take on half as many interns as before, and candidates’ work and educational backgrounds matter more than ever.

3. Mobile and frontend demand drops, AI & FDE surges. Frontend engineer titles are disappearing fastest across the industry, followed by native iOS and Android ones.

4. AI engineering comp > software engineering comp. AI engineers are more in demand than software engineers, and get higher compensation offers, especially with equity. At the 80th percentile in the US, $300K+ base salaries are the norm now for senior engineers.

5. Management’s “great flattening” continues. There are fewer engineering managers for each engineer across the industry, and fewer VP of engineering and director of engineering posts at Big Tech.

6. Big Tech seniority & tenure keep rising. Since the end of zero interest rates in 2023, it’s as if there’s fewer ways to tempt Big Tech workers to switch jobs, so they remain in situ.

7. Interview preparation signups: what do they indicate? Companies doing mass layoffs tend to see the biggest surge in devs signing up to practice interviews. A list of the top 20 companies from where engineers are preparing to interview elsewhere.

8. Where engineers go after Big Tech. From Amazon, they go pretty much everywhere. From Google, Apple & Meta, it’s mostly to AI labs. Microsoft is where the most ex-workers become their own bosses by working for themselves next.

As a reminder, in Part 1, we covered:

• Software engineering recruitment: trending up, mostly

• Big Tech and publicly-traded companies

• Who’s hiring the most software engineers?

• AI engineering: explosive demand

• Who’s hiring the most AI engineers?

• Is AI engineering replacing software engineering hiring?

Let’s get into the latest data:

1. Top AI labs now more attractive than Big Tech

In Part 1 of this mini-series, we cover the exploding demand for AI engineering:

AI engineering job openings have increased 60% in the past year at top companies, while software engineering openings grew by 7% in the same places. We also found that Big Tech is significantly growing AI engineering headcount:

Anthropic: most in demand

New data suggests that the two biggest AI labs are attracting the most candidates to apply for their AI engineering roles, which is pretty predictable.

Interviewing.io is a job interview preparation service which offers coaching for clients who are getting ready for interviews at specific companies. Based on the number of mentions by clients, Anthropic is the one most candidates are preparing for with paid coaching, and it’s not even close:

It’s also notable that OpenAI (16% of candidates) gets around the same share as Google (17%) and other large tech companies (17%). Combined, Anthropic and OpenAI account for 51% of all interviewing.io coaching requests. For context, interviewing.io only added coaching for frontier labs this year!

There are a few potential causes of the surge of interest in Anthropic:

• OpenAI replaces Anthropic as AI supplier for the US’s novel ‘Department of War’. In early March, the US Government controversially declared Anthropic a “supply chain risk”, and appointed OpenAI as its AI supplier, after Anthropic raised concerns about the future use of AI in mass surveillance and fully autonomous weapons. This raised suspicions that OpenAI agreed to cross ‘red lines’.

• Anthropic’s market dominance continues. Claude Code is the most popular developer tool, as found by our AI tooling survey in February. It seems little has changed.

• Anthropic’s value exceeds OpenAI’s. In March, Anthropic raised a $65B funding round at a $965B valuation, making it more valuable than OpenAI for the first time.

• Anthropic files to go public first. Last week, Anthropic filed to go public, beating OpenAI which has done so a week later.

Anthropic also recruited the most in-demand AI researcher, Andrej Karpathy, in May. My sense is that between the two labs, Anthropic has more momentum for the time being, and has perhaps acquired a ‘halo effect’ with its seemingly principled stance. It’s not surprising that it’s attracting more candidates.

Where are AI labs hiring from?

We looked into the sources of recruits to the three most in-demand AI labs: Anthropic, OpenAI, and Google DeepMind. Here’s what we found:

Where Anthropic hires from, in order of popularity:

1. Google (often Google DeepMind)

2. Meta

3. Stripe

4. Microsoft

5. Amazon (AWS)

6. Databricks

OpenAI:

1. Google

2. Meta

3. Apple

4. Stripe

5. Statsig (after OpenAI acquired Statsig)

6. Microsoft

7. Amazon (mostly AWS)

8. Databricks

9. Airbnb

10. NVIDIA

Google DeepMind:

1. Internal transfer

2. Meta

3. Microsoft

4. Amazon

5. Windsurf

Anthropic has the highest retention rate of all AI labs. Data from SignalFire found the 2-year retention rate (percentage of employees who stay 2 years) is:

• OpenAI: 67%. This is consistent with the rest of Big Tech

• Google DeepMind: 78%. Well above the rest of Big Tech

• Anthropic: 80%. Standout, industry-wide!

Consistent with SignalFire’s 2025 finding, OpenAI 2-year retention was 67% (FAANG-level) versus Anthropic (80%), and DeepMind (78%).

2. Harder for graduates & interns to get hired

It’s well known that it’s getting harder to be hired as an early-career engineer, and new data underlines this.

Intern intakes down since 2022

Live Data Technologies looked at software engineer vs engineering intern hiring trends at 30-80 US-based tech companies, pinned to 2019 hiring numbers (100% being that year’s total number of hires). The spread is wide because Live Data Technologies selects the top few dozen companies that meet their criteria for a “large public tech company” in their database.

The findings:

Zooming into intern hiring, here’s a visualization of it as a percentage of all appointments:

Alex Hamilton, analyst at Live Data Technologies, says:

“We’ve seen overall software engineering hiring start to come back since the 2023 tough market. However, intern intake just kept falling alongside it, which isn’t what you’d normally expect.

Historically, intern programmes have tended to bounce back pretty quickly once companies start hiring again. That hasn’t happened this time, and 2024 and 2025 are the first years in the series where the two lines are moving in opposite directions.

Where you do see companies holding intern intake steady or growing it, it’s almost always a reflection of where they are as a business, be that earlier-stage or faster-growing companies, rather than any kind of broader market recovery”.

Graduate jobs trending down

Anecdotally, we hear new grads continue to have a hard time finding a position. Our new recruitment data on major US tech companies confirms it:

“New grads” in this data are software engineers who graduated less than a year before getting a job as a software engineer. In 2025, just one in 10 engineering hires at larger companies were recent grads, down from nearly three in 10 in 2023.

Pedigree matters more for new grads

We looked closely at the places from where new graduate software engineers are joining US-based tech companies, and found the share of successful candidates from “elite” universities is growing:

By “elite” universities, we mean one of the top 20 US colleges for computer science, such as MIT, Stanford, Carnegie Mellon, UC Berkeley, Harvard, Caltech, Georgia Tech, and Cornell.

Obviously, the influence of these places’ reputations is not a new thing, it’s what makes them “elite” universities, after all. But with new grad hiring down across tech, even graduates from these universities can expect fewer opportunities than before.

Even so, the pedigree that comes from graduating from a well-known university, or doing an internship at a well-known company, becomes ever more significant as the job market tightens.

3. Mobile and frontend demand drops, AI & FDE surges

Here’s interesting data showing the shifting prevalence of job titles on sites like LinkedIn over time:

Some takeaways:

• AI engineering’s on fire. This is not surprising and is evident throughout our study.

• Forward Deployed Engineers (FDE) are growing rapidly. We covered the sudden demand for FDEs in 2025, and this year we’re seeing the FDE role heat up again.

• Modest increase in sales engineers: Sales engineers help close large, B2B-type, deals, and are typical at companies selling to enterprises. The rise in prevalence of this position suggests more companies are targeting enterprise-scale clients. Also, my sense is that FDEs can operate like sales engineers.

• There are fewer native mobile engineers. In 2022, I observed a drop in demand for native iOS and Android engineers. Cross-platform frameworks being more capable today may contribute to fewer places investing in native applications, and a fall in demand for this discipline overall. Is the “golden age” of native mobile development over, with its standalone native iOS, native Android, and web teams for a single product?

• Frontend-only engineers are disappearing. This is one of the most interesting trends in the data. I’ve observed full-stack engineers become the norm at many places, who can do both frontend and backend development. Especially with AI, there is no reason a proficient frontend engineer should not work on backend as well, so, I expect “pure” frontend engineers will be employed only in larger companies, where demand exists for things like building a design system. We cover more on this topic in the deepdive, Design systems for software engineers.

4. AI engineering comp > software engineering comp

One poorly-kept secret in tech is that although software engineering compensation is very good at Big Tech and top startups, it’s superior for AI engineering jobs at the same places – and even better at leading AI labs:

Read more

I’m in Budapest, Hungary, this week, for Craft Conference, where I’ll be giving a keynote presentation alongside other speakers, including software engineering legend Kent Beck, who’s been on the podcast, Hillel Wayne, a formal methods expert and the author of ‘Logic for Programmers’, and Titus Winters, lead author of Software Engineering at Google.

The title of my keynote is “Slow down to speed up”, and I’ve been thinking about this topic a lot recently. Here are some things I’ve been seeing that I feel are relevant…

AI coding tools now used by pretty much all software engineers – that’s fast!

Read more

A year ago, most engineers found it harder to get responses to job applications, while hiring managers found it harder to hire than before. We’ve now spent two months gathering fresh, never-before-shared data from a range of sources to find out if that paradoxical situation still exists or not.

We collected the data with software engineer Jessica Salmon – who’s joined the team as a tech industry analyst. A big thank you to partner teams who shared exclusive details for this deepdive:

• TrueUp: a platform that scans every open job in Big Tech, top startups, and scaleups, typically paying in the top two tiers of the trimodal software engineering compensation model. Thanks, Amit Taylor.

• Workforce.ai, built by Live Data Technologies, which monitors 1M+ job changes and 300M+ employment validations each month across companies, roles, levels, functions, industries, and locations. Special thanks to Alex Hamilton for his input.

Today, we cover:

1. Software engineering recruitment: trending up, mostly. More software engineering job postings in the UK and the US, and a decline in Germany and France. “Top” tech companies are hiring 20% more vs a year ago.

2. Big Tech and publicly-traded companies. Meta went on a monster hiring spree for the last two years until layoffs, last week. Microsoft and Amazon were flat, while Google and Apple consistently hired devs. Stripe, Shopify, and Atlassian all hired more than Big Tech.

3. Who’s hiring the most software engineers? Apple, Amazon, and IBM are the top 3 by number of positions listed, while Meta dropped off the top 20 with layoffs there. The fastest-growing companies seem to be in fintech, observability, and security.

4. AI engineering: explosive demand. Most tech companies seem to be prioritizing AI engineering recruitment over software engineers. Observability companies appear to be hiring fastest.

5. Who’s hiring the most AI engineers? Apple, Google, and TikTok have the most openings, and many larger tech companies have 50-100% more AI engineering job listings than a year ago.

6. Is AI engineering replacing software engineering hiring? The data doesn’t say, but are the basics of AI engineering becoming a baseline skill for software engineering job candidates?

See Part 2 for details on:

• Top AI labs are now more attractive than Big Tech

• Harder for new grads & interns to get hired

• Mobile and frontend demand drops, AI & FDE surges

• AI engineering comp > software engineering comp

• Management’s “great flattening” continues

• Big Tech seniority & tenure keep rising

• Interview preparation signups: what do they indicate?

• Where engineers go after Big Tech

For more on the tech jobs market of 2025, check out last year’s deepdive.

1. Software engineering recruitment: trending up, mostly

Top tech companies hiring more, slowly but surely

TrueUp tracks open positions at top-paying companies, which are usually US-based. Among this group, recruitment of software engineers has been steadily increasing since March 2023:

US growth, but less in Canada and Europe

Let’s take a look at broader industry trends via software developer jobs listed on Indeed.com, the world’s largest job aggregator site. As a “jobs aggregator”, Indeed not only displays paid-for ads but also crawls many other career sites and lists the vacancies.

Looking at trends over the last five years, the number of software engineering jobs listed is still lower than in May 2021:

Obviously, 2021 was the pandemic which created a commercial boom for the tech industry, so it’s not exactly an ideal comparison. Zooming into the last twelve months shows encouraging growth signs:

Outside of the US, let’s check out software engineering vacancy trends, per Indeed:

In this list, the US and the UK are the only two countries where vacancies are up; Canada is flat, while Germany and France have seen declines. To me, it suggests US-headquartered companies are hiring more devs, mostly in the US and some in the UK, whereas European-headquartered companies are more cautious about recruitment.

The software engineering industry still grows

Let’s ask how much the software engineer population has grown or shrunk in the last seven years. Here are some numbers from profiles, as tracked by Workforce.ai:

Let’s zoom into this chart:

There are a few interesting details:

• Hiring mostly occurs between the months of March and June, and there’s very little net growth in the second half of the year. This makes sense as tech companies set annual headcount budgets at the beginning of the year and commence hiring from then, with most hires evidently made in the following months. Hiring budgets are usually spent by mid-year.

• The industry seemingly shrank in 2023. That year, it was only in Q2 that more software engineers were hired than left their jobs. It was the worst period for software engineers in two decades.

…and also in 2024 and 2025 during Q3. Mass layoffs mostly happened between August and October, the charts show.

2. Big Tech and publicly-traded companies

Big Tech: modest jobs growth – but not at Meta

Looking at the two years since May 2024, of the five largest tech companies, only Meta grew headcount aggressively:

In that time, Meta increased its headcount by nearly 20%, so perhaps its recent 10% layoffs could be considered as not a total surprise. As for other tech giants, their software engineering headcount also changed in the last two years:

• Apple: +10%

• Google: +5%

• Microsoft: -1.1%

• Amazon: -1.3%

Visualizing this:

Apple and Google seem the most “stable” places when it comes to growing the software engineering function; Apple has not had mass layoffs for decades. In contrast, Meta is the most volatile place to work; hiring devs quickly and then letting them go. Elsewhere, Amazon and Microsoft are slowly employing more software folks.

Meta’s hire-then-fire rollercoaster

Meta warrants a closer look, where morale is reportedly as low as it’s ever been right now, due to layoffs of 10% of staff, while thousands of software engineers have been reassigned to manual data labeling from previous product development or infra work. From 2004 until 2022, Meta never did mass layoffs, but in the past three years there have been three sizeable redundancy rounds (at the end of 2022, early 2024, and last week), with the most rapid hiring growth of any tech giant sandwiched in between:

Before 2022, Meta ranked with Google as one of the most stable places to work in tech. Safe to say, the company is currently the riskiest place to be in job-security terms – even if Mark Zuckerberg has said there will be no more mass layoffs this year.

Publicly-traded tech companies: mostly flat recruitment

A couple of years ago, larger tech companies were hiring even more than most of Big Tech. Software engineering headcount grew in the past two years at Shopify (+36%), Stripe (+29%), Atlassian (+23%), Snap (+15%) and Spotify (+6%).

But that’s changed recently. In the past year, headcount growth has slowed at Stripe (+15%), Atlassian (+11%), Shopify (+6%), Snap (+2%), and Spotify (+1%),

This data does not capture Atlassian letting go of 10% of staff this month, and Snap laying off 16% in April.

Among this group, Stripe stands out as a place that keeps hiring at a higher pace than nearly all of Big Tech. Atlassian and Shopify have grown their software engineering headcount faster than Google, Microsoft, and Amazon did in the past two years.

3. Who’s hiring the most software engineers?

So, which companies are hiring the most? Let’s look at this via open positions and growth comparisons.

Companies with the most open roles

“Top” tech companies with the largest number of software engineering openings:

Compared to last year:

Observations:

• The top three are unchanged: It’s still Apple, IBM, and Amazon

• New entrants to the top 20: Accenture (consulting), Tesla, Cadence (hardware), Hewlett Packard Enterprise (HPE), and SpaceX.

• Dropouts: Meta and Oracle. Last year, Oracle was #4 on the list with 1,396 open software engineering roles, but the company announced up to 30,000 layoffs in March. Also no longer leading in jobs available are Speechify, NTT Data, Meta, and DXC Technology.

• Big Tech companies hiring more than last year: Apple and Google, which has +62% more engineering roles advertised than last year.

• Big companies hiring less vs last year: Meta, Oracle, and TikTok all have much fewer openings listed. For Meta and Oracle, there’s little surprise given recent layoffs. TikTok was in US regulatory limbo until January, when a new TikTok USDS joint venture was established.

• Hardware companies hiring more software talent: Micron, Qualcomm, and AMD all have considerably more software engineering openings than last year.

Where are the “top” software engineering jobs?

Most of these jobs are in the US, just as most companies leading in recruitment numbers are US-headquartered. Other leading locations for jobs are India, the UK, and Canada:

Fastest-growing: fintech and security

Then there are large companies where recruitment has been among the fastest anywhere in software engineering:

Software engineering headcount growth at some companies was significant over the past two years:

• Ramp (fintech): +94%

• Wiz (security): +84%

• Datadog (observability): +68%

• Rippling (fintech): +55%

• Figma (design tooling): +41%

• Netflix (streaming): +37%

Of this group, Datadog, Figma, and Netflix are publicly traded, and the largest by headcount:

• Datadog: AI agents are driving demand for observability through the roof, and Datadog is the leader of the pack. There’s demand to expand their infra and offerings. AI companies are also customers: OpenAI famously – or infamously! – spent around $170M on Datadog in 2025.

• Figma: the design tooling company went public in 2025 and is heavily investing in products related to AI, such as Figma AI.

• Netflix: the streaming behemoth is transitioning from “just” being a video streamer to becoming a broader entertainment platform, doing live events, advertising, gaming, and more. We did a podcast with Netflix CTO, Elizabeth Stone, in 2025 about Netflix’s engineering culture.

4. AI engineering: explosive demand

AI engineering demand is only going in one direction: upwards, at a fierce tick. Last September, our previous State of the Tech Job Market report found that AI Engineering was the hottest segment in tech for job listings growth. Today, the trend seems to continue:

Read more

1. Tradeoffs of AI tooling. Positive and negative sentiments, like how AI tools often mean less time spent on tedious, repetitive work, but that it often leads to unrealistic business expectations.

2. Adopting AI at scale is hard. Companies are struggling to achieve adoption that feels productive at the team and org levels. Our survey finds the benefits of AI heavily depend on the engineering culture that was in place before.

3. Impact on codebase quality. Codebase quality seems to be decreasing, but management at most places does not care. Meanwhile, maintenance duty is falling upon a shrinking number of engineers who still understand increasingly complex codebases.

4. Less experienced engineers and AI. For these folks, AI seems to be less helpful and they rack up higher AI token bills. Maybe they need more space to learn, mentorship, and support.

5. AI tooling ‘addiction’. Using AI agents “feels like a slot machine” encouraging “just one more prompt”-type behavior, while some folks think that the pricing of plans is built in a way to “lure” them to prompt more and more.

6. Changes since 2024. Fewer devs are negative about AI, but there’s not all that much more positivity. The models have become much higher quality, and better tooling improves trust.

7. Where are we headed? The concept of code ownership seems to be eroding, and collaboration within teams becomes less important.

Previously in this series, we covered:

• AI tooling for software engineers in 2026: a detailed summary of survey responses, covering the most-used AI tools, trends, AI agent usage, company size and usage, and tools engineers love.

• The impact of AI on software engineers in 2026: key trends. Part 1. Concerns about mounting AI costs, more engineers hitting usage limits, and AI tools having uneven effects upon different types of engineers

Full subscribers can access a more detailed report.

1. AI tooling tradeoffs

What can be said about the impact of AI tooling on engineers this year? Well, based on the responses that readers sent in to our survey, there are some common patterns:

Code and output tradeoffs

Productivity tradeoffs

Research published two weeks ago (5 May) by Microsoft claims AI expands the pool of people who can do high-value work. Our survey found similar, while MS’s findings are based on Microsoft 365 Copilot chat usage.

Obviously, Microsoft has an interest in showing the benefits of its AI tools, but in this specific area, it’s telling that both studies agree AI does allow devs and non-devs to expand the type of work they do.

Software engineering tradeoffs

With AI tools, their ‘mileage may vary’ among individuals. For example, some respondents told us they can spend much more time in a “flow state” thanks to AI tools, as they don’t have to wait for input from peers, can keep unblocking themselves, and have fewer interruptions.

But others say the opposite: that because they can start so many more tasks in parallel with AI tools, they keep context switching which knocks them out of a flow state!

The positives and negatives of these tools seem dependent on the environment respondents work in, individuals’ personality traits, and where users are on the AI learning curve.

2. Adopting AI at scale is hard

One pattern in the responses is that adopting AI at company-scale remains challenging, including:

• Costs: a growing concern covered in depth in Part 1

• Usage: getting people to use AI tools continuously is not always straightforward

• Onboarding and education: at larger companies, there needs to be support to help devs make the most of the tools

• Reviewing AI-generated output: code review is a particular pain point

• Integrating with internal systems: AI tools are more helpful when they are seamlessly integrated with internal systems, hence why many larger companies use in-house, deeply embedded coding agents

Adopting AI at company-level doesn’t lead to a magical fix for engineering problems, while the benefits of AI in an organizational sense seem to depend on what was in place before.

Case in point, AI seems to amplify pre-existing engineering culture. AI doesn’t change the underlying quality of an organization’s engineering culture. Teams with strong engineering practices get more positive benefits out of AI than those without. Teams that see benefits from AI tools already had:

• Guardrails: testing and automation around the codebase and deployments

• Documentation: they recorded their architectural decisions and engineering practices

• A quality codebase: AI agents will replicate patterns already in a codebase

A few quotes on this from respondents to our survey:

“AI is an amplifier, not a fixer. Good software engineering practices get multiplied. So do the bad ones. Embedding this properly in teams is exciting and important”. – Staff+ engineer at a large company in Europe

“I feel like AI allows both faster prototyping and increased velocity on iterations to production software; it relies on existing best practices / project templates our team already have”. – Solutions Architect at a small company in the US

A workflow that makes one dev “10x” more productive may not work for another. This is another reason why rolling out AI tools doesn’t seem to magically make everyone more productive. A senior engineer working at a large company in Canada told us:

“It feels like AI workflows are very idiosyncratic in that some people derive (I hate this framing, but…) 10x more productivity benefit from them than other apparently equally clever, educated, and diligent developers. It feels like finding a workflow that clicks with your own habits and heuristics is more important than finding a global optimum for everyone”.

AI amplifies individuals differently, so the team impact is messy to figure out. A US-based principal engineer at a large company reflected on feeling disconnected from colleagues because of how they use AI differently:

“I use AI in what I think is probably a more sophisticated way than most of my colleagues, so there can be a disconnect between my work and theirs, which is not good news because I am “The Principal” on the team”.

“The tool that works for you” approach can lead to tooling chaos, even at a team level. While it’s empowering to allow devs to choose the AI tools they feel are the most helpful, over time it becomes chaotic when teams can choose their own tech stacks, and when at large companies there are dozens of different technologies. A staff+ engineer at a 200-person business in the Middle East wrote in their response:

“We’re still trying to figure out how to deal with tooling consistency on a team level. It’s one of our biggest struggles, but possibly more due to company structure than anything else. Everyone is using different tools with little coherence. It’s been rough.”

Some companies have briefly rolled back AI to deal with the negative effects first. From an engineering lead at a 10,000+ person company in Europe:

“Since the AI boom, the quality of technical writing and reasoning from senior engineers in my org has significantly deteriorated. There’s an overwhelming volume of low-quality work product that is generated entirely or in part by AI, which has made it very difficult to conduct meaningful review of RFCs or code. We’ve also seen costly production incidents caused by code written and/or approved by AI, and – while my employer initially bought heavily into the hype – we have now rolled back some of our AI tools to deal with the drop in quality.”

3. Impact on codebase quality

A concrete pattern in our survey data is that codebase quality is decreasing due to AI. The contributing factors are not surprising:

• “AI slop”: more low-quality code generated, such as duplicated, verbose code, and poor abstractions

• Too many code reviews, which means review quality slips

• More bugs: due to faster code output and less strict reviews, more bugs sneak into codebases

We discuss the degradation of products and codebases in a recent deepdive, Are AI agents actually slowing us down?

One CTO at a European startup lists their negatives about increased AI usage:

1. “A lot of tiny bugs and low code quality if you are not careful, verify carefully, and have good structure and guardrails

2. AI agents generate too much and repetitive code, making systems harder to maintain

3. Developers lose understanding of the codebase and become numb to bad architecture and bad developer experience”

According to our survey, management often seems unfazed by decreasing quality, and instead focuses on the higher output. A principal DevOps engineer at a large European company said:

“In our company, we hand AI tools to inexperienced engineers who can’t distinguish good code from bad code and it’s falling on deaf ears in our leadership. They only seem to care about short to mid-term cost savings.”

The maintenance burden of AI-generated code is falling on the fewer engineers who understand and care. A staff engineer at a European company listed the problems that AI-generated code is causing:

• “Drive by” contributions are up: many more occasional non-core-engineer contributors adding code but not sharing the maintenance burden

• Contributing without adding guardrails: many engineers and most of engineering leadership are not using reasonable guardrails like tests

• AI slop from folks who have nothing to do with the codebase: huge volume of slop incoming from people who don’t understand the codebase, but will commit and create PRs without fully understanding what they’re doing

• Complexity is exploding: thanks to the above

The maintenance budget is falling upon fewer devs, while the task of refactoring bloated codebases and reducing complexity is left to those still sufficiently in touch with the codebase, thereby making the maintenance burden even worse.

But some leaders “get it.” A CEO at a 20-person company told us:

“While AI has made generating code ‘cheaper’, the monitoring and maintenance worry me; the things that have traditionally cost the most in software. We’re increasing the rate of shipping large amounts of code with less understanding and increasing the unpredictability, so how do we work the predictability back on top?”

There is industry pressure on companies to adopt AI tooling and impose its usage upon engineers, driven by a mix of factors:

• Seeing actual benefits of AI and hearing that other teams and companies enjoy success with it

• Fear of being left behind by competitors, or becoming less relevant

• Anxiety about investor interest if a company is seen as not adopting the latest AI tools.

This often leads to:

• Top-down mandates to use AI

• Expectation of headcount reduction, with smaller teams doing the same amount or more

• Management treating AI productivity gains as a baseline rather than a bonus.

One staff engineer at a 10,000-person US company explained it like this:

“AI is part of almost every work conversation. The entire company expects it to increase productivity and reduce the need to hire people. I keep trying to get better at using it and trying to make it more reliable so I can do more. I do worry about the quality of the work and atrophy of certain skills. It’s unclear to me if those skills even matter anymore.”

Pushing AI adoption blindly triggers red flags. Respondents shared what makes them worry about things going south in their workplaces:

• Focus on tracking AI usage, but not the quality of the output. This will likely lead to product regressions and unhappy customers

• Pushing for universal adoption. Some companies target 50%, 80%, or 100% AI usage for certain tasks, seemingly blind to how some targets can worsen the quality of output, or simply create wasteful usage. See the trend of tokenmaxxing.

• A focus on velocity, but without recognition of quality work. Expecting more velocity and output seems to be the baseline, and there is no recognition for work truly well done.

The “move fast and break things” mantra famously championed by Facebook seems widespread across the industry with AI tools. A senior manager working at a large, European-headquartered company told us:

“I see a trend: move fast and break things, and end up breaking things too often. We have to learn to focus on testing and resiliency a lot more, as with AI-driven development we introduce more bugs than before. But the velocity gain is bigger for now”.

Output over quality is leading to the death of code review at some places. As a lead engineer working at a small company summarized:

“We’re at the death of code review. I used to do very deep code reviews where I’d take the time to understand the architecture and organization and provide feedback on maintainability and efficiency. I have no motivation in spending that time to review a giant PR where it’s clear that even the original author didn’t bother to do that”.

4. Less experienced engineers and AI

The first generation of software engineers who have never developed without the help of AI are now entering the industry. Here is a response from a young engineer working at a startup as an intern:

“I have never worked as a developer without AI. Writing this scares me a bit, actually, but it’s the truth!”

But this will be the new reality for those joining the industry. So, what needs to happen to help a new generation of “AI-native” grad engineers grow professionally?

AI is an amplifier which could amplify the lack of experience. A staff engineer in the US at a large company told us:

“Agentic AI is a fascinating mirror. It can code as well as the user who drives it. If that user is a junior engineer, now you have a faster junior engineer. If the user is a staff engineer, now you have a faster staff engineer.

What agentic AI doesn’t do is magically convert a junior engineer into a staff engineer, because the user driving it still needs enough experience to know what a good solution looks like”.

A junior engineer in Australia shares their experience of how frustrating working with AI tools is:

“I think AI agents are great for vibe coding or prototypes where the code quality and functionality doesn’t matter that much. I think it’s also useful for senior engineers who know what they’re doing.

For junior engineers like myself, these AI tools are stressful to use. I don’t have the experience or knowledge to tell AI exactly what to do or quickly confirm its output, so I spend a lot of time on just triple checking and redoing stuff. I’m overall frustrated, but I’m trying to embrace it as we’ve been asked to by the company”.

Less experienced engineers seem to use more AI tokens and rack up higher bills. Several respondents observed this: director-level folks noted that junior engineers are in the top-spender category in their orgs, and it is junior devs who spend tokens on unproductive use cases.

There should be more space for junior folks to grow because they use AI more, one staff engineer respondent said:

“Companies need to give some breathing room to Junior engineers and help them learn and acquire knowledge using AI tools as a booster and not as a replacement”.

Junior folks seem to be delegated fewer opportunities that could help them grow. This is because senior people can turn to AI for tasks, including those which they would have previously delegated to an intern or new grad. A few responses mention this:

“AI allows me to have work done that I would usually delegate to a junior or pay a SaaS for; e.g., writing drafts, summarizing the news. “ DevSecOps lead at a small company, Europe

“I’ve begun to automate any repetitive task that we previously relied on juniors and offshore contractors for.” - Engineering manager, at a large company, US

“I no longer have to delegate work by writing a very long document and briefing a junior engineer.” - Principal engineer, large company, Europe

Why not consider mentoring junior devs in your organization? It’s clear that less experienced engineers are having a rocky start to their careers, so delegating stuff to them instead of to AI could be of high value for newer generations of talent.

5. AI tooling ‘addiction’

It seems that the rapid feedback loops of AI-assisted development create addictive tendencies, and there’s a noteworthy presence of “addiction lingo” in some responses to our survey:

Read more

Four decades ago, the writer of ‘The Mythical Man-Month’ (1975), drew on folklore about werewolves to publish a paper about the prospects of a so-called silver bullet for software development that would make professionals much more productive at their craft.

Frederick P. Brooks published “No Silver Bullet – Essence and Accident in Software Engineering” in 1986, and as the title suggests, it is pessimistic about the existence of any silver bullets. The term refers to a super weapon capable of dropping otherwise near-unstoppable werewolves and other creepy supernatural beings in European folk tales.

Since its release, this paper might have become even better-known than Mythical Man-Month (MMM). In 1995, the second edition of that book included Brooks’ later essay as chapter 17, along with an additional chapter of reflections.

In this article, we look into whether the essay was correct in its disbelief in silver bullets, or whether any did indeed slay the beast of unproductivity for developers over the course of time. Also, how does AI agents generating so much code, as of today, challenge the entire premise – or not?

We cover:

1. “No silver bullets” – why has it held up? No single new technology or methodological breakthrough by itself introduced magnitudes-of-improvement to the areas that really matter in software engineering. Is that unusual?

2. Is SRE a silver bullet? Google’s Search team introduced the SRE discipline, and won orders-of-magnitude superior reliability to its competitors. But why only Google Search?

3. Was open source + GitHub a silent silver bullet? No development had a bigger impact on the wider tech industry than the open source wave since the 2010s. Has it been a silent silver bullet, an overlooked cause?

4. Could AI be a silver bullet? At first glance, AI generates 100x-or-more code output. But productivity, reliability, and simplicity improvements are a bit unimpressive – at least for now.

Brooks was a computer scientist who led IBM’s System/360 and OS/360 operating systems development, ‘The Mythical Man-Month’ was published in 1975. Last year, we did a deepdive into this engineering classic (Part 1, Part 2, Part 3, Part 4), delving into its predictions and legacy.

1. No silver bullets?

The paper delves into folklore for its motif, a ‘silver bullet,’ and uses it to pose the question of whether there would be any “silver bullets” on the horizon (in 1986) that could be similarly fatal to software engineering complexity. From the paper (emphasis mine:)

“Of all the monsters who fill the nightmares of our folklore, none terrify more than werewolves, because they transform unexpectedly from the familiar into horrors. For these, one seeks bullets of silver that can magically lay them to rest.

The familiar software project has something of this character (at least as seen by the non-technical manager), usually innocent and straightforward, but capable of becoming a monster of missed schedules, blown budgets, and flawed products. So, we hear desperate cries for a silver bullet, something to make software costs drop as rapidly as computer hardware costs do.

But, as we look to the horizon of a decade hence, we see no silver bullet. There is no single development, in either technology or management technique, which by itself promises even one order of magnitude improvement in productivity, in reliability, in simplicity.

Skepticism is not pessimism, however. Although we see no startling breakthroughs, and indeed, believe such to be inconsistent with the nature of software, many encouraging innovations are under way. A disciplined, consistent effort to develop, propagate, and exploit them should indeed yield an order-of-magnitude improvement. There is no royal road, but there is a road.

The first step toward the management of disease was replacement of demon theories and humor theories by the germ theory. That very step, the beginning of hope, in itself dashed all hopes of magical solutions. It told workers that progress would be made stepwise, at great effort, and that a persistent, unremitting care would have to be paid to a discipline of cleanliness. So it is with software engineering today.”

In 1995, Brooks revisited his idea that silver bullets weren’t real in the software domain. From the Mythical Man-Month’s anniversary edition:

“No Silver Bullet” asserts and argues that no single software engineering development will produce an order-of-magnitude improvement in programming productivity within ten years (from the paper’s publication in 1986). We are now nine years into that decade, so it is timely to see how this prediction is holding up.

Whereas The Mythical Man-Month generated many citations but little argument, “No Silver Bullet” has occasioned rebuttal papers, letters to journal editors, and letters and essays that continue to this day.

Most of these attack the central argument that there is no magical solution, and my clear opinion that there cannot be one. Most agree with most of the arguments in “NSB,” but then go on to assert that there is indeed a silver bullet for the software beast, which the author has invented. As I reread the early responses today, I can’t help noticing that the nostrums pushed so vigorously in 1986 and 1987 have not had the dramatic effects claimed.”

Brooks re-concluded that there had been no technological breakthroughs of the type postulated in NSB.

But motivation can also have silver bullet-like effects and always has had, he found via more research into scientific evidence that motivation can boost productivity. In his own words:

“Since “NSB,” Bruce Blum has drawn my attention to the 1959 work of Herzberg, Mausner, and Sayderman.

They find that motivational factors can increase productivity. On the other hand, environmental and accidental factors, no matter how positive, cannot; but these factors can decrease productivity when negative. “NSB” argues that much software progress has been the removal of such negative factors: stunningly awkward machine languages, batch processing with long turnaround times, poor tools, and severe memory constraints.”

Today, it’s a long time since the mid-nineties; with the benefit of hindsight, were there any silver bullets flying between then and 2022, which fit the bill as slayers of unproductiveness? I suggest a few, below. If you can name other silver bullets since the launch of Windows 95, please do so in the comments!

• Version control: (late 1990s.) CVS, Subversion, and later, Git. Version control allowed engineers to collaborate much more fluently, leading to more teamwork and – in some cases – less full-on solo labor.

• IDEs: (early 2000s). Modern IDEs like Visual Studio, IntelliJS, and others make context-rich editing easy and fast. They also allow for faster, less error-prone refactoring and more efficient debugging.

• CI/CD and automated testing: (mid-2000s). CI systems started to spread during the 2000s with the likes of CruiseControl (2000s) → Jenkins and SaaS CI solutions from the 2010s (e.g., Travis CI, CircleCI, GitLab CI, GitHub Actions).

• Open source and package managers: (2010s). Open source has been around for decades, but GitHub’s rapid adoption made it easier to create and discover, coupled with package managers in the Node, Python, and other language ecosystems to build on top of open source solutions.

• StackOverflow: (2010s). The popular programming Q&A site made it easier to get unstuck by finding solutions to common problems, with the capability to ask questions and get responses from the large user community within hours. By 2025, the site was pretty much dead.

• Cloud: (early 2010s). AWS launched in 2006, then Azure and Google Cloud in 2008, and they went mainstream in the 2010s. Today, cloud is everywhere, and increasingly more infrastructure startups build on top of hyperscalers (the biggest public cloud platforms), such as Vercel / Netlify (platform-as-a-service), Supabase / Turbopuffer (databases) and others.

• Platform teams & DevEx teams: (mid-2010s). Mid-sized and larger tech companies created dedicated platform teams to own infrastructure and internal platforms. In the 2020s, larger tech companies have created ‘Developer Experience’ teams to build better internal dev tools and workflows. We previously covered Uber’s program/platform split, and Uber’s developer experience evolution.

• SRE: the Site Reliability Engineer (SRE) profession grew during the 2010s, much influenced by Google.

These technologies increased developer efficiency and productivity, but none by itself was a productivity accelerator in isolation.

Obviously, by 2022 the craft of building software had developed greatly since ‘No Silver Bullets’ came out; and was more efficient, faster, and more collaborative than ever. One highly anecdotal way to identify this is via the disappearance of cake from some tech workplaces. Back in the day, cake was distributed at work for major product milestones being hit: the shipping of a new product was often marked with awards and tasty baked treats – at least on teams building browsers, like the IE and Firefox teams.

But by the 2010s, shipping frequency had increased by so much and was an everyday, unremarkable occurrence at some places, according to Matt Brubeck, a former engineer on the Firefox team:

“Back when Firefox 2 was released (six years ago this week!), the Internet Explorer team started a friendly tradition of sending Mozilla a cake as congratulations. This continued for Firefox 3 and Firefox 4. After Firefox switched from major releases once or twice a year to incremental updates every six weeks, they sent us a cupcake for the next few updates instead. :)” Mozilla engineer, Matt Brubeck

Today, Firefox ships a stable version about once a month, as does Chrome. In this context, marking each release with more cake could inadvertently cause some health issues on the team – too much cake, that is! From this September, Chrome will switch to shipping every two weeks.

Agile and Scrum is worth a mention; not as a technology, but a methodology: Scrum encourages teams to move in smaller cycles and deliver more frequently, via sprints that typically range from a week to a month. In the early 2000s, this methodology spread quickly and brought efficiency improvements to many tech companies. However, by the early 2020s, many startups and some of Big Tech had moved on, as covered in How Big Tech runs tech projects and the curious absence of Scrum:

“Scrum got in the way of shipping on a daily basis. The whole idea of Scrum revolves around Sprints, of committing to tasks at the beginning of the sprint, working on these during the sprint, and demoing what we did at the end.

The process felt unnatural and like it had been forced on a fast-moving web team. We soon moved to a more fluid way of working, taking the Kanban approach. We stopped caring about sprints, and dropped most rituals that come with Scrum. We just cared about knowing what we’re working on now, and what it was we’d get done next.”

Basically, Scrum worked and still does so for teams wanting to shorten shipping cadence from months to weeks. But for teams shipping daily, it often gets in the way.

One area that improved significantly has been the pace of shipping incremental software. In 1975, shipping software several times per day with elements like version control, CI/CD, feature flags and engineers being oncall, might have sounded far-fetched. Back then, software delivery was measured in months and years. In this way, we’ve perhaps made improvements overall in the regions of 10x to 100x over the years.

But that came via combinations of new tools like version control and CI/CD, new approaches & methodologies, and testing – and also from shifting constraints; for example, it’s now possible to revert backend changes rapidly, and code shipped in binaries can be controlled by feature flags in many cases.

Even so, improvements were mainly in iteration speed and not necessarily in the complexity of the software shipped. With all that progress, shipping complex and high-quality software still takes comparable time, often years, as 50 years ago. A prime example is the upcoming video game, Grand Theft Auto VI, probably by now the most highly-anticipated game ever, which is set to launch in November, after at least six years – and potentially 12 – of total development time:

• Initial planning started in 2014 (12 years of development)

• Development started in earnest in 2020 (circa six years of full-on development)

• The studio, Rockstar, confirmed development was underway in February 2022 (at least 4.5 years of full development)

The video game development timeline is as long as it ever was, and even longer, as developers strive to meet players’ expectations on things like graphics, lighting, and physics. GTA 6 looks like being the most complex installment in the long-running series. So, perhaps there’s not been much change in software delivery timelines because when we have more capabilities to work with, the goals get more ambitious and the bar for “standout” software keeps rising.

2. Is SRE a silver bullet?

Brooks’s definition of a silver bullet:

“A single development, in either technology or management technique, which by itself promises even one order of magnitude improvement in productivity, in reliability, in simplicity.”

In simplicity and productivity terms, I struggle to name a single approach that delivered a 10x-or-more improvement by itself. But in the area of reliability, one company that has pioneered novel approaches since the 2000s is Google. Google.com is probably the single most reliable piece of internet software of all. In the last 15 years, Google Search has suffered a single outage, on 8 August 2022, which lasted around an hour. Otherwise, there have been no global outages (of course, there have been several for other Google services).

In 2003, Google created the ‘Site Reliability Engineering’ (SRE) role. SRE veteran, Dave O’Connor, shared with us:

“The fervent belief of Google’s founders was that speed and reliability mattered more than features. This belief was coupled with the understanding that we couldn’t achieve it traditionally, which made it an existential issue. The level of investment in building out all layers of the serving stack was a case of “because we can”, but also “because we have to, as nowhere else does what we need”.

There was never a question of whether traditional ‘ops’ would work at Google. We needed a specialized role, staffed by folks familiar with the problem space and engineering methods required to make it work.

In 2003, the SRE role was born. Ben Treynor Sloss had been tasked with building Google’s “production team” and in his own words, he built “what happens when you ask a software engineer to design an operations team.” This turned into the birth of the SRE function at Google. From the outset, SRE was staffed in varying measures by systems/operations experts and software engineers. A large part of the remit of the team was to build the tools and practices required to operate Google’s fleet.”

Over time, the rest of the industry caught on to SRE and DevOps. From our SRE deepdive:

“Eventually, other companies caught onto the scaling issues, especially the hyperscalers. Each had their own approach, but over time, the notion grew industry-wide that making things reliable was a real-life engineering discipline, not simply ‘ops’.

This step saw a number of terms coined to describe this engineering, including ‘DevOps’. At its core, this was the notion that the disciplines and practices of reliability engineering should be ingrained into the overall engineering organization. At places other than Google, this mostly took the form of combined developer/operations roles (i.e. “you build it, you run it”), which differed from Google’s implementation, but the practices were similar.

Around this time, Google started opening up about SRE, eventually publishing the first SRE book, and follow ups. Conferences such as USENIX SRECon, Devops Days, and other movements have solidified reliability engineering as a discipline that scales well beyond Google. Indeed, the company has become a consumer of many state-of-the-art developments.”

So, at Google Search, the SRE role could be described as a genuine silver bullet for the tech giant. The company’s obsession with reliability helped it build what is probably the most reliable public-facing service of all. On the assumption that SRE plays a significant role in the approach, I would feel comfortable with calling SRE a silver bullet for Google Search.

SRE, as a concept, is commonplace across Google, but the reliability of its other services is not so impressive. For example, Google Cloud has had many outages, and Gmail also goes down every now and then. I’m sure that without SRE, reliability would be worse, but in general, Google services’ availability these days is probably a magnitude higher than the availability of most online services in the 2000s.

Similarly, GitHub has an SRE role but the service is at zero nines of availability, partially explained by a 3.5x increase in load in two years. But in other ways, the zero nines is likely self-inflicted.

This makes me wonder if the existence of silver bullets depends greatly on teams and individual contexts. SRE seems like a good case to consider:

• a “silver bullet” for Google Search

• … but not for other Google services

• … and definitely not for the broader industry

Could it be that when implemented in the right place, in the right way, and with the correct investment, then SRE – and an incredible focus on reliability – will yield a 10x-or-higher increase in reliability?

My hunch is that Google Search has such standout reliability not just because of SRE, but because Search might be the only organization in Google with reliability as a founding value, embedded in the team’s culture, with unmatched investments of time and money.

Google has published several books that explain their techniques and practices, but for other teams to get those results, they would need to invest similarly in reliability.

3. Was open source + GitHub a silent silver bullet?

Perhaps there’s a silver bullet which is easily missed: open source. In the first-ever Pragmatic Engineer Podcast episode, I asked software engineer Simon Willison what the biggest “productivity leaps” have been during his career. He named open source:

Read more

Nine years later, he felt the time was ripe for an updated edition, with cloud computing much more widespread than in 2016. So, Martin teamed up with software engineer and investor, Chris Riccomini, a former colleague at LinkedIn and the author of The Missing README, for a full refresh of the book which brings it right up to date for the present day.

Martin was recently on The Pragmatic Engineer Podcast, where we discussed this updated volume and many related cloud computing matters. We also looked into some topics that have become less relevant over time, like details on MapReduce.

I asked Martin if this newsletter could share an excerpt of the updated edition of the book about a timeless, important topic, and he generously agreed. So, today we cover:

1. Cloud versus self-hosting tradeoffs

2. Doing the right thing as a software engineer

These excerpts are only part of the book; the first edition has been on my shelf for years and is now in well-worn condition. I jumped at the chance to get the second edition, and if you’re interested in building resilient systems, I recommend it as an excellent resource.

Get the second edition of the book

My usual disclaimer: as with all my recommendations, I was not paid for this article, and none of the links are affiliates. See my ethics statement for more.

The excerpt below is from “Designing Data-Intensive Applications” second edition, by Martin Kleppmann and Chris Riccomini. Copyright © 2026 Martin Kleppmann, Chris Riccomini. Published by O’Reilly Media, Inc. Used with permission.

1. Cloud versus self-hosting tradeoffs

This excerpt is from Chapter 1: “Trade-Offs in Data Systems Architecture”

For anything that an organization needs to do, one of the first questions is whether it should be done in-house or outsourced. That is, should you build or should you buy?

Ultimately, this is a question about business priorities. A common rule of thumb is that things that are a core competency or a competitive advantage of your organization should be done in-house, whereas things that are non-core, routine, or commonplace should be left to a vendor [20]. To give an extreme example, most companies do not fabricate their own CPUs, since it is cheaper to buy them from the semiconductor manufacturers.

With software, two important decisions to be made are who builds the software and who deploys it. The spectrum of possibilities is illustrated in Figure 1-2. At one extreme is bespoke software that you write and run in-house; at the other extreme are widely-used cloud services or SaaS products that are implemented and operated by an external vendor and that you access only through a web interface or API.

The middle ground is off-the-shelf software (open source or commercial) that you self-host, or deploy yourself – for example, if you download MySQL and install it on a server you control. This could be on your own hardware (often called ‘on-premises,’ even if the server is in a rented datacenter rack and not literally on your own premises), or on a virtual machine (VM) in the cloud (infrastructure as a service, or IaaS). There are more points along this spectrum, such as taking open source software and running a modified version of it.

A related question is how you deploy services, either in the cloud or on premises – for example, whether you use an orchestration framework such as Kubernetes. However, choice of deployment tooling is beyond the scope of this book, since other factors have a greater influence on the architecture of data systems.

Pros & Cons of Cloud Services

Using a cloud service, rather than running comparable software yourself, essentially outsources the operation of that software to the cloud provider. There are good arguments for and against this approach. Cloud providers claim that using their services saves time and money and allows you to move faster compared to setting up your own infrastructure.

Whether using a cloud service is actually cheaper and easier than self-hosting depends very much on your skills and the workload on your systems, however. If you already have experience of setting up and operating the systems you need, and if your load is quite predictable (i.e., the number of machines you need does not fluctuate wildly), then it’s often cheaper to buy your own machines and run the software on them yourself [21, 22].

On the other hand, if you need a system that you don’t already know how to deploy and operate, adopting a cloud service is often easier and quicker than learning to manage the system. Hiring and training staff specifically to maintain and operate the system can get very expensive. You still need an operations team when you’re using the cloud, but outsourcing the basic system administration can free up your team to focus on higher-level concerns.

Outsourcing the operation of a system to a company that specializes in running it can potentially result in better service, since the provider gains operational expertise from providing the service to many customers. On the other hand, if you run the service, you can configure and tune it to perform well on your particular workload. A cloud service would likely be unwilling to make such customizations on your behalf.

Cloud services are particularly valuable if the load on your systems varies a lot over time. If you provision your machines to be able to handle peak load, but those computing resources are idle most of the time, the system becomes less cost-effective. In this situation, cloud services have the advantage that they can make it easier to scale your computing resources up or down in response to changes in demand.

For example, analytical systems often have extremely variable load. Running a large analytical query quickly requires a lot of computing resources in parallel, but once the query completes, those resources sit idle until a user makes the next query. Predefined queries (e.g., for daily reports) can be enqueued and scheduled to smooth out the load, but for interactive queries, the faster you want them to complete, the more variable the workload becomes. If your dataset is so large that querying it quickly requires significant computing resources, using the cloud can save money as you can return unused resources to the provider rather than leaving them idle. For smaller datasets, this difference is less significant.

The biggest downside of a cloud service is that you have no control over it:

• If it is lacking a feature you need, all you can do is politely ask the vendor whether they will add it; you generally cannot implement it yourself.

• If the service goes down, all you can do is wait for it to recover.

• If you are using the service in a way that triggers a bug or causes performance problems, diagnosing the issue will be difficult. With software that you run yourself, you can get performance metrics and debugging information from the operating system to help you understand its behavior, and you can look at the server logs. With a service hosted by a vendor, you usually do not have access to these internals.

• If the service shuts down or becomes unacceptably expensive, or if the vendor changes their product in a way you don’t like, you are at their mercy; continuing to run an old version of the software is usually not an option, so you’ll be forced to migrate to an alternative service [23]. This risk is mitigated if alternative services expose a compatible API, but for many cloud services there are no standard APIs, which raises the cost of switching, making vendor lock-in a problem.

• If the cloud provider is in another country and a political conflict arises between that country and your own, you risk being locked out of the service due to imposed sanctions.

• The cloud provider needs to be trusted to keep the data secure, which can complicate the process of complying with privacy and security regulations.

Despite all these risks, it has become more and more popular for organizations to build new applications on top of cloud services, or to adopt a hybrid approach in which cloud services are used for some aspects of a system. However, cloud services will not subsume all in-house data systems. Many older systems predate the cloud, and for any services that have specialist requirements that existing cloud services cannot meet, in-house systems remain necessary. For example, very latency-sensitive applications such as high-frequency trading require full control of the hardware.

Cloud-Native System Architecture

Besides having a different economic model (subscribing to a service instead of buying hardware and licensing software to run on it), the rise of the cloud has also had a profound effect on how data systems are implemented on a technical level. The term “cloud native” is used to describe an architecture that is designed to take advantage of cloud services.

In principle, almost any software that you can self-host could also be provided as a cloud service, and indeed, such managed services are now available for many popular data systems. However, systems that have been designed from the ground up to be cloud native have been shown to have several advantages: better performance on the same hardware, faster recovery from failures, being able to quickly scale computing resources to match the load, and supporting larger datasets [24, 25, 26]. Table 1-2 lists some examples of both types of systems.

Layering of cloud services

Many self-hosted data systems have simple system requirements; they run on a conventional operating system such as Linux or Windows, they store their data as files on the filesystem, and they communicate via standard network protocols such as TCP/IP. A few systems depend on special hardware such as GPUs (for ML) or remote direct memory access (RDMA) network interfaces, but on the whole, self-hosted software tends to use generic computing resources: CPUs, RAM, a filesystem, and an IP network.

In a cloud, this type of software can be run in an IaaS environment, using one or more VMs (or instances) with a certain allocation of CPUs, memory, disk, and network bandwidth. Compared to physical machines, cloud instances can be provisioned faster and come in a greater variety of sizes, but otherwise they are similar to traditional computers: you can run any software you like on them, but you are responsible for administering it yourself.

In contrast, the key idea of cloud-native services is not only to use the computing resources managed by your operating system, but also to build upon lower-level cloud services to create higher-level services. For example:

• Object storage services such as Amazon S3, Azure Blob Storage, and Cloudflare R2 store large files. They provide more limited APIs than a typical filesystem (basic file reads and writes), but they have the advantage that they hide the underlying physical machines; the service automatically distributes the data across many machines so that you don’t have to worry about running out of disk space on any one machine. Even if some machines or their disks fail entirely, no data is lost.

• Many other services are, in turn, built upon object storage and other cloud services. For instance, Snowflake is a cloud-based analytical database (data warehouse) that relies on S3 for data storage [26], and some other services, in turn, build upon Snowflake.

As always with abstractions in computing, there is no one right answer to what you should use. As a general rule, higher-level abstractions tend to be more oriented toward particular use cases. If your needs match the situations for which a higher-level system is designed, using the existing higher-level system will probably meet your needs with much less hassle than building it yourself from lower-level systems would. On the other hand, if no high-level system meets your needs, building it yourself from lower-level components is the only option.

Separation of storage and compute

In traditional computing, disk storage is regarded as durable (we assume that once something is written to disk, it will not be lost). To tolerate the failure of an individual hard disk, RAID (redundant array of independent disks) is often used to maintain copies of the data on several disks attached to the same machine. RAID can be implemented either in hardware or in software by the operating system, and it is transparent to the applications accessing the filesystem.

In the cloud, compute instances (VMs) may also have local disks attached, but cloud-native systems typically treat these disks more like an ephemeral cache and less like long-term storage. This is because the local disk becomes inaccessible if the associated instance fails, or if the instance is replaced with a bigger or a smaller one (on a different physical machine) to adapt to changes in load.

As an alternative to local disks, cloud services also offer virtual disk storage that can be detached from one instance and attached to a different one (e.g., Amazon EBS, Azure managed disks, and persistent disks in Google Cloud). Such a virtual disk is not a physical disk, but rather a cloud service provided by a separate set of machines that emulates the behavior of a disk (a block device, where each block is typically 4 KiB in size). This technology makes it possible to run traditional disk-based software in the cloud, but the block device emulation introduces overheads that can be avoided in systems that are designed from the ground up for the cloud [24]. The use of virtual disks also makes the application very sensitive to network glitches, since every I/O operation on the virtual block device is a network call [27].

To address this problem, cloud-native services generally avoid using virtual disks and instead build on dedicated storage services that are optimized for particular workloads. Object storage services such as S3 are designed for long-term storage of fairly large files, ranging from hundreds of kilobytes to several gigabytes in size. The individual rows or values stored in a database are typically much smaller than this; cloud databases therefore typically manage smaller values in a separate service and store larger data blocks (containing many individual values) in an object store [25, 28].

In traditional systems architecture, the same computer is responsible for both storage (disk) and computation (CPU and RAM), but in cloud-native systems, these two responsibilities have become somewhat separated, or disaggregated [9, 26, 29, 30]: for example, S3 only stores files, and if you want to analyze that data, you will have to run the analysis code somewhere outside of S3. This implies transferring the data over the network.

Furthermore, cloud-native systems are often multitenant, which means that rather than having a separate machine for each customer, data and computation from several customers are handled on the same shared hardware by the same service [31]. Multitenancy can enable better hardware utilization, easier scalability, and easier management by the cloud provider, but it also requires careful engineering to ensure that one customer’s activity does not affect the performance or security of the system for other customers [32].

Operations in the Cloud Era

Traditionally, the people managing an organization’s server-side data infrastructure were known as database administrators (DBAs), or system administrators (sysadmins). More recently, many organizations have tried to integrate the roles of software development and operations into teams with a shared responsibility for both backend services and data infrastructure; the DevOps philosophy has guided this trend. Site reliability engineers (SREs) are Google’s implementation of this idea [33].

The role of operations is to ensure that services are reliably delivered to users (including configuring infrastructure and deploying applications) and to ensure a stable production environment (including monitoring and diagnosing any problems that may affect reliability). For self-hosted systems, operations traditionally involve a significant amount of work at the level of individual machines, such as capacity planning (e.g., monitoring available disk space and adding more disks before you run out of space), provisioning new machines, moving services from one machine to another, and installing operating system patches.

Many cloud services present an API that hides the individual machines implementing the service. For example, cloud storage replaces fixed-size disks with metered billing, where you can store data without planning your capacity needs in advance, and you are then charged based on the space used. Moreover, many cloud services remain highly available, even when individual machines have failed.

This shift in emphasis from individual machines to services has been accompanied by a change in the role of operations. The high-level goal of providing a reliable service remains the same, but the processes and tools have evolved.

The DevOps/SRE philosophy places greater emphasis on the following:

• Setting up automation; preferring repeatable processes over manual one-off jobs

• Using ephemeral VMs and services rather than long-running servers

• Enabling frequent application updates

• Learning from incidents

• Preserving the organization’s knowledge about the system, even as individuals come and go [34]

With the rise of cloud services, a bifurcation of roles has occurred. Operations teams at infrastructure companies specialize in the details of providing a reliable service to a large number of customers, while the customers of the service spend as little time and effort as possible on infrastructure [35].

Customers of cloud services still require operations, but they focus on different aspects, such as choosing the most appropriate service for a given task, integrating services with each other, and migrating from one service to another. Even though metered billing removes the need for capacity planning in the traditional sense, it’s still important to know what resources you are using for which purpose so that you don’t waste money on cloud resources that are not needed. Capacity planning becomes financial planning, and performance optimization becomes cost optimization [36]. Additionally, cloud services do have resource limits or quotas (such as the maximum number of processes you can run concurrently), which you need to know about and plan for before you run into them [37].

Adopting a cloud service can be easier and quicker than provisioning and running your own infrastructure, although you still have to learn how to use the cloud service and perhaps work around its limitations. Integration among services becomes a particular challenge as a growing number of vendors offer an ever-broader range of cloud services targeting different use cases [38, 39]. ETL is only part of the story; operational cloud services also need to be integrated with each other. At present, we lack standards to facilitate this sort of integration, so it often involves significant manual effort.

Other operational aspects that cannot fully be outsourced to cloud services include maintaining the security of an application and the libraries it uses, managing the interactions between your own services, monitoring the load on your services, and tracking down the cause of problems such as performance degradations or outages. While the cloud is changing the role of operations, the need for operations is as great as ever.

2. Doing the right thing as a software engineer

The excerpt below is a section from Chapter 14, “Doing the Right Thing”

In the final chapter of this book, let’s take a step back. Throughout, we have examined a wide range of architectures for data systems, evaluated their pros and cons, and explored techniques for building reliable, scalable, and maintainable applications. However, we have left out a fundamental part of the discussion, which we should now fill in.

Every system is built for a purpose; every action we take has both intended and unintended consequences. The purpose may be as simple as making money, but the consequences may be far-reaching. We, the engineers building these systems, have a responsibility to carefully consider those consequences and to ensure that our decisions do not cause harm.

We talk about data as an abstract thing, but remember that many datasets are about people: their behavior, their interests, their identities. We must treat such data with humanity and respect. Users are humans too, and human dignity is paramount [1].

Software development increasingly involves making important ethical choices. There are guidelines to help software engineers navigate these issues, such as the ACM Code of Ethics and Professional Conduct [2], but they are rarely discussed, applied, or enforced in practice. As a result, engineers and product managers sometimes take a cavalier attitude to privacy and the potential negative consequences of their products [3, 4].

A technology is not good or bad in itself – what matters is how it is used and how it affects people. This is true of a software system such as a search engine in much the same way as it is for a weapon like a gun. The ethical responsibility is ours to bear; it is not sufficient for software engineers to focus exclusively on the technology and ignore its consequences.

In contrast to much of computing, however, the concepts at the heart of ethics are not fixed or determinate in their precise meaning; they require interpretation, which may be subjective [5]. What makes something “good” or “bad” is not well defined, and serious discourse on the subject among computing professionals is lacking [6]. Reasoning about ethics is difficult, but also too important to ignore. What does this entail? “Ethics” are not a checklist with which to comply; it’s a participatory and iterative process of reflection, in dialogue with people involved and accountability for the results [7].

Predictive Analytics

Predictive analytics is a major part of why people are excited about big data and AI. It’s also an area that is fraught with ethical dilemmas. Using data analysis to predict the weather, or the spread of diseases, is one thing [8]; it is another matter to predict whether a convict is likely to reoffend, whether an applicant for a loan is likely to default, or whether an insurance customer is likely to make expensive claims [9]. The latter have a direct effect on people’s lives.

Naturally, payment networks want to prevent fraudulent transactions, banks want to avoid bad loans, airlines want to avoid hijackings, and companies want to avoid hiring ineffective or untrustworthy people. From their point of view, the cost of a missed business opportunity is low, but the cost of a bad loan or a problematic employee is much higher, so it is expected for organizations to be cautious. If in doubt, they are better off saying “no”.

However, as algorithmic decision making becomes more widespread, someone who has (accurately or falsely) been labeled as risky by an algorithm may suffer a large number of “no” decisions. Systematically being excluded from jobs, air travel, insurance coverage, property rental, financial services, and other key aspects of society is such a large constraint on an individual’s freedom that it has been called “algorithmic prison” [10]. In countries that respect human rights, the criminal justice system presumes innocence until proven guilty; on the other hand, automated systems can systematically and arbitrarily exclude a person from participating in society without any proof of guilt and little chance of appeal.

Bias & discrimination

Decisions made by an algorithm are not necessarily any better or any worse than those made by a human. Everyone is likely to have biases, even if they actively try to counteract them, and discriminatory practices can become culturally institutionalized. There is hope that basing decisions on data, rather than subjective and instinctive human assessments, could be more fair and give a better chance to people who are often overlooked or disadvantaged in the traditional system [11].

When we develop predictive analytics and AI systems, we are not merely automating a human’s decision by using software to specify the rules for when to say “yes” or “no”; we are leaving the rules themselves to be inferred from data. However, the patterns learned by these systems are opaque: even if the data indicates a correlation, we may not know why. If the input to an algorithm carries a systematic bias, the system will most likely learn and amplify that bias in its output [12].

In many countries, anti-discrimination laws prohibit treating people differently depending on protected traits such as ethnicity, age, gender, sexuality, disability, or beliefs. Other features of a person’s data may be analyzed, but what happens if they are correlated with protected traits? For example, in racially segregated neighborhoods, a person’s postal code or even their IP address is a strong predictor of race. Put like this, it seems ridiculous to believe that an algorithm could somehow take biased data as input and produce fair and impartial output from it [13, 14]. Yet this belief often seems to be implied by proponents of data-driven decision making; an attitude that has been satirized as “machine learning is like money laundering for bias” [15].

Predictive analytics systems merely extrapolate from the past; if the past is discriminatory, they codify and amplify that discrimination [16]. If we want the future to be better than the past, moral imagination is required, and that’s something only humans can provide [17]. Data and models should be our tools, not our masters.

Responsibility and Accountability

Automated decision-making raises the question of responsibility and accountability [17]. If a human makes a mistake, they can be held accountable, and the person affected by the decision can appeal. Algorithms make mistakes too, but who is accountable when they go wrong? [18] When a self-driving car causes an accident, who is responsible? If an automated credit scoring algorithm systematically discriminates against people of a particular race or religion, is there any recourse? If a decision by your ML system comes under judicial review, can you explain to the judge how the algorithm made its decision? People should not be able to evade responsibility by blaming an algorithm.

Credit rating agencies are a classic example of collecting data to make decisions about people. A bad credit score makes life difficult, but at least a credit score is normally based on relevant facts about a person’s actual borrowing history, and any errors in the record can be corrected (although the agencies normally do not make this easy). Scoring algorithms based on machine learning, however, typically use a much wider range of inputs and are much more opaque, making it harder to understand how a particular decision has come about and whether someone is being treated in an unfair or discriminatory way [19].

A credit score summarizes “how did you behave in the past?” whereas predictive analytics usually work on the basis of “who is similar to you, and how did people like you behave in the past?” Drawing parallels to others’ behavior implies stereotyping people; for example, based on where they live (a close proxy for race and socioeconomic class). What about people put in the wrong bucket? Furthermore, if a decision is incorrect because of erroneous data, recourse is almost impossible [17].

Much data is statistical in nature, which means that even if the probability distribution on the whole is correct, individual cases may well be wrong. For example, if the average life expectancy in your country is 80 years, that doesn’t mean you’re expected to drop dead on your 80th birthday. From the average and the probability distribution, you can’t say much about the age to which someone will live. Similarly, the output of a prediction system is probabilistic and may well be wrong in individual cases.

A blind belief in the supremacy of data for making decisions is not only delusional, but also positively dangerous. As data-driven decision making becomes more widespread, we will need to figure out how to avoid reinforcing existing biases, how to make algorithms accountable and transparent, and how to fix them when they inevitably make mistakes.

We will also need to figure out how to realize the positive potential of data and prevent it from being used to harm people. For example, analytics can reveal financial and social characteristics about personal lives. On the one hand, this power could be used to focus aid and support to help those who need it most. On the other hand, it is sometimes used by predatory businesses seeking to identify vulnerable people and sell them risky products such as high-cost loans or worthless college degrees [17, 20].

Feedback loops

Even with predictive applications with less immediately far-reaching effects on people, such as recommendation systems, there are difficult issues that we must confront. When services become good at predicting the content users want to see, they may end up showing them only opinions they already agree with, leading to echo chambers in which stereotypes, misinformation, and polarization can breed. We already know the impact that social media echo chambers can have on election campaigns.

When predictive analytics affect people’s lives, particularly pernicious problems arise because of self-reinforcing feedback loops. For example, consider the case of employers using credit scores to evaluate potential hires. You may be a good worker with a good credit score, but suddenly find yourself in financial difficulties due to a misfortune beyond your control. As you miss payments on your bills, your credit score suffers, and you will be less likely to find work. Joblessness pushes you toward poverty, which further worsens your score, making it even harder to find employment [17]. It’s a downward spiral due to poisonous assumptions, hidden behind a camouflage of mathematical rigor and data.

As another example of a feedback loop, economists found that when gas stations in Germany introduced algorithmic prices, competition was reduced and prices for consumers went up because the algorithms learned to collude [21].

We can’t always predict when such feedback loops may happen. However, many consequences can be predicted by thinking about an entire system (not just the computerized parts, but also the people interacting with it), in an approach known as “systems thinking” [22]. We can try to understand how a data analysis system responds to different behaviors, structures, or characteristics. Does the system reinforce and amplify existing differences between people (e.g., making the rich richer or the poor poorer), or does it try to combat injustice? Even with the best intentions, we must beware of the possibility of unintended consequences.

Surveillance

The excerpt below is from another section in Chapter 14, “Doing the Right Thing”

As a thought experiment, try replacing the word “data” with “surveillance”, and observe whether common phrases still sound so good [23]. How about this: “In our surveillance-driven organization we collect real-time surveillance streams and store them in our surveillance warehouse. Our surveillance scientists use advanced analytics and surveillance processing in order to derive new insights.”

This thought experiment is unusually polemical for this book, “Designing Surveillance-Intensive Applications”, but strong words are needed to emphasize this point. In our attempts to make software “eat the world” [24], we have built the greatest mass surveillance infrastructure ever seen. We are rapidly approaching a world in which every inhabited space contains at least one internet-connected microphone, in the form of smartphones, smart TVs, voice-controlled assistant devices, baby monitors, and even children’s toys that use cloud-based speech recognition. Many of these devices have terrible security track records [25].

What is new compared to the past is that digitization has made it easy to collect large amounts of data about people. Surveillance of our location and movements, our social relationships and communications, our purchases and payments, and our health data has become almost unavoidable. A surveillance organization may end up knowing more about a person than that person knows about themselves; for example, identifying illnesses or economic problems before that individual is aware of them.

Even the most totalitarian, repressive regimes of the past could only dream of putting a microphone in every room and forcing every person to constantly carry a device capable of tracking their location and movements. Yet the benefits that we get from digital technology are so great that we now voluntarily accept this state of total surveillance. The difference is just that the data is being collected by corporations to provide us with services, rather than government agencies seeking control [26].

Not all data collection necessarily qualifies as surveillance, but examining it as such can help us understand our relationship with the data collector. Why are we seemingly happy to accept surveillance by corporations? Perhaps you feel you have nothing to hide; in other words, you are totally in line with existing power structures, you are not a marginalized minority, and you needn’t fear persecution [27]. Not everyone is so fortunate. Or perhaps it’s because the purpose seems benign; it’s not overt coercion and conformance, merely better recommendations and more personalized marketing. However, combined with the discussion of predictive analytics from the last section, that distinction seems less clear.

We are already seeing behavioral data about driving, tracked by vehicles without drivers’ consent, affecting their insurance premiums [28], and health insurance coverage that depends on people wearing a fitness tracking device. When surveillance is used to make decisions that hold sway over important aspects of life, such as insurance coverage or employment, it starts to appear less benign. Data analysis can also reveal surprisingly intrusive things; for example, the movement sensor in a smartwatch or fitness tracker can be used to work out what you are typing (e.g., passwords) with fairly good accuracy [29]. Sensor accuracy and algorithms for analysis are only going to get better.

Takeaways

Thanks to Martin for writing this book, and to himself and Chris for doing a revamp for the second edition. The volume is now even more relevant to how we build systems in 2026 and beyond. You can purchase a hard copy from the publisher’s website or Amazon.

The first edition has a timeless quality because it focused on the fundamentals of large systems, and the new second edition follows the same approach, as laid out in its preface:

“Although the landscape of technologies for processing and storing data is diverse and fast-changing, the underlying principles endure. If you understand those principles, you’re in a position to see where each tool fits in, how to make good use of it, and how to avoid its pitfalls. This book focuses on those principles.”

Since the first edition appeared nine years ago, some things have changed in the tech industry:

• Much greater focus on the cloud. Building large systems on top of cloud infrastructure is more common. This brings lower complexity as cloud primitives hide a lot of implementation complexity, but it also means accepting more risk because when the cloud is down, so is your system.

• Systems which AI tools build upon are more relevant. Vector databases, DataFrames (for training datasets), and the processing of large amounts of training data with batch processing systems are relevant to anyone building production AI systems.

• Local-first software. Martin focuses on this area in his work, and with AI, we could see more demand for running models locally. Operating systems like Ubuntu are also focusing on this.

• Formal methods. The advent of AI-generated code means this topic is getting more attention industry-wide, and the second edition covers it.

• Regulation and legal context. Regulations like the EU’s General Data Protection Regulation (GDPR) are something software engineers increasingly need to know about, and the book now covers it.

If I had to summarize the evolution of the book in its second edition, it would be more focus on cloud and AI, and more on local-first software, testing, and how regulations affect engineers. Interestingly, this mirrors how the tech industry has developed over time, too.

I very much appreciate that the book closes with the final chapter focused on “doing the right thing” as a software engineer. Software systems have wide-ranging societal impact, and engineers working on these systems have a great say in what gets built, and how it gets built. As engineers, we owe it the very least to ourselves to consider the broader impact of our decisions — and doing so might also force us to make important ethical choices. There’s less discussion of the ethics angle on software engineering: and I’m glad that Martin and Chris did not shy away from going deeper into this topic.

If you’d like to get more background on the book – and on the hard parts of building large-scale systems – check out our podcast episode with Martin Kleppmann.

Today’s article focuses on Linux and Ubuntu, and we’ll cover Windows in a follow-up issue. Obviously, I reached out to Apple but heard nothing back, unsurprisingly. If you’re reading this and happen to work at Apple, it’d be great to learn more!

Jon Seager is VP of Engineering at Canonical – the company behind Ubuntu – and has provided new details about what the team there has built for AI support, and some new ideas that they’re brewing up. Today, we cover:

1. Hardware enablement: support for GPUs, NPUs and DPUs. When you turn on a machine with AI accelerators, Ubuntu aims for the hardware to perform at its full potential. This means having proper driver support for PCs and cloud data centers’ computing units.

2. Hardware partnerships. Working closely with NVIDIA, AMD, and Intel means Ubuntu can support those vendors’ new hardware from release day.

3. CPU architecture variants. New versions in a CPU family add to, or change, features. An operating system needs to support a new version of the CPU architecture variant in order to fully utilize it. Ubuntu does this for the x86‑64 family, making it a lot more performant on newer CPUs – while still supporting older CPUs.

4. Local-first bet & plans for agentic workflows. There’s a big focus on running local models and using “inference snaps” which help choose the right model with the right quantization. There is the intention to support agentic workflows at the OS level, one day, which is currently at the early exploration stage.

5. Developer ecosystem. There’s a plan to add more support for AI dev tools, a focus on sandboxing at the OS level, a push to support ARM64 laptops more, and we touch on the popularity of Windows Subsystem for Linux (WSL).

6. Engineering culture. A skeptical attitude to AI at Canonical has given way to one where experimentation is encouraged and devs lean into AI tools, but there are no targets for token usage or amounts of AI-generated code.

7. What other Linux distributions are doing. Arch Linux takes the “DIY your AI setup” approach, Omarchy makes it easy to install AI tools, while Red Hat Enterprise Linux ships with AI integrated into the command-line and support for AI accelerators & popular AI tools.

The bottom of this article could be cut off in some email clients. Read the full article uninterrupted, online.

Read the full article online

1. Hardware enablement: support for GPUs, NPUs & DPUs

Jon mentioned he detects a “Dotcom Boom”-era vibe in the industry, like around when “web 1.0” was created, and indeed, lots of startups today aim to be the Google-style success story of this “AI era”. At Canonical, the team asked: what does that mean for Ubuntu as an operating system?

For instance, should Ubuntu join the competition and try to position itself closer to AI, or keep focusing on what they’ve done for decades: build an operating system? Jon said:

“We need to make sure to remain a relatable and accessible system. I don’t think we should blur the line between application features and the OS itself. So, the most powerful thing we can do is hardware enablement.”

Hardware enablement means that if a computer (typically, a laptop) has AI-related hardware, Ubuntu should allow it to make full use of it. This involves adding support for GPUs, NPUs, DPUs and other types of accelerator cards. Let’s briefly go through each.

GPUs

As is likely widely known by readers, ‘GPU’ stands for Graphics Processing Unit. Originally built for graphics rendering, its #1 use case is no longer in video games but for AI training and inference. GPUs come in two forms:

• Integrated GPUs: located on the same die (integrated circuit) as the CPU, like GPUs on Apple’s M-series processors

• Discrete GPUs: separate chips on their own board; often for gaming, or in standalone GPU rigs for AI and ML workloads

NVIDIA leads the market in discrete GPUs for rigs with its Blackwell family, and in standalone GPU cards with the NVIDIA RTX series. Other vendors like AMD offer GPUs for data centers (like the Instinct MI300 Series) and for PCs with the AMD Radeon series.

NPUs

Neural Processing Units (NPUs) are also called “AI accelerators.” This is a dedicated block on the System-on-a-chip (SoC), on modern processors especially designed for running AI inference efficiently on‑device. Since 2022, many modern processors have had a dedicated NPU block, including all Apple’s M-series chips (from M1 and up), Intel’s Core Ultra and Core Ultra “Series 2”, AMD’s Ryzen AI 300 series, and also Qualcomm’s Snapdragon X Elite and Snapdragon X Plus.

A number shared for each NPU is TOPS. TOPS means Tera (trillions) of Operations Per Second, and the said operation is a “multiply-accumulate” (MAC) one, which Qualcomm describes as:

“A multiply-accumulate (MAC) operation executes the mathematical formulas at the core of AI workloads. A matrix multiply consists of a series of two fundamental operations: multiplication and addition to an accumulator. A MAC unit can, for example, run one of each per clock cycle, meaning it executes two operations per clock cycle. A given NPU has a set number of MAC units that can operate at varying levels of precision, depending on the NPU’s architecture.”

How TOPS is calculated: TOPS = 2 × MAC unit count × Frequency / 1 trillion.

“Frequency” refers to the clock speed (cycles per second) at which an NPU and its MAC units (as well as a CPU or GPU) operate, which directly influences overall performance. Processors at higher frequencies allow for more operations, but higher frequencies also mean more energy consumed, heat generated, and battery life decreased. The TOPS number that’s quoted for processors is generally the peak operating frequency.

NPUs are often ideal for low-power, local inference, and for running smaller, local models. They can be useful for things like Local speech‑to‑text (dictation, captions, meeting transcription), video background blur/replacement or auto‑framing, small local language summarization, etc. NPUs are more typical of laptop and PC processors, although some phone processors ship with them like the iPhone (A-series chips) and Google’s Tensor processor in Pixel phones. Basically, NPUs promise to bring efficiently-running local models on laptops one step closer.

DPUs

Data Processing Units (DPUs) are typically found in data centers, moving massive amounts of data fast. NVIDIA’s explanation:

“The CPU is for general-purpose computing, the GPU is for accelerated computing, and the DPU, which moves data around the data center, does data processing.

A DPU is a new class of programmable processor that combines three key elements. A DPU is a system on a chip, or SoC, that combines:

• An industry-standard, high-performance, software-programmable, multi-core CPU, typically based on the widely used Arm architecture, tightly coupled to the other SoC components.

• A high-performance network interface capable of parsing, processing and efficiently transferring data at line rate, or the speed of the rest of the network, to GPUs and CPUs.

• A rich set of flexible and programmable acceleration engines that offload and improve applications’ performance for AI and machine learning, zero-trust security, telecommunications, and storage, among others.”

Several major chipmakers manufacture DPUs, of which NVIDIA’s BlueField family is the most widespread. Others include AMD Pensando DPUs (Elba, Giglio), and Intel IPU / DPU cards (E2100, E2200 series).

DPUs are most commonly deployed inside Hyperscale cloud providers (AWS, Azure, GCP, OCI), or in AI and high-performance computing (HPC) data centers, or larger private clouds. DPUs make sense when GPU traffic is huge, or when the network telemetry overhead is so great that it could overwhelm the CPUs processing the data transfer.

2. Hardware partnerships

It’s easiest to add support to hardware by working with leading chip manufacturers, so Ubuntu has relationships with hardware vendors for that reason. As a result, the OS sometimes offers day-one support for cutting-edge AI supercomputers.

Partnership with NVIDIA

In September 2025, Canonical announced it would package and distribute the full NVIDIA CUDA toolkit directly within Ubuntu’s repositories. This deal collapsed into a single standard apt install, something that had previously been a multi-step manual installation process of downloading from NVIDIA’s site, importing GPG keys, pinning a separate APT repo – and praying nothing broke.

Packaging and distributing the CUDA toolkit makes developing with CUDA easier. From Jon:

“One of the trickiest things for developers who have to use this tech is the dance of matching the right version of Python, with the right version of CUDA, with the right driver. Projects end up with different versions of CUDA, and then machines end up breaking because the driver configuration gets inadvertently broken along the way.

The number one thing we can do as an operating system is to make this setup as easy as possible.”

Ubuntu’s strategy of working directly with chipmakers seems to be working. NVIDIA recently discontinued its custom NVIDIA DGX OS — a modified Ubuntu it maintained for years — and now ships plain Ubuntu. Jon:

“Previously, NVIDIA shipped NVIDIA DGX OS for which NVIDIA had an agreement with Canonical where they could take Ubuntu, modify it with the kernel modules and software they needed, do some product-specific optimization, and ship that as NVIDIA DGX OS.

This more recent development sees NVIDIA just shipping Ubuntu as it comes.

When NVIDIA released the DGX Spark, a $4,000 AI workstation with an ARM64 chipset, it shipped running vanilla Ubuntu as the only supported operating system.”

At CES 2026 in January, Canonical announced Ubuntu support for the NVIDIA Vera Rubin NVL72 rack-scale architecture, with day-one platform readiness in Ubuntu, version 26.04 LTS (Long-Term Support: at least 15 years for enterprise customers).

AMD and Intel

It’s clear Ubuntu and NVIDIA enjoy a strong partnership, but Canonical aims to remain neutral, Jon says:

“We have an amazing partnership with NVIDIA, but we do the same with Intel, the same with AMD, the same with Qualcomm, and the same with MediaTek because in reality there is hardware being released every day, and if we don’t maintain those partnerships, the ecosystem becomes even more fragmented than it already naturally is.”

Last December, Ubuntu announced native support for AMD ROCm, and also ships with Intel’s OpenVINO toolkit. Ubuntu 26.04 LTS will be the first major distribution to natively package all three GPU compute stacks — NVIDIA, AMD, and Intel — with long-term enterprise support. Under Ubuntu Pro, ROCm LTS releases receive up to 15 years of security maintenance.

Security maintenance means that if vulnerabilities or critical incompatibilities are discovered in an LTS version, Canonical will patch them even if the upstream vendor no longer supports those versions and no longer backports security patches.

AMD Instinct accelerators are gaining traction in HPCs and sovereign AI deployments, as enterprises look for alternatives to CUDA-locked hardware. AMD’s SVP and Chief Software Officer, Andrej Zdravkovic, said the partnership would make it “easier for developers and enterprises to deploy AMD solutions on supported systems.”

Chip vendors want to collaborate because it means less work for them to add operating system-level support. Jon:

“It’s a win-win on both ends. Silicon companies are in the business of building the best chips they can, and partnering with Canonical means they have to concentrate on fewer things which are not their core focus. My hope is that partnering with Canonical helps them to focus on what they’re best at, while enabling us to help with what we’re best at: integrating, shipping and maintaining a Linux distribution.”

3. Architecture variants

Modern x86 processors support multiple instruction set generations: x86_64 v1, v2, v3, v4, and v5. ARM has a similar hierarchy. Each generation adds capabilities, such as AVX-512 instructions that accelerate machine learning workloads.

Let’s take the x86_64 instruction set. The instruction set is versioned. These are the versions:

• For x86_64: v1, v2, v3, v4, v5…

• For ARM: ARM v8.2, v8.3, v9…

Until recently, Ubuntu ran slower on newer CPUs in order to keep supporting older ones. So, when installing Ubuntu compiled for AMD64, the OS supported architecture variants for AMD64 v1.

Supporting v1 has the advantage that the oldest of AMD64 processors can run this Ubuntu version. But if Ubuntu decided to support v2 instructions, then v1 processors could not run the OS! The OS did not use the new instructions; for example, a modern processor with hardware accelerators like AVX-512, didn’t use them.

Canonical has reworked its build infrastructure to produce binaries with specific architecture variant support. So, in the case of running an x86_64 v3 compatible processor, you can download an Ubuntu OS variant that’s compiled specifically for x86_64 v3.

One tradeoff the Ubuntu team had to make was building binaries several times, which takes up more processing time and storage at their end. Then again, the Ubuntu team doing this once means that users don’t need to do recompilation, which made it an easy tradeoff, Jon told me.

Now, Ubuntu supports x86_64 v3 as an architecture variant and plans to do more. Jon says:

“Today, we’ve released x86_64 v3 as a variant, but the capability in our build and delivery pipelines unlocks the ability to add variants for the next RISC-V RVA versions, for ARMv9, ARMv10, ARMv11 and so on.

We will start now onboarding variants to make sure that when you go and buy your latest Snapdragon laptop, your operating system and all of the parts of it are using the silicon to its fullest.”

Adding support for architecture variants was a significant undertaking. Jon explains:

“This work was especially complex because combined with having the hardware physically available in the build farm, Canonical also needed to make the build scheduler aware, and thread the capability through the build systems of Debian packages, Snaps, OCI images, virtual machine images, etc. As it stands, the capability exists for Debian packages, and support for further package types will land shortly.

In addition to the build infrastructure, work needed to be done on downstream package managers (apt, snap, …) and schedulers to ensure they pull the right version of packages, and consideration needs to be given to what happens if a VM containing x86_64 v3 code ends up trying to boot on v1 hardware, and so on.”

4. Betting on local-first & plans for agentic workflows

If you’ve tried to run an LLM locally on your machine, you’ll know it comes with friction. Jon:

Read more

Tech interviews have two parts: the technical interview – with a focus on things like coding, software architecture, problem solving – and the behavioral part – with a focus on past experience, and the situations that show you’d be a good fit at the company you’re interviewing with, along with things like attitude, motivation, culture fit. Technical interviews are going through a big change, thanks to AI tools: some companies are bringing in new, AI-assisted types of interviews, while others are trying to make “pre-AI” type interviews work.

What doesn’t seem to be changing is the second type of interviews: the behavioral ones. I’ve found the topic of behavioral interviews from a software engineer’s perspective somewhat under-discussed – even though this interview carries huge weight in securing an offer and what level you come in at. No matter how strong your technical skills are, especially at mid-sized and larger companies, you are unlikely to get an offer if you are deemed to not be a fit for what the company is looking for.

Steve Huynh was an engineer at Amazon for 17 years – I previously did a podcast episode with him on the reality of being a principal engineer at Amazon. During this time, Steve conducted nearly 1,000 interviews, of which around 600 were Bar Raiser ones. Bar Raiser interviews are unique to Amazon: it’s an interview conducted by someone outside of the hiring team, with the goal of ensuring that the new hire raises the company’s talent bar.

After leaving the e-commerce giant, Steve spent 2 years researching and writing the book Technical Behavioral Interview: An Insider’s Guide.

Today, we cover two topics on interviews and behavioral interviews:

1. Learnings from conducting ~1,000 behavioral interviews at Amazon. Steve reflects of major observations from his 17 years at Amazon, covering:

• You’re over-prepared for one interview and unprepared for the other

• How you deliver the story matters as much as the story itself

• The interview is an audition for what it’s like to work with you

2. What companies are looking for during behavioral interviews. An excerpt from Steve’s new book, Technical Behavioral Interview, covering ~75% of a full chapter of the book (out of the 14 total chapters.) We get into:

• Understanding fit: role and company

• The four dimensions that determine your level

• What each level looks like

• Reading and calibrating your own level

• Researching what companies really value

Longtime readers might remember Steve from my podcast with him a year back: What is a Principal Engineer at Amazon? With Steve Huynh

My usual disclaimer: as with all my recommendations, I was not paid for this article, and none of the links are affiliates. See my ethics statement for more.

With this, it’s over to Steve:

1. Learnings from conducting ~1,000 behavioral interviews at Amazon

A Bar Raiser is a specially trained interviewer whose job is to ensure that every hire raises the average talent level at Amazon. I had veto power over any candidate. I sat on nearly a thousand interview loops across every level from intern to Principal Engineer.

After 50 or so interviews as a Bar Raiser, the patterns became impossible to miss. And this was the biggest one:

The candidates who didn’t get offers seldom failed because they lacked technical skill. They failed because of how they presented themselves.

For sure, technical preparation is crucial, and I’m not telling you to skip it. But most candidates have massive blind spots when it comes to non-technical matters, which is a big problem. Why? Because that blind spot is where most hiring decisions are made.

The Bar Raiser who trained me put it this way:

Technical skills are the ante. They get you into the game. But they’re not what wins you the hand.

I didn’t fully appreciate what that meant until I’d seen candidates who were technically very strong get rejected because of everything else.

Think about it. By the time you’re sitting in a final round of interviews, you’ve already passed at least one technical screen or take-home assignment. The company already knows you could probably do the job. They already know you want to work with them.

But that’s not what the final round is for.

The final round is when the team figures out whether they want to work with you. Being technically proficient is part of it, but it’s not all of it. Can you explain your thinking clearly when you’re stumped? How do you handle it when things go wrong? Can they picture you in a design review or in a tough conversation with a partner team?

Fit.

Fit is what decides most hiring outcomes, yet it’s the thing most candidates spend the least time preparing for. After nearly a thousand interviews, I can tell you exactly where the gap is and how you can close it.

Learning #1: You’re over-prepared for one interview and unprepared for the other

The average candidate preparing for a tech interview probably spends 95% of their time on technical preparation and 5% on everything else. Some spend literally zero on everything else.

I get why. Technical preparation feels concrete. You can grind coding problems and measure your progress. You can study system design patterns and feel yourself getting sharper. There’s a clear input/output relationship. Do more problems, get better at problems.

For most technical interviews, even if you haven’t seen the exact problem before, you can still do a decent job. It’s simply not possible to prepare for every problem, so it’s expected that you can reason through an unfamiliar coding question and pick up on hints the interviewer gives you. You can work through a system design problem by applying fundamentals you already know. It’s expected that you will encounter new questions during an interview, so it isn’t fatal if you’re a competent engineer who can think on your feet.

However, the non-technical rounds are the opposite. You cannot wing them and expect to do well. When an interviewer says, “Tell me about a time something went wrong on a project and how you handled it” and you haven’t thought about that question before, there is no hint they can give you. There’s no reasoning your way through it in real time. You either have a prepared story ready to go, or you’re going to mumble your way through a word salad while the interviewer watches.

I’ve seen this play out hundreds of times. A candidate would crush the coding round, then I would ask them about a difficult decision they made, and they would fall apart. They would pick a half-remembered example, start rambling, backtrack to add context they forgot, in the process losing track of the question. Then, five minutes later, they would land on something like, “So, yeah, it worked out in the end.”

These candidates were often strong coders, but that didn’t matter. At the debriefs, the feedback was always some version of “I couldn’t get a concrete answer about their experience. Every story was vague and unconvincing.” We couldn’t extend an offer when a candidate couldn’t articulate how they worked.

The technical bar was met, but the hiring decision was made in the behavioral round.

Here’s what’s frustrating about this. Non-technical preparation takes a fraction of the time for technical.

If you’re going to spend 80 to 100 hours preparing for an interview cycle, spending a single weekend on your stories might be the highest-leverage investment you make.

Ten hours of story prep can completely change the outcome of your behavioral rounds. Meanwhile, your 80th hour of LeetCode will give you almost nothing you didn’t already have at 60.

The returns on technical prep diminish rapidly. The returns on story prep are exponential because almost nobody does it at all.

What to do: How are you currently splitting your interview prep time? If it’s 99% technical and 1% everything else, you’re over-indexed on the part with diminishing returns and under-indexed on the part where hiring decisions get made. You don’t need to cut your technical prep dramatically. Just reallocate. If you’re planning to spend 80 hours preparing, take 10 of those hours and move them to non-technical preparation. That reallocation will do more for your odds than 10 more hours working on practice problems.

Learning #2: How you deliver the story matters as much as the story itself

You can have the most impressive accomplishment of your career ready for your interview and completely waste it with bad delivery. The most common version of this is what I call the “ramble and stumble.”

The candidate starts talking, and you genuinely can’t tell if they’re figuring out the story as they go or if they’ve simply never said these words out loud before. Or they might give you five minutes of context and then still backtrack to add details they forgot. By the time they reach the outcome, you’ve lost track of how you got there.

Here’s something that’s always struck me as odd. If you had a big presentation at work, you’d spend hours preparing for it, right?. You’d think about the structure, the flow, the key points. You’d rehearse it. You might even do a couple of dry runs with a colleague. Nobody wants to walk into a presentation and wing it.

But in a job interview, where the stakes are arguably higher than any single presentation you’ll ever give? People wing those constantly. They walk in having never practiced their stories out loud. They might have thought about them, but they’ve never spoken the words, heard how they sound, or timed how long they take. Then they’re surprised when the words come out as a mess.

Think about any other high-stakes skill. You wouldn’t expect to be good at golf without practicing at the driving range. You wouldn’t expect to give a great keynote the first time you stepped on stage. Nobody calls a musician fake for rehearsing before a concert.

But for some reason, many people feel that preparing interview stories is inauthentic. As if it’s cheating somehow. As if the “real” version of you is the one that stumbles through an unrehearsed answer under pressure.

It’s not. The real you communicates clearly what you’ve done and what you’re capable of.

What to do: Good delivery doesn’t require a lot of charisma or natural presentation skills, but it does require practice. Start with the two questions that come up in virtually every interview: “Tell me about yourself” and “Why do you want to work here?” Write down your answers. Then record yourself delivering them. Watch the recording and take notes. Where did you ramble? Where did you fill space with filler words? Did you look nervous? Then do it again. And again. Keep going until you watch the recording back and think “That sounds like someone I’d like to work with.”

Once those two are solid, pick stories from your career and do the same thing. This process will be uncomfortable at first. Most people hate watching themselves on camera. Do it anyway. Thirty minutes of this will up-level your interview performance much more than 20 hours of coding exercises could ever do.

Learning #3: The interview is an audition for what it’s like to work with you

Most candidates think the interview is an exam. If you get the right answers, then you’ll pass the test and get the job. That’s simply not how it works. Yes, you are being evaluated, and what you say matters. But there is no answer key. The interviewer doesn’t have a rubric with the “correct” responses to which they compare your answers. They’re forming an impression of you as a person, and that impression is far more nuanced than “right” or “wrong.”

By the time you’re sitting across from the interviewer, you’ve already jumped through some technical hoops. The company already has evidence from your resume that you can code or design systems at the level they need. That bar has been cleared. The final round goes deeper on the technical side, but it’s also trying to answer a completely different question: Would we want this person on the team? Would we trust their judgment in a crisis? Would they make our team’s software better or worse?

As a Bar Raiser, my specific job was to determine whether a candidate would raise the bar, meaning that they would be better than at least 50% of the people already at the company in that role.

The thing most people don’t realize is that the type of coding we asked about in interviews wasn’t what we did on the job. Nobody was writing algorithms on a whiteboard during their workday. The questions we asked tested problem-solving ability in an artificial environment.

But the behavioral questions, the soft questions, those tested situations we dealt with every single day. Navigating disagreements, handling projects that were going sideways, influencing without authority, making tradeoffs with incomplete information. These weren’t hypothetical scenarios pulled out of a textbook. They were just another Tuesday.

So when I asked a candidate to tell me about a time they had to push back on a stakeholder, I wasn’t waiting to hear the right answer; I was picturing them in our next planning and prioritization meeting. When they described how they handled a conflict on their team, I was asking myself whether I’d want to be in that room with them. Every answer was a preview of what it would be like to work alongside that person day to day.

The candidates who treated it like a test tried to figure out what I wanted to hear and then gave me that answer. That’s exactly the wrong approach. They gave polished, rehearsed answers with no rough edges and perfect endings where everything worked out and every decision was the right one. I’d walk out thinking “I have no idea what it would actually be like to work with this person.” And when that uncertainty showed up across multiple interviews in the debrief, it almost always turned into a “No.”

What to do: For each story you’re preparing, stop thinking about what the interviewer wants to hear. Instead, think about what you’d want to hear from someone interviewing to join your team. You’d want to hear how they actually think. You’d want the real version of what happened, including the parts that were hard and the calls that were close. You’d want to walk away feeling like you understood what it would be like to work with them on a tough problem. Give your interviewer that same thing. Be honest and let them see how you think. That’s worth more than any polished answer.

What ~1,000 interviews taught me

After all those interviews, the lesson I keep coming back to is simple.

The people who get hired are the ones who can walk into a room and tell a clear story. This story is about their work and their capabilities, and makes the interviewer think, “I want to work with that person.”

Being able to tell this story is a skill. And like any skill, it gets better with practice. Most people never practice it because they don’t think of it as something you can prepare for, but you can. And a little preparation here goes further than almost anything else you can do for your career.

2. What companies are looking for during behavioral interviews

The below are excerpts from Chapter 2 from Technical Behavioral Interview: An Insider’s Guide. Some sections have been cut out and lightly edited for this article. Copyright © 2026 Steve Huynh. Used with permission.

Technical skills alone don’t determine your offer. Otherwise, those who can solve the coding and system design problems would get the same result. Instead, companies use behavioral interviews to answer two critical questions: Do you fit with both the role and the company? And if you do fit, at what level will you be most effective?

Get both right, and you will receive an offer at the appropriate level. Get the fit wrong, and you’ll be rejected regardless of your skills. Get the level wrong, and you’ll be either down-leveled or rejected for being underqualified.

This chapter explains how companies make their assessments of fit and level by analyzing the signals in your stories. Once you understand these dimensions, you’ll pick better stories and signal the right level.

Understanding Fit: Role and Company

The primary consideration for any tech role is whether you have the technical skills to do the job. Companies will assess this mostly through the technical parts of the interview, for example, coding challenges, system design, or whatever technical evaluation matches your role. If you can’t demonstrate the core technical capability, nothing else matters.

But technical skills alone don’t predict success. Companies learned this the hard way by hiring smart people who couldn’t work effectively in their environment. That’s why behavioral interviews focus on two additional types of fit:

Role Fit: Can you handle the specific challenges and working conditions of this position? A backend role at a fast-growing startup requires different capabilities than a backend role at an established enterprise. The technical skills might be similar, but the role demands will be different.

Company Fit: Will you thrive in the environment in which this organization operates? This goes beyond surface-level culture. They are assessing whether your working style, decision-making approach, and values match with how the company gets things done.

How Companies Detect Fit Through Signals

Companies can’t directly ask the question, “Would you fit here?” What candidate would torpedo their chance of success by answering with a “No”? Instead, companies look for signals in your stories that indicate alignment or misalignment.

Role Fit Signals emerge from how you describe handling situations similar to what the role requires:

• If the role requires working with ambiguous requirements, do your stories show comfort with uncertainty?

• If the position involves cross-team coordination, do you show an ability to cope with organizational complexity?

• If the job needs rapid iteration, do your examples show shipping quickly and adjusting based on feedback?

Company Fit Signals come from the choices you made and how you describe them:

• A company that values “bias for action” looks for stories that show you moving quickly despite incomplete information.

• An organization that prizes “customer obsession” wants to hear examples of you going deep to understand user needs.

• A place that emphasizes “radical transparency” seeks stories that show you sharing information openly, even when you’re uncomfortable.

The same story can send different signals to different companies. You spending three weeks perfecting a solution might demonstrate attention to quality at one company but analysis paralysis at another. Moving fast and fixing issues later demonstrates good judgment at a growth startup but recklessness at an established healthcare company.

Common “Mis-Fits”

Even a talented candidate will get rejected sometimes if they are not a good fit. The same behaviors that are positive at one company can signal poor fit at another.

Independence vs. Collaboration: This covers both how you work and how you make decisions. Some companies need people who pick up a problem, run with it, and come back with a solution. Others expect you to bring the team along at every step. These often go together: companies that want you to work solo also tend to want you to make calls on your own, and companies that want collaborative work also want group buy-in on decisions.

If every story you tell involves going off and building something alone, consensus-driven companies will worry you’ll steamroll people or make choices that won’t stick. Flip it around: if every story involves checking with the group before you act, companies that prize individual ownership will wonder whether you can make a decision without a meeting.

Speed vs. Thoroughness: Startups often need rapid experimentation, where you ship MVPs and iterate based on feedback, while companies in healthcare or finance require careful validation before any release. This tension also shows up in how teams think about code quality: some organizations will happily spend extra weeks on clean architecture, while others want a working solution on deadline even if the code needs cleanup later. Whereas stories about methodical testing might bore a startup, your “ship it and fix it” examples could terrify a medical device company.

Excellence vs. Pragmatism: Some organizations value technical excellence and clean architecture above all else. Others need pragmatic solutions that ship on deadline even if imperfect. Focusing on perfect code fails at deadline-driven companies, just as accepting technical debt everywhere fails at companies maintaining critical infrastructure.

Innovation vs. Stability: Some roles require creating new solutions and challenging existing approaches, while others need you to maintain and optimize proven systems. If you say that you’re constantly reinventing established processes, teams that value stability will not consider you a good fit. Conversely, stories that show you only follow existing patterns will disappoint teams that are looking for creative problem-solving

Direct vs. Diplomatic: Some cultures prize radical candor and want you to say exactly what you think. Others value maintaining harmony and face-saving communication. If you are too blunt, you will not fit in well at a relationship-focused company. If you are not direct enough, you will not like working at a company that values “disagree and commit.”

Data vs. Intuition: Some companies require data to justify every decision (”data-driven” cultures), while others trust experienced judgment and move on gut feel. Showing that you make decisions based on instinct does not impress analytical companies, and telling a company that values experienced judgment that you conduct three A/B tests to choose a button color will get you struck off their list.

Specialist vs. Generalist: Large companies often want deep experts who master one domain, while smaller companies need people who are comfortable wearing multiple hats. Know which sort of company you are walking into.

Once you understand fit, you can pick stories that match the company and the role.

The Four Dimensions That Determine Your Level

Companies assess your level through four dimensions that appear in every story you tell. Each dimension reveals different aspects of your capability. Together, they show the company where you operate most effectively.

Scope (Dimension #1)

Scope provides a measure of the number of people on your team and, extending outward as you advance, whose work was affected by your actions. The greater the number affected, the higher your level for this dimension.

Entry Level: Your work affects your own productivity and starts to help other team members. For example, you might improve how you handle assigned tasks or fix issues that were slowing down a few teammates.

Mid Level: Your work affects aspects of the team and shapes how it operates. You might redesign a process that changes a significant part of how your team works or solve problems that affect most of the team’s effectiveness.

Senior Level: Your work directly impacts your entire team and is beginning to influence at least one other team. Perhaps you create solutions that change how your whole team operates and affect workflows in adjacent teams, or you solve problems that require coordination with other groups. You may also start collaborating more closely with product or design partners on your immediate team’s work.

Staff Level: Your work directly impacts at least two teams and is beginning to have an influence on the broader division or organization. Examples of this include developing technical strategies that change how multiple teams make decisions and solving problems that require buy-in across several parts of engineering. Your influence extends beyond engineering into product, design, and program management as you shape solutions that affect how cross-functional partners work.

Principal Level: Your work affects many teams or changes how large parts of the organization operate. Perhaps you have created technical strategies that have influenced how dozens of teams make decisions. Or you have solved problems that cut across a large engineering organization. At this level, your influence regularly extends into business strategy, shaping decisions alongside product, design, program, and business leadership.

Contribution (Dimension #2)

Contribution captures what you did, not what happened around you. It is important to be precise about the line between “I” and “we.” Companies will expect to see evidence of increasing leadership and ownership as you advance in your career.

Entry Level: You execute assigned work and are beginning to take ownership of small pieces. Examples: implementing solutions designed by others; fixing bugs in existing systems; taking full responsibility for well-defined features within larger projects.

Mid Level: You own complete solutions from problem to implementation while also guiding others. Perhaps you have identified issues, designed the approaches, implemented them, and you have verified that they work, and you have helped your teammates understand the reasons for your decisions.

Senior Level: You lead initiatives requiring coordination. You’re expected to make progress even when the requirements are unclear or the path forward is uncertain. Examples of this include driving technical decisions for your team; mentoring others through complex problems; architecting solutions to be implemented by others; and ensuring quality work outcomes for many people.

Staff Level: You lead cross-team initiatives and establish technical direction, often in situations where the right approach isn’t obvious and stakeholders have competing priorities. This could look like defining technical approaches that are adopted by multiple teams, creating systems that enable other teams to solve problems on their own, or driving agreement on complex technical decisions across several teams.

Principal Level: You create organizational capabilities and establish new ways of working. At this level, you’re frequently operating in highly ambiguous environments where you must define the problem before you can solve it. You might define technical standards that guide dozens of teams, build systems that enable others to solve entire classes of problems, or transform how the organization approaches its hardest challenges.

Impact (Dimension #3)

Impact shows what changed for the better as a result of your work. Companies want to see that your work produced results worth the investment. Strong stories put numbers on the impact and connect technical wins to business or user outcomes.

Entry Level: You improve your personal productivity and are starting to help the team work better. Examples include reducing the time you spend on repetitive tasks, fixing issues that were slowing down teammates, or improving the quality of code in the areas you touch. Even simple measures matter at this level: time saved or bugs prevented.

Mid Level: You improve team effectiveness in specific areas and influence team-wide practices. Perhaps you reduced deployment times for specific workflows, eliminated categories of bugs in your domain, or you created tools that have made the team more productive in particular areas. You can quantify these improvements and connect them to broader outcomes like feature velocity or reliability.

Senior Level: You transform how your entire team works and are starting to have an impact beyond your team. For example, you might have introduced new workflows that changed your team’s capabilities. Or perhaps you eliminated major sources of operational problems, or the improvements that you have created have been adopted by adjacent teams. Your impact extends beyond just engineering metrics to product outcomes, user experience, or operational costs.

Staff Level: You improve how multiple teams operate and drive organizational improvements. These sorts of impact come from achievements such as establishing practices that several teams adopt, solving infrastructure problems that were impeding multiple teams, or creating new capabilities that open up new types of work across teams. Your measurable impact can be tied to business metrics like revenue, customer retention, or time-to-market.

Principal Level: You create organizational capabilities and drive strategic changes. Impact at this level could come from establishing technical foundations that dozens of teams use to build upon, solving problems that were blocking major business initiatives, or creating leverage that compounds benefits across the company. Your impact is measured in business outcomes and strategic capability, not just technical improvements.

Difficulty (Dimension #4)

Difficulty reflects the complexity of problems you’ve tackled, the constraints you have faced, and the trade-offs you have managed. Under this category, solving easy problems with big impacts is less impressive than hard problems solved well.

Entry Level: You work on straightforward problems within established patterns. For example, you might face challenges learning new technologies or debugging unfamiliar code, but the path forward becomes clearer once you understand the problem or ask for help.

Mid Level: You work through challenges and obstacles in your work. The problems you tackle have more moving parts and less obvious solutions. These could be competing requirements or having to work through technical complexity you haven’t seen before. Or perhaps you have had to manage dependencies within your team that affected your timeline or figure out solutions when the approach wasn’t immediately obvious.

Senior Level: You manage constraints and make technical decisions with team-level architectural implications. The problems you solve involve multiple interacting systems and competing concerns. You might have to balance needs across multiple stakeholders with different priorities. Maybe you make architectural decisions that affect how your whole team works, or you have to work around technical limitations that require creative solutions, or solve problems that require you to address both technical and business factors.

Staff Level: You manage competing trade-offs across multiple teams while handling problems with significant technical and organizational complexity. Examples of difficulty at staff level include:

• Balancing different technical approaches when teams have genuinely conflicting needs.

• Creating solutions that affect how several teams work together.

• Making architectural decisions that have to work across diverse contexts.

• Getting teams to agree when the technically optimal solution differs for each team.

Principal Level: You handle fundamental trade-offs between competing organizational needs or solve problems where no clear solution exists. The complexity at this level often involves novel problems that lack established patterns or precedents. You might balance technical excellence against delivery speed at organizational scale; work within organizational constraints while maintaining technical integrity; create approaches for entire classes of problems the company hasn’t solved before; or make decisions that affect company strategy and require executive buy-in.

What Each Level Looks Like

Here’s how the same types of accomplishments look across each level. These aren’t templates. They’re meant to help you develop a sense for the difference between a mid-level story and a senior one. Compare adjacent levels and notice what actually changes as you move up and down.

Researching What Companies Really Value

You’ll never have perfect information about what a specific company values, but a little focused research will often reveal surprising insights that most other candidates will miss. The difference between having even partial intelligence and going in blind can be whether or not you emphasize the right things in your stories.

Start With Your Recruiter

Most candidates treat recruiters as gatekeepers to avoid, but if you do this, you will waste your best source of insider information. Recruiters want you to succeed, because their performance is based on the number of accepted offers received by the candidates they put forward. They have prep materials, they know the interviewers’ focus areas, and they understand what they are looking for.

Ask your recruiter directly: “What should I know about this company’s current challenges?” Or “What competencies matter most for this role?” Or “Can you share any interview prep materials?” Many recruiters have documents about interview format, team priorities, or even the specific behavioral competencies they evaluate. The questions that are used as examples in the prep materials have a high likelihood of being asked in the interviews.

Mine Publicly Available Information

When companies repeat certain words when describing job opportunities, they’re telling you what matters. For example, a job posting that mentions “fast-paced” several times signals something different than one emphasizing compliance. Those words are there for a reason.

Where to dig:

• Engineering blogs: How do they describe their wins? What problems do they celebrate solving?

• Tech talks and conferences: What topics do their engineers present? Speed of delivery? Scale? Innovation?

• Open source contributions: What they choose to open source reveals their priorities. If they open source developer tools, this suggests they value community. If they are happy to make internal tools public, this shows transparency.

• Technical documentation: The existence of public API docs or technical guides (and the quality thereof) shows how they support both users and their own teams.

• Status pages and postmortems: Companies that publish detailed postmortems demonstrate that they value learning from failure. A company that shares their incident response processes likely has a strong operational culture.

Even companies without engineering blogs will leave traces. Product release patterns tell you about their development pace. Technology choices show their priorities: newer frameworks suggest a focus on innovation, whereas relying on proven technologies indicates they prefer stability.

Look for Patterns in Discussions

Glassdoor, Blind, and Reddit contain gold buried amongst rubble. Ignore the rubble (e.g., individual rants). Instead, look for patterns across multiple posts. If five different people mention “lots of process” or “no work-life balance” or “amazing learning culture,” that’s a pattern you will want to know about.

Pay attention to what people complain about and what they praise. Complaints about “too many meetings” may suggest the company has a collaborative, consensus-driven culture, or, alternatively, that productivity within the company is inhibited by an excessive number of meetings. Praise for “autonomy” indicates they trust their people to make decisions without checking in. Both types of comments reveal what behaviors the companies will reward.

Talk to Current Employees

If you know someone at the company, ask them directly what behaviors get rewarded and, conversely, what behaviors will cause people to struggle. Skip surface-level queries about culture, and ask specific questions:

• “When someone gets promoted here, what do they do to earn it?”

• “What behaviors get negative feedback?”

• “How does the team make decisions when there’s disagreement?”

• “What surprised you most about working here?”

Current employees will tell you truths the company website never would. Perhaps they’ll tell you that at their company, “customer obsession” really means checking usage data before writing code, or that “ownership” means being available to resolve production issues at two o’clock in the morning.

What You’re Really Looking For

All this research serves one purpose: understanding what stories will resonate at your interview. Think of it as finding the real intersection between your experience and what they care about.

If research reveals they prize speed over perfection, then emphasize stories that tell how you shipped quickly and iterated. If they value technical depth, highlight examples of diving deep to understand root causes. If they care about collaboration, make sure your story focuses on cross-team work rather than solo accomplishments.

The research will also help you decide whether this company is the right place for you. If everything you learn suggests they value the kinds of behaviors you don’t naturally demonstrate or don’t want to develop, then perhaps you don’t need to pursue that particular role.

Putting It All Together

Companies aren’t just evaluating whether you can do the job. They’re also assessing whether you’ll thrive in their specific environment and at what level you’ll be most effective. These two dimensions determine not just whether you will get an offer, but also whether that offer will position you for success.

Understanding fit helps you know which of your experiences will connect most with what the company values. This small company needs someone who ships fast and figures things out alone. That enterprise needs someone who navigates processes and builds consensus. Neither is inherently better than the other. They’re simply different environments that reward different approaches.

Understanding levels helps you position your stories appropriately. The same project can demonstrate entry-level execution, mid-level ownership, or senior-level leadership depending on your actual contribution and how you frame it. Get this wrong and you will either get rejected for overreaching or down-leveled for not properly communicating your capabilities.

The payoff is immediate. You’ll pick better stories, focus on the right details, and make it easier for interviewers to see what you can do. You’ll make better decisions about which roles actually match who you are and what you want to do. The goal isn’t to get any offer. The goal is to get the right offer at the right level at the right company to ensure your success.

Takeaways

Gergely, again. Thanks to Steve for both sharing his learnings, as an interviewer, and for sharing nearly a full chapter from his whole book. The book goes a lot deeper than the above sample chapter. A few of the ones I found helpful:

• High-signal storytelling (Chapter 3): a framework for explaining your work in a way that “sticks” with the interviewer

• 9 competencies with many examples and stories throughout the book: ones like “delivery” (Chapter 6), “earning trust and dealing with conflict” (Chapter 8) and “Strategic leadership and thinking big (Chapter 13)”

• Examples of what interviewers typically see as key signals, yellow flags and red flags

If you would like to have a fresh resource to prepare for behavioural interviews at tech companies, the full book offers far more explanations, tactics and exercises to do so:

Get the full book on Amazon

Steve also writes a newsletter titled A Life Engineered: you can sign up to it here.

It’s helpful to understand how and why companies hire, and what they look for. To us engineers, hiring processes often look illogical from the outside. We’ll ask things like:

• “Why does the interview process not resemble day-to-day work?”

• “I already have open source code I wrote: why does the company need to do a coding interview to confirm what is clear: that I need to code?”

• “Why did I get a rejection, even though I did well on all of the interviews?”

It feels to me that there are similarities between hiring and dating: both parties show up with goals and expectations in their head, which are often not communicated. Sometimes there’s a match; sometimes there is not. This phase of a relationship is often about “selling:” as a candidate on the job market, it’s about selling yourself, and convincing the company that you would be a fit for what they are looking for.

Doing your research on the company is underrated, and not all that many candidates do so, in my observation. When I was a hiring manager at Uber, roughly half of the people who got on the call with me did not do any research about the company, and perhaps 1 out of 10 candidates did any research on the team they interviewed for – when we had public blog posts about our work, on the company blog! So those showing up prepared helped them stand out in the “motivation” dimension, from the get go.

It all starts with being able to pass the “technical” interviews – but it’s a mistake to sleep on the “behavioural parts.” To state the obvious: candidates who do not do well on the technical interview rounds will not get offers. But I’ve personally had to say to several candidates who did great on the technical side of things, but turned out to be misaligned with what we were looking for, as confirmed on the behavioral rounds.

And I do believe you can uplevel in doing better on these behavioral rounds: starting with researching what the company’s culture is like, practicing how to present yourself better, and putting yourself in the shoes of the interviewers, understanding what they are looking for.

I know plenty of software engineers who refuse to do any preparation for interviews, staying “if the company doesn’t want me as I am, they don’t deserve me anyway.” This is a valid strategy, and can work for highly in-demand professionals, the same way as showing up to a first date in sweatpants and slippers can still work out for highly attractive and desirable people. For the rest of us not as incredibly in-demand for a position we’re applying for: it’s probably worth putting in additional effort, in hopes for better outcomes during interviews.

For today’s issue, we’ve dug into your 900+ responses to look for trends in AI tool usage among software engineers and engineering leaders. This article surfaces insights that are less about specific tools, and more about the effect these tools have on tech professionals. We cover:

1. Costs. Unsurprisingly, companies pay for most tool usage, and those responsible for budgets are increasingly nervous that AI-related costs are headed only one way: up.

2. Usage limits. Around 30% of respondents say they have hit limits. Switching tools, upgrading plans, or moving over to API pricing are common responses.

3. Impact on “Builders.” Folks who make larger code changes and do “quality-of-life” work are builders, and they’re also dealing with more AI slop. Some also grapple with a loss of professional identity.

4. AI tools speed up “Shippers.” Engineers who focus more on getting things done are the most positive about AI tools. But they also add tech debt faster and might build the wrong things.

5. “Coasters:” learning faster while generating AI slop. Less adept engineers can uplevel faster with AI, but they generate a lot of “AI slop” while doing so, which frustrates builders.

6. Changing software engineer & engineering manager (EM) roles. Engineers have to orchestrate and context switch more often, while engineering managers can be more hands on. It’s interesting to see the engineer and manager roles becoming more similar.

7. Other impacts on the craft. We’re going from “how” to build to “what” to build, solo devs are seeing improved results, workloads are increasing with AI tools, and more.

See also two other articles in this mini-series:

• AI tooling for software engineers in 2026: a detailed summary of survey responses, covering the most-used AI tools, trends, AI agent usage, company size and usage, and tools engineers love.

• The impact of AI on software engineers in 2026: key trends. Part 2. Tradeoffs of AI tooling, why adopting AI at company-level is hard, what’s changed in two years, and more.

Full subscribers also have access to a more detailed, 65-page report on our findings.

1. Costs

Concern about the cost of AI tools is a trend throughout the survey, with around 15% of respondents mentioning it in some way.

Tech companies foot the bill for the majority of spending on AI tools. More respondents say their employers pay for AI coding tools than those who say they pay themselves, and predictably, employers fund more expensive packages than what individuals buy personally.

Companies commonly pay for “max” plans with the likes of Claude Code, Cursor, and Codex (around $100-200/month per engineer), although some companies’ budgets only stretch to $20/month per engineer – around the price point of GitHub Copilot, and the cheapest Claude or ChatGPT subscriptions.

The most-mentioned AI tool spending patterns:

• When companies pay: ~$200/month plans. Many have enterprise subscriptions, sometimes with subsidies and vendor lock-in. Some companies allow usage-based coverage on top of monthly plans.

• When personally paying for tools: ~$20/month or free tiers. This can stack up across different tools. Around 5% of respondents have separate work and personal subscriptions, and free tier usage is widespread for personal use.

For now, companies seem to be in the experimentation phase with AI tools, and several respondents say that they believe their companies have unsustainable AI-tooling budgets. This is likely because businesses are figuring out the best way of leveraging the tools, and the message to engineers at such places is to not worry about price and usage while that unfolds. A CTO at a small, US-based company shares:

“Right now, we’re not sweating the costs because we’re trying to evolve best practices for the tools, but that has resulted in some devs really blowing through budget, so we may start instituting caps on spending.”

Breaking the budget

At small and mid-sized companies, leadership teams seem more comfortable about going over budget, than engineers running out of budget. There are more accounts from C-level folks and founders about racking up large bills than there are from engineers. A CPTO (Chief Product and Technology Officer) at a mid-sized company:

“I ran up several monthly bills of $600 with Cursor. We have the dev team subscribed to ~$100/month plans. We’re now in the process of moving the rest of the team to Claude Code, as we can get more resources for around $100/month in cost.”

Top spenders can be allocated higher budgets. A number of tech businesses have separate, larger budgets for their heaviest AI users. A senior C++ engineer working in the video game industry says:

“I’ve become my team’s AI champion. In theory, my limits are higher than normal, but I keep myself limited to what others can use, so I can show them useful things they can do.”

UK and EU companies worry more about budgets than US-based ones. Most responses which mention finance teams pushing back against spending even $30-50/month per engineer on AI tools are based in the UK and EU. One amusing example is a 10-person, seed-stage startup where the CEO questioned why they were paying as much as £25/month per engineer for one of the cheapest AI tools around.

In general, it feels like European companies want to see clear value-add in order to justify an increase in tooling spend, whereas US companies are more comfortable with investing first and measuring impact later. At present, the impact of these tools is hard to quantify.

A niche approach is that of AI teams educating devs to use cheaper models. Some European companies go as far as offering education to new joiners on using cheaper models. From an AI Enablement Lead at a 1,000+ person, digital transformation company:

“Within our organisation, we’ve had incidents where our Claude users have overshot their limits. We’re now attempting to educate devs in knowing the difference between different models (knowing when to use Claude Sonnet versus Claude Opus).”

Cost trajectory worries

The cost trajectory of AI tools is generally considered unsustainable in our survey. Devs using the tools heavily tend to hit usage limits, and their employers then have to pay more. At places with API-based pricing, usage is increasing. Those in leadership positions who are responsible for budgets are generally concerned about the direction of costs.

Subsidies are keeping things at bay – for now. A common enough pattern in our survey is of heavily-subsidized enterprise plans that come with vendor lock-in. Several responses raise concerns about what will happen when the subsidies run dry. Experienced engineering leaders recall that cloud providers also played the same game of subsidizing for a few years, then raising prices when a customer was fully “locked in.”

The AI hype cycle is dampening awkward conversations about budgets at some places. A principal engineer at a Fintech tells us:

“The AI hype has created a special, generous budget for AI tools, and there’s no effective budget – yet!”

But some finance teams are getting grumpy. A CTO at a sports-tech company says:

“It’s hard to keep our CFO supportive about investing in these tools because the productivity benefits have proven difficult to conclusively prove. The point that resonated the most was the loss of value when people hit daily limits: having to stop work immediately! Surprisingly, our CFO is still pushing back, despite having experience of getting a lot of value through their own AI usage with their spreadsheets.”

Most survey respondents think the price of AI tools will have to rise. If that happens, it would cause problems at several companies – particularly those in Europe:

“I cannot see how the spend on AI tools is fiscally sustainable in its current form; Max 100 with Claude Code is $100 a month. A single small task powered by Kimi K2.5 using OpenCode is $5, mostly in input cost. If we assume that the third party inference providers are doing so at a sustainable price, the much more expensive Opus model cannot be sustainable, never mind profitable at these plan costs.” — Founder at a seed-stage company, Europe.

“From the economic perspective, at some point, these companies will need more funding or profit, I’m curious how much it costs them to have a proper agent, and still become profitable. It feels slow when you run out of credits when working on repetitive tasks.” — Principal Software Engineer at a seed-stage company, Europe

2. Usage limits

Another major trend in our survey results is the topic of usage limits:

• Hitting limits: ~30% of respondents. Running out of tokens or hitting reset limits is frustrating and disruptive, especially when you’re working on a task or are in a flow state. The majority of respondents who complain about hitting limits are on cheaper plans (typically $20/month.) But this issue is also mentioned about higher subscription levels.

• Under the limit: ~20% of respondents. Avoiding usage caps generally correlates with being on more expensive plans with higher limits, or in roles with enough non-coding work for it not to matter, or when devs do enough work “manually” for AI usage to not be an issue.

Why users of AI tools hit limits

Common reasons cited in the survey:

Being a new AI user or a power user. These are two distinct groups, but an engineering manager at a mid-sized company in Canada says that each one similarly blows through token limits for different reasons:

“We’re mindful of trying to manage costs by setting AI spend limits across the org. We have two subsets of users at odds with each other:

1. Individuals who are still learning and blow through their credits at an inordinate rate, forcing us to keep limits low.

2. Power users who hit the limit through regular use and apply pressure to raise the limit.

It’s a tough balance.”

Using Opus for all work. A few engineers mention being careful about how they use Opus because it previously ate up their token budgets. Here’s a software engineer at a mid-sized company in Europe:

“I made the mistake of using Opus in the past and burning through budgets quickly. Now, my routine is to start in ‘plan’ mode with Opus. I then paste the acceptance criteria and description of the issue and let the plan mode figure it out. I then switch to Composer or Sonnet and have the agent take over from there.”

Mistakes that eat up tokens are easy to make. These include starting on work or a problem from the wrong end, using AI directly for a task rather than opting for a simple script, trying some new tool or technique that ends up consuming tokens (OpenClaw and Ralph Loops are cited), and others.

What happens when the limit is hit?

Hitting the limit with an AI tool is inconvenient and happens to many developers, who take a variety of next steps:

Switch the model or tool. Around a quarter of respondents who hit limits mentioned switching. From a software engineer at working at Atlassian:

“In my company, for Cursor and Windsurf we have monthly limits. Our internal coding tool (called codelassian) also has daily prompt and hourly token limits. When I hit a limit in one tool, I switch to the other.”

Upgrade to a pricier plan. When it’s an option, this is a no-brainer at most places, especially as the alternative would be devs twiddling their thumbs waiting for the limit to be reset. A senior engineering manager at a mid-sized company says:

“In my team, we are regularly hitting session limits with Claude. We upgraded some teammates to the Max 20x plan – and on this plan we have not been hitting limits, so far.”

Adopt API-based pricing. This is the easiest way to keep working without abandoning a task you’re knees-deep in. A senior engineer at a large company says:

“The company provides both the Claude and Copilot corporate offerings. When the limits are reached, I tend to use API keys that my teammates give me.”

3. Impact on “builders”

We identified three different types of professional in the survey:

• Builders: those who care about quality, good architecture, following good coding practices, and who talk about the craft of software engineering, etc.

• Shippers: those who primarily focus on outcomes for a product, features, testing, and experimenting with users. A fair number of leaders, managers, and engineers who were more hands-off with coding before AI tools are in this category, as are product engineers.

• Coasters: engineers who are not considered particularly good or great engineers, but they get the work done. They often do this without much taste or concern for quality, and seem to be mostly coasting along and doing what they’re told.

The overall consensus in our survey results is that AI will amplify and multiply tendencies and patterns that existed before, and the impact of the tools varies accordingly among users. Let’s start with the impact we’ve observed upon builders in the responses:

Builders say they get value from AI tools in the following areas:

Larger code changes. Builders generally find AI helpful for work like:

• Refactoring

• Migrations

• Improving test coverage

• Carrying out large codebase changes

All these are changes that are laborious, but not very challenging technically. They also require experience in knowing what you want to do and how to do it.

Accomplishing “quality of life” tasks. Builders mention that with AI tools, they get to fix and improve things like nagging bugs that otherwise wouldn’t be “worth it” in time invested, but the barrier to entry is lower with AI.

A good example of this is in last week’s podcast with David Heinemeier Hansson (DHH), the creator of Ruby on Rails, in which he revealed how one of their engineers optimized P1 – the fastest 1% of web requests:

“One of our most agent accelerated people asked: “What about P1? What about the floor? Can we fix the floor?” He found that the floor [of request speed] was 4 milliseconds.

Well, 4 milliseconds can add up if you have a bunch of fast requests. So, he just said: “We’re going to optimize P1. The fastest 1% of requests, we’re going to make them even faster.” He took it from 4 milliseconds to less than half a millisecond. He did this P1 project over a couple of days, like a side project.

He had an intuition that there was something here. He let agents run with it. The work ended up being 12 pull requests, and about 2,500 lines of code changed.

This is exactly why the explosion of the pie suddenly lets us look at problems we would never have contemplated looking at before.”

Typing is no longer a bottleneck. Some builders report falling even more in love with coding with the help of AI and agents, since physically typing out code is no longer a bottleneck for them. They enjoy being able to prompt. From one “builder”:

“For someone who loves to build – but also values code quality, performance, reliability, and security – I ship a lot more quality code faster, if for no other reason than because the AI can read and write 100x faster than me. I get to stay at the conceptual level of shipping a product, and I can dive into debugging with the agent as needed. But if the agent has a good handle on the situation I can give it as much of the tedious parts as I wish.” – Staff Engineer, at a large tech company, US

The negative sides of AI tools, as experienced by builders:

• More AI slop. Builders seem to be the most overwhelmed and derailed by reviewing a lot more AI-generated code. They can get frustrated with low-quality code shipped by colleagues which could be categorized as “AI slop.”

• More debugging. AI-generated code introduces bugs and issues, and builders tend to spend the most time debugging and fixing those issues.

• Identity loss. Some builder-types report a sense of identity loss and even some grief. Much of this relates to no longer doing hands-on coding because they cannot justify it, since AI agents generate pretty decent code faster than someone can type it.

4. AI tools speed up “Shippers”

The “shipper” archetype thrives when they get things to production quickly. This group is by far the most enthusiastic about AI tools in survey responses. They are also the ones who praise – or hype up – the tools because of their personal experiences of shipping much faster with them.

The biggest upsides mentioned by shippers:

Read more

Martin and Kent go back decades, and Martin jokes that his career is “mostly about writing down Kent Beck’s ideas.” They first collaborated in the 1990s, and each has published influential books – ‘Extreme Programming Explained’ and ‘Test-Driven Development’ by Kent, and ‘Refactoring’ and ‘Patterns of Enterprise Application Architecture’ by Martin.

At the Pragmatic Summit, they each shared a wealth of hard-earned learnings and decades-worth of perspective, along with a healthy dose of skepticism. Needless to say, the conversation did not disappoint, and this article summarizes what we discussed in their own words. You can also check out the full recording.

We cover:

1. Technology shifts similar to AI. The arrival of the microprocessor, introduction of object-oriented languages, the Internet, and agile software development principles were all major changes – but one big difference was that it took time for these technologies to be adopted. Not so with AI.

2. Agile and AI similarities. With Agile, company incentives were often misaligned, “snake oil” vendors were everywhere, and a “mid pack” of developers who resisted the change saw their career prospects hit. These trends look likely to repeat with AI.

3. What’s happening inside companies. There’s some confusion – and even panic – at large companies, while AI tools don’t work nearly as well on large and complex codebases as on greenfield projects. Also, a “re-soloing” of software development is inbound.

4. Avoiding burnout with AI agents. Set and maintain boundaries, and pay attention. Martin suggests to catch when you start producing “negative value”: that’s when to take a break.

5. Unhealthy performance metrics. Companies are starting to measure things like frequency of pull requests – when they should be looking to quantify outcomes and results.

6. Lower quality on purpose? It seems every business is optimizing for speed with AI, but quality can get dropped. Also: building features is more obvious with AI, than investing in “futures.”

7. Test-Driven Development (TDD): tests no longer optional? Kent pioneered TDD, and today it’s more relevant than ever for working with AI.

8. Thriving in an AI-native industry. Focus on working with agents to express your craft, try to get more enjoyment in understanding your domain, and take on more ambitious work.

Before we start, a programming note: this week, there will be no The Pulse on Thursday — I’ll be attending AI Engineer Europe in London on Thursday and Friday, including doing a fireside chat, and hosting one with Linear CTO Tuomas Artman.

1. Technology shifts similar to AI

Do you recall a tech change as similarly promising and unpredictable as AI?

Martin: “Nothing has hit with the magnitude of AI. This is a whole size different from anything we’ve faced before. On a smaller scale, we were very much involved in the growth of object oriented languages, which scared a lot of people. It didn’t scare us so much because we were part of it.

Looking back, the internet had a huge impact on us all, and of course, Agile software development, too. Agile had a very big impact on a lot of organizations: you could tell by how hard they resisted it. We had to persuade people of the importance of these technological changes; yes, even the internet! It may sound surprising but there were people who didn’t think it was important.

The thing about AI is that today there is no argument about how important it is.”

Kent: “The other analogy I have is the introduction of the microprocessor. Before that, computers were big boxes; you couldn’t move them around. If you wanted another computer, you’d mortgage your house for it. Having a computer was a big deal.

I was a kid in Silicon Valley with my dad as a programmer when the Intel 4004 hit the market [in 1971]. We went: “Wait a minute, that chip is a computer? Oh my goodness!” The possibilities of computing suddenly expanded thanks to it. If you could figure out how to write software on this chip and figure out how to design hardware around this thing, you could suddenly do things you hadn’t even imagined.

And so I think part of AI is this expansion of imagination. I’m writing projects that are ridiculously ambitious: I’m working on a persistent Smalltalk. I’m writing library-quality code for Rust.”

Balancing skepticism and curiosity

What was the feeling in the industry during those revolutions, and the differences between professionals who thrived back then and those who didn’t?

Martin: “There was a mix of people chasing the hype and those saying, “this new thing is nothing special.” I think you’ve always got to have that balance of skepticism and curiosity, and to be selective about it. I mean, I have been completely skeptical about some big changes: Blockchain was one I was extremely skeptical about.

My skepticism is well-rooted because I’ve seen so much “snake oil” over the years. In fact, my skepticism has to be absolute and total, which means I have to be skeptical about my skepticism! To be that skeptical also requires curiosity: you’ve got to be curious enough to say “how do I probe in order to detect signs of something useful?”

You also need to be aware that your early interactions may not actually be a true signal. When I started playing around with AI, it was with GitHub Copilot a year and a half ago. I was pretty unimpressed; it would give you something wonderful, but most of the time it gave you such garbage that you would just delete it right away. If that had been my only impression of AI, I would’ve immediately flipped the “bozo bit” on it, like I did with blockchain.”

Kent: “Here’s the thing, the capabilities of AI can change week to week. I’ll try something with Gemini one week and it fails miserably. Then Claude Code works pretty well, and then it doesn’t. And then I try Gemini for the same thing and it works, when it hadn’t worked last week!

People want an answer, but the answer’s always changing. In this environment, you can’t possibly have the answer. That’s the bad news, but the good news is that nobody else has the answer either. So, you’re just as smart as everybody else because we’re all equally ignorant.”

2. Agile and AI similarities

In 2001, the ‘Agile Manifesto’ came out, of which you were both co-authors. I think many companies are expecting the same thing with AI as Agile promised: better, faster, cheaper software. But how did Agile adoption really play out?

Kent: “It turns out people don’t want faster, cheaper, better! Inside some companies, the incentives are misaligned with actually achieving that. And so as geeks trying to achieve these improvements and saying: “it’s 40% better, 12% cheaper and less fattening,” people will punish you if that doesn’t align with their incentives inside organizations.

In the ideal organization, everybody would care about the same things, but that’s just not the way it works! So, if AI is coming along to promise the same things, we’re going to see the same reaction as before.”

Martin: “An obvious difference is the sheer magnitude and speed there is with AI. Also, I think there will be a big difference between people who use it well and people who use it badly. The trick is figuring out how to use it well and putting the effort in to learn. There will be a big distinction between those two groups.

But I suspect there will still be some similarities with Agile. The core notions behind Agile and extreme programming are solid and good, but a huge snake-oil industry appeared around it – the “Agile industrial complex”, as I refer to it. This is also happening with AI right now, and it’s often hard to see the difference between snake oil and the real stuff.”

AI as an amplifier

Kent: “AI is an amplifier. If you’re young and learning quickly, AI can amplify your learning. I personally think this is the golden age of the junior programmer. I get people coming to me all the time saying things like “my son started his second year in CS and wants to go into something more commercial like art history.” And I’d say, “this is like if you’re a carpenter and they just introduced the circular saw and you think, ‘oh, well, carpentry is over. Anybody can build a house now.’ Well, no! Now, you have more powerful tools. You have less of the crummy work to do.

I think that young people are going to learn faster, and experienced folks who are working effectively are going to work quicker and more effectively.”

Developers stuck in the middle

Kent: My concern is that there’s a “middle” of people who got into programming as a way to make money. If we look back at the Dotcom crash, there was a “mid pack” of such people who ended up going into real estate, more or less. But today, I don’t know where that “middle” will go, and it’s also much bigger now than 25 years ago.”

Martin: “But that middle has also been “flushed out” to some degree by retrenchment in the software industry at the end of the zero interest rate period. So, that’s an interesting difference because we’ve had these things occurring at once: the AI boom, and the economic headwinds of the past 2-3 years.

This is an interesting mix that wasn’t present in the ‘90s with the Dotcom Boom. Back then, it was pretty much all a solid boom.”

Return of “let’s get rid of programmers!”

Kent: “Another interesting confluence of factors is the periodic, “we can get rid of all the programmers, woo-hoo” trend, which started with Cobol in the 1970s. With Cobol, business analysts were supposedly going to be able to write the programs, and the logic was that we wouldn’t need programmers anymore. That comes back repeatedly.

Agile, however, was definitely not a “let’s get rid of programmers” trend. With Agile, we wanted programmers to be more effective in their jobs. And since we started it, and were programmers, we were able to push that agenda pretty effectively.

However, today the “get rid of programmers” trend is repeating. As programmers, it behooves us to think about why they keep wanting to get rid of us. Some of that’s about us as programmers, and some of it not. Still, we should think about why people periodically want to axe us. In the end, this trend amps up the fear factor that everybody’s experiencing.”

“Re-soloing” of programming

Kent: “A big trend is the “re-soloing” [reduced in-person collaboration] of programming.

A big part of extreme programming (XP) was creating a safe social environment for basically antisocial people. On an XP team, people are talking to each other for hours a day, and are happy to do so because it’s set up to be a positive experience.

Now, I see programmers saying, “I’ve got six agents, so really I’m managing a team.” No, you’re not: you’re using six tools at once, which is fine, but it’s very different from having a conversation with somebody who sees things slightly differently, or has a different energy level from you on the day.

We used to have programmers in individual offices with doors, and you’d shut the door and slide the pizza underneath. That was easy to manage, but then along came this messy, social, complicated, chaotic process of software development, which just happened to produce really good results.

But now, instead of 50 people on my team, I can have five and they don’t have to talk to each other, and each can have 10 agents. Is that the same? No, it’s not.”

More effective two-pizza teams & the future of pairing

Martin: “Are we seeing two-pizza teams [of 5-10 people] becoming one-pizza teams because agents don’t eat pizza, or do we see two-pizza teams staying and becoming much more effective and capable? My bet is on more effective two-pizza teams.

We’re beginning to see some interesting feedback in terms of pair programming. With pair programming, is it one human and the genie (AI) programming, or is it two humans and one genie? If it’s two of us, perhaps we can control the genie a bit better, and we also have interaction.

I’ll be very interested in reports of people trying to control genies in pairs, possibly even beyond pairs. There’s also the whole ‘mob programming’ thing, and how that will go with genies. I don’t necessarily think that one person and many genies is the right answer.”

Kent: “My experience of pairing with two humans, plus one or more genies, has been very positive. And the fact the AI is slow is really nice. Every time models come out and are faster, I’m like, “Oh, there’s less time to talk.” When the AI goes away for three minutes, we can talk about our philosophy of naming, or how we express conditionals, or about what we should be doing next. But if it pops back in 15 seconds, you don’t have time for that conversation.”

4. Avoiding burnout with AI agents

Do you find yourself getting close to burnout, especially when spinning up multiple threads? Do you have strategies for managing the mental impact?

Read more

And now, inference engineering is becoming more widespread, too, as open LLM models grow more capable. This is because with closed models, inference engineering is done only by the AI engineers who build the model, whose number might add up to a few thousand globally. In contrast, with the open models which tech companies are adopting, it’s possible to tweak them to perform better at inference. For example, Cursor built its new Composer 2.0 model on top of the open Kimi 2.5 model, and successfully used plenty of inference engineering approaches to make it even faster.

So, based on this industry-wide prevalence and the related need for superior technical performance, it’s worth understanding a bit about what inference engineering actually is, and some interesting approaches worth knowing about, as a software engineer.

For some answers, I turned to Philip Kiely, a software engineer who has been working for four years at the inference startup, Baseten. With his hard-earned experience, Philip has written an excellent, in-depth book about precisely this topic, “Inference Engineering.”

In today’s issue, we cover:

1. Setting the stage: why is inference so important? More capable, widespread, open models are driving demand for inference engineering.

2. What is inference? As the phase that comes after training a model, the inference layer introduces new engineering challenges like batching, caching, and quantization.

3. When is inference engineering needed? Investing in this area is typically worth it when your product and usage scales up, and there are product requirements which a current, off-the-shelf solution lacks.

4. What hardware does inference use? Datacenter GPUs are the most common, while on-premises, air-gapped GPUs are also employed.

5. What software does inference use? Commonly-used software includes NVIDIA’s CUDA and Dynamo, as well as hardware-agnostic projects like PyTorch, vLLM, and others, which are growing in popularity.

6. What infrastructure does inference need? Autoscaling is a baseline requirement. Kubernetes is a popular choice for autoscaling inside a cluster, while multi-cloud inference might be necessary for high-scale use cases.

7. Five approaches to make inference faster. Quantization (reducing the numerical precision of a model’s weights), speculative decoding (taking advantage of spare compute to generate “draft tokens”), caching, parallelism (tensor parallelism and expert parallelism) and disaggregation (separating the prefill and decode phases to run on separate workers, not the same GPU).

This deepdive uses a few abbreviations and concepts that are everyday lingo for inference engineers, but maybe are not for those less versed in the domain:

• CUDA: Compute Unified Device Architecture. NVIDIA’s proprietary API to program NVIDIA GPUs for high-performance computing, including LLM-related use cases.

• TTFT: time to first token. Think of this as the “time to process the prompt.” This metric determines the perceived responsiveness of models and GenAI systems.

• TPS: tokens per second. Akin to a model’s “typing speed.”

• ITL: intertoken latency. The time between generating one token and the next.

• KV cache: key-value cache. The cached results of the attention algorithm, reused between requests to speed up inference. We cover more on KV cache in the Scaling ChatGPT deepdive.

• Prefill / decode: the two phases of inference. Prefill is when the model takes the full input and processes tokens, outputting the KV cache. Decode is the phase in which the model generates one token at a time.

• MoE: Mixture of Experts. An architecture that enables models to be pretrained with far less compute. More details on this approach.

Below is an introduction to inference adapted from Philip’s book, “Inference Engineering,” which is free to download as an e-book. Physical copies are currently sold out, but Philip is printing more as fast as possible.

My usual disclaimer: as with all my recommendations, I was not paid to mention this book, and no links in this article are affiliates. See my ethics statement for more.

With that, it’s over to Philip:

1. Setting the stage: why is inference so important?

Inference is the most valuable category in the AI industry, but inference engineering, on the other hand, is still in its infancy. In their work, inference engineers work across the stack from CUDA to Kubernetes in pursuit of faster, less expensive, and more reliable serving of generative AI models in production.

When ChatGPT launched in late 2022, there were perhaps a few hundred inference engineers in the world, and they didn’t call themselves that. These specialists mostly worked at frontier labs like OpenAI, Midjourney, and Anthropic, or at big tech companies like Google and NVIDIA.

Back then, it looked like this might be the way of the AI industry: that training generative AI models would be so hard and expensive that only a handful of companies would develop closed models and thereby require inference engineering for production serving. In that alternate future, the rest of the world would be mere consumers of AI via APIs, renting intelligence a token at a time.

Three years later, it turns out that training generative AI models is indeed both hard and expensive – but it’s not so hard and expensive to be limited to a handful of players. Instead, a proliferation of open models – more than two million and counting on Hugging Face (the “GitHub for AI”) – means that today every engineer can now deploy their own intelligence to power AI products.

Research labs around the world, from OpenAI and NVIDIA Nemotron in America, to Mistral AI and Black Forest Labs in Europe, to Alibaba Qwen, DeepSeek AI, Z AI, and Moonshot AI in China, regularly release open models of all modalities.

Despite closed models getting smarter and cheaper, the movement into open models is accelerating, which differ by the availability of their weights:

• Closed model: A proprietary model whose weights are unavailable to the public, like GPT-5 and Claude Sonnet.

• Open model: A model whose weights are publicly available, like Llama or DeepSeek, and which is usually released under the MIT license, or a similar permissive license (some models restrict commercial use, so always double-check license terms).

Before December 2024, there was a meaningful gap in intelligence between closed and open models, but when DeepSeek V3 and R1 were released, that gap disappeared. Note from Gergely: we previously covered how DeepSeek’s release rocked the AI industry.

Today, new closed models are matched by open models within months if not weeks, and occasionally, open models like Kimi K2 Thinking even exceed closed models’ capabilities for brief periods.

Despite the fact that open models are constantly chasing closed models on benchmarks, they nonetheless change the equation for AI product builders. And as both types improve, closed and open models cross capability thresholds.

In 2022, it was impossible to build the kinds of AI-native products that define the industry today. But over time, closed models got smarter and new categories like customer service voice agents and AI-powered IDEs became possible. The early models were slow, expensive, and unreliable, but the capabilities were there and AI engineers began building companies around them.

As open models crossed the same capability thresholds, these folks began using them to replace closed models. Many also began fine-tuning open models to cross capability thresholds faster, and even exceed closed model quality in their specific product and domain.

Switching to open models means the opportunity to use inference engineering to make the models powering AI products better in new ways:

• Latency: Closed model APIs are built for throughput, but open models can be optimized for real-time applications.

• Availability: While APIs for GPT and Claude are stuck at two nines of uptime, it’s possible to achieve four nines or better with dedicated deployments of open models.

• Cost: Open models are often at least 80 percent less expensive at scale.

So, whereas three years ago it looked like inference engineering was a niche field, the fact is that today, every company aiming to build truly differentiated and competitive AI products needs an inference strategy.

AI-native startups like Cursor, Clay, Gamma, and Mercor are redefining hypergrowth by building products that rely on open and in-house models. Leading digital native companies like Notion and Superhuman succeed by deeply integrating AI capabilities into their category-defining products.

Elsewhere, a new generation of blended research and engineering teams – World Labs, Writer, Mirage, and dozens more – are building businesses by training and productizing their own foundation models.

Adoption is even strong in enterprise and regulated industries, which historically were slow to adopt new technologies. Companies like OpenEvidence, Abridge, and Ambience are making generative AI ubiquitous in healthcare, while at the world’s largest companies, AI initiatives are moving past the pilot stage into massive user adoption. Market-wide demand for inference means that everyone from developers to executives has the opportunity to learn inference engineering and use it to advance their career and business.

I’ve been incredibly fortunate to have a front-row seat in the fastest-moving market in history over the last four years at Baseten, where we power mission-critical inference for the best AI products, including every company listed in the previous paragraphs.

The good news is that you are early. There are still relatively few professionals working on inference, and newcomers can become experts quickly. Also, the potential and impact of inference is becoming ever clearer, but the domain is still in its infancy. That means there are enormous opportunities to solve novel, interesting, and deeply technical problems at all levels of the stack.

2. What is inference?

Inference is the second phase of a generative AI model’s lifecycle:

• Training: The process of learning model weights from data.

• Inference: Serving generative AI models in production.

During the past decade’s machine learning (ML) boom, hundreds of thousands of data scientists and ML engineers became familiar with the full lifecycle of training and inference for ML models.

Inference for classic ML models is relatively straightforward. In the early days of Baseten, we ran inference for models built with tools like XGBoost on lightweight CPUs with a simple software stack.

In contrast, inference for generative AI models is complex. You can’t simply take model weights, get some GPUs, and expect inference to be fast and reliable enough for large-scale production use. Doing inference well requires three layers:

• Runtime: Optimizing the performance of a single model on a single GPU-backed instance.

• Infrastructure: Scaling across clusters, regions, and clouds without creating silos, while maintaining excellent uptime.

• Tooling: Providing engineers working on inference with the right level of abstraction to balance control with productivity.

These three layers must work together to create a system that can handle mission-critical inference at scale.

The runtime layer is responsible for ensuring an individual model running on a GPU (or across several GPUs in a single instance) runs as performantly and efficiently as possible. This layer depends on a sophisticated software stack, from CUDA, to PyTorch, to inference engines like vLLM, SGLang, and TensorRT-LLM. Low-level optimization is important, with kernels like FlashAttention delivering significant performance gains.

The runtime layer relies on a number of model performance techniques that apply new research to the challenges of inference on generative AI models:

• Batching: Run incoming requests in parallel, weaving them together on a token-by-token basis to increase throughput.

• Caching: Reuse the KV cache – the cached results of the attention algorithm – between requests that share prefixes.

• Quantization: Lower the precision of select pieces of the model to access more compute and reduce memory burden.

• Speculation: Generate and validate draft tokens to produce more than one token per forward pass during decode.

• Parallelism: Efficiently leverage more than one GPU to accelerate large models without introducing new bottlenecks.

• Disaggregation: Separate the two phases of LLM inference, prefill and decode, onto independently scaling workers.

These model performance techniques are used for all modalities and not just LLMs, such as vision language models, embedding models, automatic speech recognition, speech synthesis, image generation, and video generation, which extend the capabilities of AI systems and require their own inference optimizations. But these runtime optimizations are not enough: no matter how performant a single instance of a model server is, it will eventually receive more traffic than it can handle. This is not a CUDA problem or a PyTorch problem, it’s a systems problem that needs to be solved at the infrastructure layer.

The nature of infrastructure problems changes at each level of scale. At first, the problems are around autoscaling: knowing when to add and remove replicas, and figuring out how to do so quickly.

Past a certain scale – generally a few hundred GPUs – infrastructure problems are defined by capacity. To get access to enough GPUs, inference engineers begin spreading workloads across multiple regions and cloud providers. This quickly leads to silos, where models in one cluster may be starved for resources while other clusters have unused capacity. The final level of scale in infrastructure is a global system that treats all available resources as a single unified pool of compute.

Thoughtful multi-cloud infrastructure also improves reliability, protecting against downtime in any individual region or cloud provider. And for global applications, running inference near to end users improves end-to-end latency.

Once these runtime and infrastructure capabilities are built, they need to be presented at the appropriate level of abstraction. Inference providers like Baseten and internal teams building inference need to consider what tooling and developer experience to provide as the critical third layer in a complete inference platform.

Of course, developer experience is subjective. For inference, one extreme is the black box: give a platform model weights, and get back an API. At the other extreme is providing only basic constructs for compute, network, disk, and so forth.

The right developer experience is somewhere in the middle, where inference engineers have enough control to run mission-critical inference confidently, and enough abstraction to work productively.

This article – which is an excerpt of Inference Engineering – presents an overview of the technologies and techniques that power inference across all three layers of runtime, infrastructure, and tooling.

3. When is inference engineering needed?

Inference engineering adds speed and scale to AI products by optimizing production serving of generative models. Optimization means identifying the best solution from a range of options.

Before optimizing model performance and building robust infrastructure, you need to know what “best” means for your product; many performance improvements come from making tradeoffs in latency, throughput, and quality. In practice, optimization is often about finding the right balance, rather than maximizing a single factor.

For example, NFL players are big, fast, and strong. But they’re not as big as sumo wrestlers, fast as Olympic sprinters, or strong as champion powerlifters. Their bodies and skills are optimized to fulfill the specific demands of their position over the course of a full season.

Similarly, your inference system must be optimized to fulfill the specific demands of your model, product, and traffic. The more constraints you can introduce, the better the outcomes that can be achieved.

You should know:

• Model requirements: Which model(s) do you need to run inference on?

• Application interface: How will inputs be delivered to the model, and how is the output expected to be formatted?

• Latency budget: How fast does your product need to respond to a user action, end-to-end?

• Unit economics: How much sense does it make to spend on a per-request, per-user, or per-month basis?

• Usage patterns: How many concurrent users are you serving, and is there any pattern to their usage (e.g., more activity during business hours)?

Early in building an AI product, the answers to these questions may not be clear. At this point, it’s often better to use off-the-shelf APIs whenever possible, rather than investing in dedicated inference. But as a product scales, the requirements become clear and inference engineering becomes a worthwhile pursuit.

4. What hardware does inference use?

Inference engineering relies on accelerators: powerful hardware designed to load terabytes of data and perform trillions of operations per second.

The most common type of accelerator for inference is the GPU, and the market leader in GPUs for inference is NVIDIA. My book focuses on inference engineering for NVIDIA GPUs in the datacenter, and also covers other vendors of datacenter accelerators and local inference.

Across vendors, there are three types of GPUs on the market:

• Datacenter GPUs: Racked servers with interconnected high-performance GPUs. Example: NVIDIA B200.

• Workstation GPUs: Individual desktop GPUs for professional workflows. Example: NVIDIA RTX Pro 6000.

• Personal computing GPUs: Individual desktop GPUs for everyday use. Example: NVIDIA GeForce RTX 5090.

Inference at scale uses datacenter GPUs mounted on racks: refrigerator-sized chassis with standardized power, networking, and cooling.

Datacenter GPUs like the NVIDIA B200 offer the highest individual performance, and more importantly, include high-bandwidth GPU-to-GPU interconnects, are installed in highly standardized configurations, and are available by the millions in datacenters worldwide.

I doubt there is a B200 GPU running under your desk, but if there is, please send me a picture! Instead, inference on datacenter GPUs runs in one of three modes:

• Cloud: GPUs are rented in someone else’s datacenter, usually hyperscalers like AWS and GCP, or neoclouds like CoreWeave and Nebius.

• On-premises: GPUs are purchased and installed in a datacenter that you control directly.

• Air-gapped: GPUs are installed on-prem and you need to physically access the GPUs to run inference.

Most inference engineers use cloud GPUs. Large enterprises and governments run on-prem and air-gapped deployments, but cloud-based GPUs offer the flexibility and access that fast-growing AI products need to scale.

Even with constraints, navigating the hardware landscape is complex. From variations between cloud providers, to NVIDIA’s personal naming conventions, there are many nuances in selecting the right accelerator.

5. What software does inference use?

NVIDIA’s market dominance in the inference space is in no small part due to the robust, mature software ecosystem around its hardware. Hardware iteration cycles are slow. Best-in-class hardware companies like Apple and NVIDIA release new architectures and generations at most annually, with two-year release cycles being more common. But software iteration is fast. Often, to run a newly released open model on day zero, you need to install a nightly build or other pre-release version of each software dependency just to get support for the new model.

Software’s fast iteration cycle and lower barrier to entry dramatically expands the landscape of inference engineering. There are countless companies building software at various levels of the inference stack, in contrast to hardware, which centers on NVIDIA and a few competitors.

For inference engineers, these are some of the key software players:

• NVIDIA: Invests heavily in its own sometimes-proprietary software ecosystem, from CUDA up to Dynamo.

• Hugging Face: Maintains a model registry for all open models plus transformers (models built on the transformer architecture), and diffusers (models built on the diffusion-based generative ones).

• The Linux Foundation: Maintains hardware-agnostic projects like PyTorch and vLLM.

• LMSYS Org: Develops essential tools for inference and evaluation, most notably SGLang.

There are thousands more companies, universities, and research institutions making essential open-source contributions to inference. Over time, technologies have been built at increasing levels of abstraction:

• CUDA: Direct communication to the GPU for explicit control over computations and memory.

• Deep learning frameworks: Abstractions over CUDA for training, exporting, and running neural networks in Python.

• Inference engines: Highly configurable PyTorch-backed inference for common architectures.

• NVIDIA Dynamo: Sits on top of inference engines to power large-scale deployments.

Most inference engineering today happens at the higher levels of abstraction, configuring and deploying inference engines and orchestrating inference across multiple GPUs. No matter which level of the stack you work at, it’s essential to have a strong mental model for the adjacent levels of abstraction to guide your work.

6. What infrastructure does inference need?

When you scale production traffic, your assumptions are rigorously tested. Everything from input and output sequence lengths, to traffic patterns, to what topic a user decides to chat about; they all impact your observed performance in production. And maintaining secure, robust infrastructure is an entirely different skillset from optimizing model inference on the GPU.

No matter how fast and efficiently a single instance can serve a model, the service will be overwhelmed if traffic gets high enough. It’s an infrastructure problem, not with PyTorch or CUDA, and it requires a different mindset and different technologies.

Scaling in production introduces new complexities about where and how to get GPUs, balance traffic across them, and prevent downtime. The goal of autoscaling is to ensure you always have enough resources to serve all incoming requests, while maintaining latency SLAs and without wasting money on idle GPUs.

Autoscaling systems use Kubernetes, an open-source container orchestration system, along with a cluster-level system for provisioning and deallocating compute. Kubernetes can run one or more replicas of a model container, each on its own instance. An instance includes the GPUs and other hardware resources that the container requires.

Unless your traffic is unusually consistent, there probably isn’t a specific number of replicas that perfectly matches your needs.

Autoscaling is the practice of dynamically adjusting the number of replicas allocated to a given model within a cluster. There are two ways to make autoscaling decisions:

• Utilization: Scale up or down based on GPU utilization signals like memory usage or compute usage.

• Traffic: Scale up and down based on the number of requests being processed in the system.

Utilization and traffic don’t always match. For example, in LLM prefill, a few requests with hundreds of thousands of uncached input tokens could cause much higher utilization than many small requests with high cache hit rates.

Traffic-based scaling decisions can be made proactively, while utilization is a lagging indicator. Use both in combination to keep system resources matched with demand.

When designing a traffic-based autoscaling system, you want to configure five factors:

• Min replicas: What is the minimum number of replicas that stay running, regardless of traffic?

• Max replicas: What is the maximum number of replicas you can allocate when traffic is high?

• Autoscaling window: How long is the sliding timeframe used to measure traffic and make autoscaling decisions?

• Scale down delay: For how long after a scale-down is suggested do you wait, in case there’s another traffic spike?

• Concurrency target: How many requests can each replica handle at once?

The exact configuration determines how well the autoscaling system achieves its goals of maintaining latency SLAs without wasting resources. For example, increasing the scale-down delay prevents premature scaledowns for spikey traffic, but could result in unnecessary spend after traffic has properly cooled down.

Autoscaling within a single cluster works up to a certain point, but high-volume deployments serving a global user base need thousands of GPUs distributed around the world.

It’s straightforward to build multi-cloud inference as a collection of siloed compute across different cloud providers. But in these setups, there’s no way to use inter-cloud compute fluidly, and moving workloads across clouds is a tedious, error-prone process.

True multi-cloud inference requires building a multi-region, multi-provider bin packing tool, which treats distinct pools of compute as fungible with each other. Like Kubernetes within a single cluster, multi-cloud capacity management must take a global view, enabling global scheduling.

Running true multi-cloud inference unlocks:

• Capacity: Pool capacity from multiple providers for greater, more flexible GPU access.

• Redundancy: Split inference across providers for resiliency against outages.

• Latency: Run inference close to end users to reduce network latency overhead.

• Compliance: Run inference in compliance with data sovereignty and other regulatory requirements.

Scaling from one cluster in one cloud to many clusters in many clouds requires a new coordination layer. A multi-cloud architecture contains:

• Control plane: Handles model deployment and global scaling decisions, receives real-time event streams.

• Workload planes: Handles direct inference traffic and in-cluster scaling decisions, reports utilization and demand.

This separation of responsibilities ensures that individual workload planes can serve traffic independently. If something happens to the control plane or any given workload plane, other workloads should be unaffected.

7. Five approaches to make inference faster

One of the coolest things about working in inference engineering is that, unlike many industries where new academic research takes years or decades to be adopted, techniques from new papers are live in production within months or even weeks.

But there is a gap to bridge between research and production, and some of the most visible inference engineering work of all comes from doing so.

Real-world traffic defies constraints. But with volume, you can adapt systems over time to match the changing nature of usage. Tuning the parameters of inference engines, speculation algorithms, and model servers isn’t a one-time task. Instead, either through iterative deployments or dynamic runtime adjustments, you can continuously improve the performance of an inference system.

Finding the right combination of techniques and configurations takes patient experimentation. I remember an internal hackathon during which one of Baseten’s inference engineers worked on an autocomplete model for code, and ended up trying 77 different configurations via a handwritten script before finding a non-obvious solution that doubled TPS (tokens per second) for a customer’s model.

Sometimes, techniques are symbiotic or incompatible, which makes inference optimization even more complex. For example, quantizing the KV cache alleviates a bottleneck in disaggregation, but increasing batch sizing reduces the compute available for speculation. An inference engineer’s challenge is always to create a balanced set of optimizations that delivers more than the sum of its parts.

Let’s look into the key categories of applied research for inference acceleration: quantization, speculation, caching, parallelism, and disaggregation.

Approach #1: Quantization

Quantization means reducing the numerical precision of a model’s weights. It improves latency (both TTFT [time to first token] and TPS, increases system throughput, and opens up headroom for other optimizations like disaggregation, speculation, and prefix caching to be even more effective. But when it goes wrong, quantization can materially reduce a model’s output quality.

Models are trained with weights, activations, and other components represented in a certain native number format. Usually, this is BF16 or FP16, although 8-bit and 4-bit native precisions are becoming more popular for training.

Post-training quantization works by changing those model weights and other values from their native number format, to a lower-precision format. Cutting precision in half improves performance in both phases of inference:

• Prefill: Compute-bound prefill now runs on lower-precision Tensor Cores with twice the FLOPS.

• Decode: Memory-bound decode now loads half as much data per value, effectively doubling memory bandwidth.

Working with quantized data introduces overheads, so it’s not linearly twice as fast to go from 16 to 8 bits. In practice, quantization down a single level of precision generally offers 30%-50% better performance for LLMs. The catch with quantization is that it runs the risk of reducing a model’s output quality, and has the potential to introduce precision errors throughout the calculations that power inference.

Precision errors compound over time. Consider what happens when you square and cube different precisions of Pi:

Most of the work in quantization is in preventing precision errors and minimizing their impact on the final model output.

Sixteen-bit, 8-bit, and 4-bit precisions are the primary formats for inference. Number formats contain:

• Precision: The number of bits used to express a single value in the format. For example, FP16 uses 16 bits.

• Type: Whether these bits are interpreted to represent an integer (non decimal) or a floating-point number (decimal).

• Scale factor: A multiplier used to map values from a low-precision format back to the higher-precision format.

Combined, these attributes determine the two factors behind how well a number format represents the values used in inference:

• Dynamic range: The difference between the lowest and highest value that can be represented in the format.

• Granularity: The number of parameters or other values that are quantized along a single scale factor.

Dynamic range is essential to low-precision inference without quality loss. Sixteen bits can represent 65,536 distinct values, while 8 bits can only represent 256 different values. The dynamic range is the distribution of these values – the difference between the smallest and largest available value.

Dynamic range explains why floating-point formats are better than integer formats for inference. Floating-point formats have three properties:

• Sign: A single bit that represents whether a number is positive or negative.

• Exponent: A set of bits that, taken together, represent an exponent factor.

• Mantissa: A set of bits that together represent the base value multiplied by two to the exponent.

An FP8 number in an E4M3 data format means it has a 4-bit exponent and a 3-bit mantissa, with the remaining bit for the sign. Integer formats only have sign and value bits.

The exponent in floating-point numbers gives it a higher dynamic range, meaning it can better express very large and very small numbers. This is important because outlier values are significant in inference, and floating-point number formats better represent outliers after quantization.

Within floating-point formats, there are multiple options at each precision, like FP4, MXFP4, and NVFP4. These formats differ in granularity, or in the number of values quantized by a single scale factor.

Quantization can be applied at three levels:

• Tensor level: Calculate a single scale factor for the entire QKV tensor.

• Channel level: Calculate a different scale factor for each feature vector within the tensor.

• Block level: Within each feature vector, divide the vector into blocks of N values and calculate a scale factor for each block.

More granular quantization has a lower chance of smoothing over outliers, which preserves quality. However, more granularity also introduces extra overhead for storing and applying scale factors.

The components of a model have varying sensitivities to quantization. Reducing the precision of more sensitive components runs a higher risk of quality degradation. From the least to most sensitive components:

1. Weights: the linear layers are least sensitive to quantization.

2. Activations: The intermediate output of activation functions are only somewhat sensitive to quantization. They are rarely quantized as they are such a tiny fraction of the model’s weights.

3. KV cache: The cached values from the attention calculation are moderately sensitive to quantization.

4. Attention: The attention layers of a model are highly sensitive to quantization, especially equations like softmax.

Within each component, you can get more selective about quantization.

Even in linear layers and activations – generally the least sensitive to quantization due to their size – early and late layers, like the input and output layers of the neural network, may be left in their original precision as these layers are more sensitive.

While quantizing weights and activations helps performance, KV cache quantization gives an additional boost to techniques like prefix caching and disaggregation. The KV cache is a valuable resource and quantizing it allows inference engines to store more of it in memory and read it more quickly.

However, the KV cache for each token is used by each subsequent token. This means precision errors introduced by quantization can compound from token to token. Compounding errors are exactly why attention layers are the riskiest to quantize: not only is attention very sensitive to dynamic range, but each attention calculation relies on the results of each previous attention calculation. Therefore, over a sequence of thousands of tokens, errors accumulate quickly.

All but the most aggressive quantization schemes run functions like softmax in their original precision.

A moderate approach to low-precision inference uses a format like FP8 with high dynamic range – if possible, a microscaling format like MXFP8 – to carefully quantize select linear layers, activations, and often KV cache values. Even with these high dynamic range formats, components of the attention layer are rarely quantized.

Approach #2: Speculative decoding

The decode phase of LLM inference is an autoregressive process in which tokens are generated one at a time. The bottleneck on decode is memory bandwidth, with compute sitting idle at low-to-moderate batch sizes as weights are read from memory.

Speculative decoding takes advantage of that spare compute to try and generate multiple tokens per forward pass through the target model. If an inference engine could generate two, three, or even more tokens for each round-trip of weights through memory, it would generate far more tokens per second. Note, speculative decoding only improves TPS / ITL (inter-token latency), not TTFT (time to first token.)

There are multiple algorithms for speculative decoding and they share a common mechanism:

1. The speculator generates one or more draft tokens.

2. The target model – or the underlying model that you’re trying to accelerate – performs validation on these tokens to check if they match what the model would generate.

3. The target model accepts any valid draft tokens and generates an additional token itself, completing the forward pass.

This generates N+1 tokens per forward pass, or iteration through the decode loop, where N is the number of accepted draft tokens.

Generating draft tokens is not free, it takes both compute and memory. However, it is much faster for a target model to validate a draft token than to generate an original token. If you imagine a sudoku puzzle, solving it is hard, but checking if the solution is correct is very easy. For the target model, generating a token is like solving a sudoku, while validating a draft token is like checking a finished sudoku.

The performance uplift from any speculative decoding strategy depends on three factors:

1. Draft token cost: Time taken to generate a draft token.

2. Draft sequence length: The number of draft tokens generated per forward pass.

3. Token acceptance rate: The percentage of draft tokens accepted by the target model.

Token acceptance rate is high early in the draft sequence, but draft tokens get less reliable deeper in the sequence.

Aim for short, high-percentage sequences because while generating and validating tokens is inexpensive relative to generating tokens in the original model, it still comes with meaningful overhead. Additionally, once a single draft token is rejected as wrong, all subsequent tokens in the sequence are also rejected.

Working with speculation is interesting because so many factors affect token acceptance rate. The big one is the temperature parameter – higher temperatures yield token distributions that are harder to predict, reducing the effectiveness of speculative decoding. But even factors as simple as subject matter can make a difference on acceptance rate if the draft model or additional decoder head used for speculation is better versed in, say, math than history.

Another limitation on speculative decoding is that it’s most useful at low batch sizes where there are spare compute cycles. At higher batch sizes, speculative decoding must be dynamically disabled as compute is too saturated to afford verification.

Each speculation algorithm navigates these tradeoffs differently, and careful implementation of the right algorithm for the situation can lead to major improvements in TPS.

Approach #3: Caching

During prefill, the inference engine builds a KV cache (a store of keys and values for each token) on the input sequence. It then updates the KV cache for each token during decode. As inference is autoregressive, the value for each new token depends on the value of every previous token in the sequence.

Every inference engine uses KV caching by default on a request-by-request basis. Without KV caching, LLM inference would be unbearably slow since each previous value in the entire sequence would need to be recalculated for each subsequent token.

However, engineers can get more utility from the KV cache by reusing it between requests rather than solely within each inference sequence.

Consider the following two prompts, each with four tokens on most tokenizers:

By default, the inference engine has to run prefill on all four tokens of each prompt. But the first tokens of each prompt – “Weather in” – form a shared prefix between the pair.

With prefix caching, you can reuse the KV cache from the first request to improve TTFT on the second request by skipping prefill on the first two tokens and reading in the existing KV cache instead.

When you see pay-per-token APIs charge less for “cache hit” input tokens than “cache miss” tokens, this is why: reusing cached tokens takes very little compute power and time. As an inference engineer, you can apply the same principle to reduce latency, improve throughput and therefore save money on your own deployments.

Saving two tokens won’t make a big impact on TTFT, but prefix caching can skip prefill on thousands of tokens in certain domains:

• Complex system prompts: Agents, customer-facing chatbots, RAG scaffolds, and tool calls often feature long, complex system prompts on every call.

• Code completion: Code completion, code generation, and other coding functions require passing the same thousands of lines of code as shared context.

• Documents and retrieval: Document summarization, question answering, and retrieval all add repeated context ahead of user prompts.

• Multi-turn conversations: Ordinary conversations repeat back every message in a chat template, increasing the savings from prefix caching with every turn.

Prefix caching works from the start of the input sequence until the first non-repeated token. The fourth token in the weather example, a question mark, is shared between the two input sequences. However, the prefix ends at the first non-repeated token, so the fourth token isn’t read from cache.

Since prefixes end at the first unique token, your context engineering determines TTFT savings. Consider a different approach to the same prompt:

Here, there is no savings from prefix caching, as the very first token differs between the two sequences, even though every subsequent token is the same. To take advantage of prefix caching, ensure that novel tokens are as late in your prompt as possible.

Approach #4: Parallelism

Tensor Parallelism (TP) should be your default strategy for multi-GPU model inference. It supports dense models like Llama 405B, and the MoE (mixture of experts) models that currently dominate the open model landscape.

TP works by splitting apart each layer of the model (as opposed to Pipeline Parallelism, which keeps layers intact) and distributing the layer fragments across the allocated GPUs. For each layer, the expense of reading from weights’ memory and executing matrix multiplication is shared across the GPUs.

However, the results of each layer need to be communicated in an all-reduce fashion (across all eight GPUs) into a single output before the next layer can be computed. In nodes with high-bandwidth intra-node NVLink and NVSwitch, this communication overhead is minimized.

Increasing Tensor Parallelism improves TPS on a per-user basis, assuming the model is large enough and the sequences are long enough that the communication overhead doesn’t outweigh the faster forward pass – which is the case for most frontier models.

Expert Parallelism (EP) neatly divides experts across GPUs, so that in a model with 128 experts served in EP8 across eight GPUs, each GPU hosts 16 full experts.

EP improves total system throughput, making inference more scalable and less expensive. With individual experts processing tokens separately, each token takes just as long, but the system as a whole can handle more simultaneous tokens.

Many deployments use a mix of TP and EP to achieve both benefits.

EP requires less inter-GPU communication than Tensor Parallelism. The Expert Router, which determines which experts each token activates, is replicated onto each GPU as it is a relatively small component of the model. Inter-GPU communication is necessary for passing tokens from expert to expert, but unlike TP, it is not required to collect the results of each layer.

Thanks to this lower communication overhead, EP scales well to multi-node deployments and systems with limited interconnect bandwidth.

Approach #5: Disaggregation

Disaggregation combines three important ideas in inference engineering:

1. Prefill is a compute-bound process that determines the time to first token (TTFT), while decode is a memory-bound process that determines TPS.

2. Specialization improves performance in everything from kernel selection to inference engine parameter tuning.

3. You can effectively parallelize model serving over multiple GPUs, or even multiple nodes, if you can avoid bottlenecks from lower-bandwidth interconnects.

When prefill and decode run on the same node under heavy traffic, they have a higher chance of interfering with one another. Ideally, prefill uses more compute resources, while decode uses more memory, and the two can co-exist efficiently. However, with larger batches and more compute-intensive optimizations, prefill and decode start competing for resources.

Disaggregation, or disaggregated serving, is the idea of separating prefill and decode into separate engines on separate GPUs or nodes.

Disaggregation turns LLM inference into a three-step process:

1. The prefill engine takes the input sequence and generates a KV cache while computing the first token.

2. The prefill engine sends the KV cache over the hardware interconnect to the decode engine.

3. The decode engine computes all subsequent tokens.

In conditional disaggregation, the request is first sent to the decode engine, which checks if the input sequence is already cached, or is short enough to handle locally:

1. If so, the decode engine handles prefill locally, skipping disaggregation.

2. If not, the decode engine transfers the request to the prefill engine for disaggregated serving.

Conditional disaggregation is better for real-world traffic.

Another benefit of disaggregation is that with separate prefill and decode engines, you can optimize each engine individually and the system as a whole. For example, the compute-bound prefill engine requires a lower TP than the memory-bound decode engine.

Takeaways

This is Gergely again. Thanks to Philip for this deepdive into inference engineering, which is around 10% of the contents of his new book, ”Inference Engineering.” If you’d like to go deeper into this topic, you can download the full book for free:

Get the full e-book, for free

This title will also be available in physical, printed form: sign up to the waitlist to be notified when it’s available.

It’s encouraging that inference engineering is no longer a “monopoly” belonging to a few leading AI labs. Top AI model makers like OpenAI and Anthropic control all aspects of their AI models – from training to inference – so there’s no inference engineering to be done with them.

However, thanks to increasingly capable open models, engineering teams have the opportunity to tweak how they use models, and this is where the theory and practice of inference engineering becomes invaluable.

Even so, the discipline of inference engineering still seems to only make sense for a subset of tech companies. To justify investment in inference engineering, you need to be spending big money on inference from vendors. This is the point at which it can make sense to invest time and money to see if you can set up your own inference stack on top of open models, and swap out some existing usage.

I wonder if inference engineering is the AI version of the “build vs buy” dilemma. For software-as-a-service (SaaS), the question for every company is whether to build it in-house, or buy from a vendor. For example, should you build a project management software (it’s possible!), or just buy an existing one? And what about feature flagging, not to mention observability?

Experienced engineers all understand the pros and cons of building it yourself (time and maintenance, which is a constant drag.) Tuning and operating your own LLM stack is a much newer field, and inference engineering is at the heart of building better inference stacks than what comes “out of the box” with open models.

Picking up the basics of inference engineering feels like a valuable skill – and it’s also new and interesting. If you become well-versed in inference engineering, you could create optionality for your own team and company in LLM usage. Running your own inference stack on top of an open model gives control of what you’re running and of pricing. Inference engineering helps create options for achieving better performance from an open model by using the approaches covered in the extract above from Philip’s book.

After I left Uber, we stayed in touch, and a few months ago I heard he was exploring the next opportunity, and found out from him that Sam’s job search looked nothing like most people’s: he didn’t apply for a single role. Instead, there were reachouts from former colleagues desperate to hire them.

When we talked for this article, Sam had three warm leads which wanted to interview him ASAP. One startup was not even hiring, but the founder was ready to create a new position just for him.

I posted a message on LinkedIn about Sam:

“I hate the term “10x engineer” but this engineer is a role model for what a standout engineer is - in fact, some of my writing of standout engineers reference my interactions with folks like them (e.g. my article on the product-minded engineer, this one: https://lnkd.in/et7nWBgW)

And still, from the outside, this engineer is nearly completely invisible.

No social media footprint. The LinkedIn profile lists his companies worked at, and nothing else: no technologies, no projects, nothing. Their GitHub is empty for the last 5 years, and has perhaps a dozen commits throughout the last 10 years.”

This is Sam’s GitHub contributions for the last several years: absolutely nothing.

One of the most upvoted comments on my post was by cloud technologist Olivier Frolovs, who requested an article on Sam for others to learn how he operates. Now, Sam has generously agreed to an interview and has asked to remain anonymous, hence the nom de plume (pseudonym).

He doesn’t seek public attention and has a strong professional reputation. During our chat, he offered pointers for engineers looking to excel, and also as proof that an empty GitHub profile and zero social media presence don’t mean you can’t be a truly standout developer.

I also interviewed one of his former managers from Uber for that perspective. Today, we cover:

1. Getting things done. High-level task breakdowns, combined with communicating delays as tradeoffs to stakeholders.

2. Setting boundaries. Saying “no,” prioritizing family or work – and being clear about it, and treating prioritization as a daily practice.

3. Office politics. Participate selectively and cautiously, build relationships with influential colleagues, pre-sell ideas, direct communication.

4. Negotiation and conflict. Approach engineers before their managers, build bottom-up consensus, and start with relationship-building.

5. Promotions, keeping up-to-date, and finding the next job. A personal take on The Big Tech promotion processes, keeping up with the industry, and relying on referrals more.

6. Becoming a manager. Ownership and independence separate “good” engineers from “great” ones.

7. Feedback from Sam’s ex-manager. A former manager reveals what made Sam stand out to them – and shares some potential growth opportunities.

Background

In this article, my questions are in italic and Sam’s voice is in normal text.

Sam, how did you get into tech?

I was always intrigued by computers, software, and – as it came around – the internet. My dad had a black-and-white screen laptop for work with Windows 3.1 and some games. We got our first personal computer with Windows 95 and I remember it vividly.

I started developing websites for my primary school and the company my mom worked at when I was 12. I got paid a small sum, so that was my first-ever paid programming project! Around then, I taught myself Visual Basic 6 and started building 2D and 3D mini-games. I found the website DirectXVB (which is still live today), emailed the website’s owner with issues I ran into, and they helped me with pointers. Later, I taught myself PHP and built more dynamic websites.

From the age of 14, I stopped all coding – I just got tired of it! – and focused on my studies, and getting into college. I chose a non-computer science major for university, but picked up coding on the side, and I rediscovered that spark, so, in the first year of my Master’s, I decided to switch and do a Computer Science Bachelor’s. It was during college that I started to build apps and websites, and it’s when I got truly hooked on software development.

From agency, to large company, then Uber

I joined an agency as their first hire, building apps for local companies. It was a small team, we learned with trial-and-error, and getting done at 2-3 o’clock in the morning was common enough. I stayed 18 months and learned a lot about ownership, the importance of an eye for detail, and collaborating with others.

My favorite part of the job was the few times I worked directly with a designer: our agency employed freelance designers who were not involved in most of the projects because the company was trying to save money by having them work less, and not be involved in planning and rollouts. But during the implementation phase, I’d find myself talking with the designer and bouncing implementation and design ideas around.

I then joined a small startup where we built our own product. A highlight was having two designers on the team fulltime, whom I could work with and learn from. Engineering also felt like a level up: everyone cared about software quality and UX details.

Our startup got acquired by a larger company and most of us moved to the Bay Area. We stayed together as a team and were told we would maintain a “startup culture.” The founders tried their best to stay true to their word, but they couldn’t shield us from the reality of working in a corporation.

I learned a lot about corporate processes, and it was more interesting than I’d expected. As I was getting closer to the senior engineer level, I had to understand how internal politics worked, how to “massage” peer teams to help support our proposals, and how to talk with engineering leaders like senior managers and directors. Our company was also hugely focused on the annual company event: it was eye-opening for me to see just how much effort went into preparation. It consisted of several rehearsals and dedicated engineering work to showcase our stuff in a way that was near-flawless on the day.

After a few years, I felt ready for a change and joined Uber. I took a “title cut,” something akin to the “seniority rollercoaster.” At Uber, I worked in a new area and got promoted several times. After Uber, I worked at another Big Tech, and now – very recently – I’ve begun at a startup.

1. Getting things done

Feedback at Uber about you during performance calibrations was that you’re excellent at getting things done. What’s your process?

When I started out as a junior dev, I pulled long hours so I could deliver on time – regardless of how much effort it took. I don’t know what it was, but I always felt that failing to deliver on time was never an option.

I still vividly remember one project where I worked incredibly hard but still failed to deliver with the quality I expected from myself. As embarrassing as it is, I was so exhausted that I almost started crying on the spot. One of my coworkers comforted me and told me:

“Man, you’re crying about the wrong thing. No one died, no one got hurt, and no one will even care that we’re a few days late, save for the project manager. But even he’s used to everything being late. Go home, have some sleep, come back tomorrow and take it easy.”

They were right, of course. Still, I’m pretty sure this inner pressure to be unsatisfied with “good enough” explains a lot about how I work.

Having a high-level breakdown of the work, and communicating to stakeholders has been my “secret”, later in my career. After a few years as a dev, my estimation skills got better and I had to pull fewer late nights. I also found a hack that greatly helped was doing a high-level breakdown as early as possible, in all cases. As soon as I understand what the work is, I break it all down, ideally on a whiteboard or paper.

Importance of communication

You were also seen as a strong communicator, whether it was with engineers, engineering managers, or product managers. How do you get your point across?

Communicating delays as “tradeoffs” works extremely well. As soon as I start a project which I’m the lead on, I establish communication channels with key stakeholders: product managers, my engineering leadership, and business stakeholders, via email or Slack, I keep them in the loop at least weekly, and about anything that could be a roadblock.

In my experience, delays are not an issue as long as they are communicated upfront with an explanation and potential alternatives. When we hit a roadblock that slows down our work, I would never communicate that we’re “behind”. I would offer alternatives like:

• We can still ship on time, but we’d need to cut X and Y features for this release

• If we are not comfortable cutting X and Y features, then we will need to push out the target date by 2 weeks. If we are comfortable, we can push it out by 1 week

The trick, I’ve found, is to make it clear to stakeholders that we have a choice: choose more features to ship, or choose a lower-priority feature to drop.

I learned most of my hacks from people who are good at getting things done, and they have a few attributes:

• Task breakdown: early in my career, there was a senior engineer who was methodical about breaking down tasks and making estimates, even for seemingly trivial projects – and it worked!

• Communication tools: I observed the few really organized product managers, engineering managers, and tech lead, and made their communication styles into a “package” that worked for me; things like email updates, facilitating kickoff meetings, launch announcements (including how to communicate a failed/sunset project as a successful launch), and more.

Being good at communication means you have a solid foundation, and then develop a feel for how to best utilize the tools you have. There’s no “one-size-fits-all” approach: people react better or worse to different things. Try to get to know folks around you and put yourself in their shoes.

Doing great work

What does “standout” work look like to you?

I think about the quality of my work similarly to the quality of work I do at home. I have moved houses and renovated them several times and I greatly care about the quality of that work. And I’ve seen plenty of contractors come to my place, perform their work, and then leave without actually caring about the quality. They just want to “get s*** done” and be out of there. I never understood how someone can keep doing their job without feeling a lot of love for it!

I need to get energy from everything I do, not just in my job. Whether it’s playing games with my kids, helping my wife with her website, or building a new website feature for the company I work for: I approach it all with the same attitude.

Equally, if I no longer get energy from the work I do, then I basically stop enjoying it and this can be a nudge to start to look for something else. If it continues for a long time, this urge can become more persistent, and that’s the point when I have switched companies or teams. I can go on for some time without getting energy from my work, but it drains me. I try to catch myself before it gets too bad, and I’ve managed to do so, up to now. This is why I quit my last job without having anything lined up: I stopped getting energy from it for many months and talked with my management chain about it, but they were unable and unwilling to change anything. I needed a change, so it was me who made it.

Stepping outside of domain expertise

You frequently went outside of your domain, working with engineering teams on other platforms and contributing to codebases you’re not expert in. You seemed to have a great relationship with most engineers, in contrast to some devs. How did you do this?

I am pretty curious and prefer to talk directly with engineers. So, when I’d work on a project with engineers on a different stack, I would ask them to explain their high-level architecture approaches, and roll up my sleeves to make small code changes in a stack I was unfamiliar with.

Once you understand the high-level structure of a different codebase, and you also know how to make a few small changes, suddenly, it’s so much easier to figure things out on your own!

An approach that consistently worked for me is approaching problems from the customer’s perspective, and being genuinely curious. For example, I might ping an engineer working on a different system and ask:

“I noticed a customer has this problem, and to fix it, we probably need to touch the system you own. I don’t know much about this system: can you explain how it works, and what we could perhaps do to solve this issue that causes frustration for the customer?”

By making it clear that my goal is to solve a customer problem, I’m not coming across as just digging around for nothing. And by making it clear that I’d like to learn from them, it avoids being seen as someone trying to confirm what they are doing, which could come across as arrogant – especially when the other engineer is the expert on their own system. I’ve found fellow engineers are happy to explain their understanding and decisions.

2. Setting boundaries

At Uber, I recall you were very good at setting boundaries and saying “no.” How do you do that?

Honestly, I find it tough to say no – but I learned that it’s worse when I don’t. I found that saying ‘yes’ to everything usually results in an unmanageable, unbalanced pile of work. Prioritizing is key: I always remind myself to focus on what matters most. For me, the “most important” thing for any given topic could be:

• A shipping deadline that needs to be hit and is non-negotiable

• Family needs

• An urgent task that needs to be done on the same day

Family is very important to me. When I worked at Uber, I had a reasonably long commute to the office. I blocked out my calendar so I could leave on time in order to be home for dinner with my family. This did not mean I stopped work immediately; I would sometimes work during my commute and, when necessary, I logged back on to continue working after my kids were in bed.

When we had important deadline agreements at work, I made an agreement with my partner that I stayed longer in the office because I knew it was important to put in extra effort and deliver standout work then.

It goes back to prioritizing and focusing on the most important thing. Looking back, I’d say most of the time, the most important thing for me was family, and that work overrode this every now and then.

My approach to prioritizing keeps changing, though. Demands at home keep changing and expectations at work also change; after Uber, other jobs increasingly focused on async and remote work. This meant more flexibility to accommodate family time – but work could spill over into evening hours if I did not finish everything.

If I can give one piece of advice, it’s to understand what is important for you. Know your number one, number two, number three priorities: and arrange your workday so you do your top priorities. Don’t compromise on the most important one!

3. Office politics

At work, how plugged in were you to office politics?

I was aware of politics and tried to build relationships with “influential” people. I try to stay away from “cocky” types, and to find what I want to achieve through different folks.

The importance of politics is something I really started to understand when working at Uber. Initially, I was ignorant, but the more experience I got in Big Tech, the more it became obvious. It took a while before I was able to participate in it. I never liked it; I tend to be direct and transparent, but that does not work in every situation.

Did you take part in it to get stuff done?

Yes, sometimes by being direct and transparent, and communicating the right amount of information, you can get a lot done. Occasionally, it required me to “massage” an idea on multiple people before going to the person who called the shots.

What is your view of engineers who are seen as“political”?

It’s part of the game and sometimes it’s useful to have a good relationship with those people, as you can use that for your own benefit, as well. I personally would never invest much time in understanding and practising politics, as I prefer to focus on building and product.

4. Negotiation & conflict

Negotiating with teams

You were perceived as being good with other teams, and at removing roadblocks for your own. How did you approach this?

Read more

Last week, we took an inside look into how Uber is adopting AI, internally. The rideshare giant has built close to a dozen internal systems to deal with code generated by AI agents. However, when quantifying the impact of AI, the focus was on how much output has increased, and how devs who use more AI also generate more pull requests; these are the “power user” devs who generate 52% more PRs than devs who use AI less. There was no mention of product quality – at all!

And there are signs that product quality is dropping overall. Today, we dig into this under-discussed topic, covering:

1. Anthropic: degraded flagship website. An annoying UX issue irritated paying Claude customers – and no one at Anthropic noticed. The company moves very fast, generates 80%+ of production code with Claude, but quality and user experience seem to be taking a backseat.

2. Amazon: AI-agent reliance triggers SEVs. Amazon’s retail org has a leap in outages caused by its own AI agents. Now, senior sign off is needed for junior engineers’ AI-assisted changes.

3. Big Tech: “use AI or you’re unproductive.” Companies like Meta and Uber are tracking AI token usage in performance reviews, putting pressure on engineers to use it heavily — irrespective of the tools’ quality impact.

4. OpenCode: more time spent cleaning up. Dax Raad, OpenCode’s creator, warns that AI agents are lowering the bar for what ships, discouraging refactoring, and don’t speed teams up.

5. 5. Startups: founders see LLMs slowing down long-term velocity. Sentry’s CTO and others observe that while AI removes the barrier to getting started, it also produces bloated, hard-to-maintain code that slows long-term development.

6. Research: AI agents underperform claims. Some studies show AI coding tools produce short-lived velocity gains followed by significant tech debt increases.

7. How do we solve it? Engineers with strong architectural sense become more critical than ever, proposed solutions include formal validation methods, and perhaps reviving some old school QA ideas.

1. Anthropic: degraded flagship website

This article’s genesis was last week, when I’d finally had enough of a persistent UX bug on Claude’s flagship website: the prompt I typed in regularly got lost. Below is a video of me typing “How can I…” – and “losing” the first two words when the page loaded:

It’s pretty straightforward:

1. The page starts to render and the textbox is displayed

2. The user starts to type their prompt, but the page has not finished loading subscription data

3. The subscription information loads around a second later

4. The textbox is reset and the typing is lost

This is a pretty basic bug you might expect in a prototype, except that this is the landing page of Claude.ai, and it’s a bug that impacts every paying customer – easily millions – every day. Even worse, the bug happens every time you visit the site.

Somehow, nobody at Anthropic tested the site to catch a plainly obvious bug which impacted 100% of paying customers. At the same time, no company uses AI coding tools more than Anthropic: around 80% of the company’s code is now generated by Claude Code, so we can assume a good part of the website is also created that way.

My complaint about Anthropic’s website being broken went a bit viral and got the attention of the developer team:

To their credit, three days later the bug was gone. There’s no longer a “double load” of the textbox: it takes a bit longer to load but only does so once.

Still, it makes me wonder how much longer this issue would’ve continued had nobody complained. Also, how many more bugs are present on the Claude website that nobody highlighted on social media? How many more features could be shipped in a state that is subpar for production-grade software with millions of paying customers?

Anthropic seems to be prioritizing moving very fast over doing so with high quality. There is no denying that the company is moving at incredible speed and running laps around competitors. A good example is how they built Claude Cowork in just 10 days. Claude Cowork handled work with Microsoft Word and Excel documents surprisingly well, to the point that it set off a “code red” inside Microsoft’s Office division, I understand.

Microsoft responded as fast as possible, but it still took 2-3 months to launch their (cloned) response, called Copilot Cowork earlier this month, with full access still to follow soon.

In the case of Anthropic, moving fast with okay quality seems to make good business sense: they build a better product than what already exists, so no matter if it’s a bit rough around the edges; they can fix quality issues post-launch and still be months ahead of the competition.

2. Amazon: reliance on AI agents causes SEVs

Anthropic can afford to move fast while it’s growing at an extremely high rate and expanding its market share rapidly. At the same time, established players like Amazon have extreme focus on reliability: AWS has become the top cloud provider not least by being extremely reliable (as well as aggressive on pricing).

Well, reliability at the online retailer seems to be getting worse, too, and the company’s AI agent, Kiro, could be causing SEVs (Amazon’s phrase for “outage”), according to The Financial Times (emphasis mine):

Amazon’s ecommerce business has summoned a large group of engineers to a meeting on Tuesday for a “deep dive” into a spate of outages, including incidents tied to the use of AI coding tools.

The online retail giant said there had been a “trend of incidents” in recent months, characterised by a “high blast radius” and “Gen-AI assisted changes” among other factors, according to a briefing note for the meeting seen by the FT.

Under “contributing factors” the note included “novel GenAI usage for which best practices and safeguards are not yet fully established”.

“Folks, as you likely know, the availability of the site and related infrastructure has not been good recently,” Dave Treadwell, a senior vice-president at the group, told employees in an email, also seen by the FT. (...)

He asked staff to attend the meeting, which is normally optional.

Junior and mid-level engineers require more senior engineers to sign off any AI-assisted changes, Treadwell added in the briefing note.”

This meeting was the regular “This Week in Stores Tech” operational one, but what was new was the note telling staff to attend this “optional” meeting, and the mandate for senior engineers to sign off code changes from juniors. The outages may have been caused by less experienced engineers over-trusting GenAI’s output. Also, there were incidents caused by AI changes, said the FT:

“Separately, the company’s cloud computing arm — Amazon Web Services — has suffered at least two incidents linked to the use of AI coding assistants, which the company has been actively rolling out to its staff.

AWS suffered a 13-hour interruption to a cost calculator used by customers in mid-December after engineers allowed the group’s Kiro AI coding tool to make certain changes, and the AI tool opted to “delete and recreate the environment”, the FT previously reported.”

Again, a tool causing an outage is not its own fault: it’s on the engineer who lets the tool run wild. If I delete two lines of code, then push it to production, and the server crashes, the fault is not with the text editor or the Git client, but with me who made the change. Similarly, if you prompt an AI agent to do something, and the AI agent goes off and does its stuff which causes an outage, then responsibility lies with the engineer who didn’t set up guardrails for the agent.

However, there is the issue that AI agents can wreak havoc in ways devs don’t quite understand or expect, until learning the hard way. This was what took down a lesser-used AWS service, according to the report:

“Amazon Web Services experienced a 13-hour interruption to one system used by its customers in mid-December [2025] after engineers allowed its Kiro AI coding tool to make certain changes, according to four people familiar with the matter.

The people said the agentic tool, which can take autonomous actions on behalf of users, determined that the best course of action was to “delete and recreate the environment”.

It sounds like an engineer gave overly broad permissions to the coding agent which then used its scope to delete a service. As mentioned, the engineer is responsible, but there is also a learning curve with these AI agents to consider: it’s not like this type of outage happened in the past. Plus, companies like Amazon are heavily incentivizing using AI agents for as much work as possible, which naturally leads to overuse.

3. Big Tech: “use AI or you’re unproductive”

Something happens at places that measure devs’ AI usage: pressure builds for all devs to use more AI or else be seen unproductive and at risk of poor performance reviews, potentially leading to a PIP or worse.

Meta is taking token usage into account during perf reviews. A current engineering manager at the social media giant told me that the token usage of each engineer is now a data point — one of many! — for performance calibrations. By itself, it is not a positive or negative signal, but someone perceived as having low impact and with low token usage is now seen as a blatant low performer. For high performers with outstanding impact, very high token usage is seen as a good thing as it conveys to the manager group that they’re personally invested in AI and are improving their workflow – as proved by results.

We previously covered how performance calibrations work at places like Meta.

Big Tech CEOs are starting to see AI “power user devs” as superior to their coworkers. Uber is a good example: the Dev Platform team started to analyze the output of engineers by whether or not they’re in the “power user” category, meaning they use AI agents at least 20 days per month. They found more PR output by engineers who are power users. So far, that’s useful data, but it’s just one piece of information, and doesn’t reveal the quality of the PRs, the impact of the engineer, or any other business outcome.

By the time this data reaches CEO level, it has turned into something else. Here’s Uber CEO, Dara Khosrowshahi, interpreting the same data points on the Diary of a CEO podcast (emphasis mine:)

“While 90% of our engineers are using AI tools of some sort, there’s about 30% of them that are using them at a completely accelerated pace. And it [using AI tools heavily] really is changing their productivity in a way that I’ve never ever seen before.”

There’s a step from observing more PRs per engineer, to judging power users as being more productive for that reason. Dara continues:

“I can imagine maybe 5 years from now, as the engineers get more and more productive, that I may not decide to add engineering headcount because at that point instead of adding an engineer, I should add agents and buy some more GPUs from Nvidia. That may be the investment in the future.”

Unsaid in the above is that by that time, only engineers “using AI at a completely accelerated pace” would be employed. Would it also mean that engineers not on the bandwagon are on the way out? I appreciate Dara speaking his mind and shedding light on the thought process of a Big Tech CEO.

Inside large tech companies, it’s becoming a career risk to not use AI at an accelerated pace, regardless of output quality. These large companies are the ones likely to be mulling layoffs, like Meta reportedly preparing to cut up to 20% of staff. And when it comes to identifying redundancies, it’s a fair assumption that things like “AI usage” and “pull requests per engineer” will be taken into account, especially as one theme of such layoffs will almost certainly be that the employer wants to focus more on AI.

So, it’s common sense (and self-preservation) to use more AI, if only not to be seen as unproductive. Their perceived output will rise and engineering leadership will share more reports about productivity being up, and interpreting more code generated and more pull requests as the proof.

4. OpenCode: “more time spent cleaning up”

Dax Raad is founder and CEO of OpenCode, an open source AI coding agent, into which you can plug in models like Claude, ChatGPT, Gemini, and others. It’s an increasingly popular alternative to the likes of Claude Code and Codex. In our recent AI tooling survey, it came up as a tool used nearly as much as Google’s Gemini CLI and Antigravity. A small team works on this increasingly influential tool, and is seeing problems with AI overuse. Dax wrote this note to the OpenCode team (emphasis mine):

Read more

Update on 11 March: the Uber team shared updated numbers as of March 2026:

• 84% of devs at Uber are agentic coding users (either using CLI-based agents or making more agentic requests than tab-completion in their IDE)

• 65-72% of code is AI-generated inside IDE-based tools. This number is, naturally, 100% for AI command line tools like Claude Code.

• Claude Code usage nearly doubled in 3 months — from 32% in Dec to 63% in Feb, while IDE-based tools (Cursor, IntelliJ) have plateaued.

The article is updated with these new and correct numbers.

I spent four years working at Uber until 2020 and experienced firsthand the company’s standout engineering culture. Uber is a company that did the speed run of going from a small startup, through hypergrowth, to being a large company facing major risk during the pandemic, when the rideshare business briefly collapsed. Today, it’s maturing as a publicly traded, profitable company, and employs almost 3,000 people in the tech function.

At the recent Pragmatic Summit in San Francisco, one of the most interesting behind-the-scenes sessions came from the ridesharing company’s principal engineer, Ty Smith, and director of engineering Anshu Chada, who pulled back the curtain on what Uber has been doing with AI tools, internally. They were candid about the amount of work it took to build up Uber’s internal “AI stack,” why all that work was necessary, and also discussed the drawbacks as well as benefits of this rapidly spreading technology. You can watch their presentation at The Pragmatic Summit here.

In today’s issue, we cover:

1. Agentic layers & systems. Four layers spanning an internal AI platform, context sources, industry tools, and specialized agents for testing and code review.

2. Internal tooling: MCP Gateway, Uber Agent Builder, and the AIFX CLI. Uber built several internal tools to make it easier for devs to use AI tools, and to make internal AI agents more effective.

3. How AI changes developer workflows. A move away from single-threaded coding in an IDE, to orchestrating multiple parallel agents. Engineers naturally gravitate toward kicking off new agents, which starts to create resource and cost challenges.

4. Minion: running background agents at scale. Uber built Minion, an internal background agent platform with monorepo access and optimized defaults. It’s a clever abstraction layer that works well in practice.

5. New internal dev tools. More AI-generated code means more code reviews and more noise, so Uber built Code Inbox for smart PR routing, uReview for high-signal AI code review comments, Autocover for generating 5,000+ unit tests per month, and Shepherd for managing large-scale migrations end to end.

6. Challenges. AI adoption is slower than expected, even at a forward-thinking company like Uber. Top-down mandates are less efficient than engineers sharing their wins with peers.

7. Impact in numbers. 92% of Uber devs use agents monthly, 65-72% of code is AI-generated inside IDEs, and 11% of pull requests opened by agents. At the same time, AI-related costs are up 6x since 2024, and token cost optimization is a growing priority.

Longtime readers might recall we’ve covered Uber’s engineering culture over time:

• Developer Experience at Uber – with Uber’s founding engineer on Developer Platform, Gautam Korlam (2025)

• How Uber built its observability platform (2023)

• Inside Uber’s move to the Cloud (2023)

• How Uber is measuring engineering productivity (2022)

• The Program and Platform split at Uber (2021)

Let’s get into it:

AI is not new at Uber, but rolling it out companywide is. The company has used machine learning and AI technologies in many systems, including its Marketplace platform, which are responsible for routing and matching drivers with riders, forecasting demand, etc. What is relatively new at nearly all tech companies is the process of integrating AI across engineering and beyond.

The official strategy at the ridesharing giant is to become a “GenAI-powered” company:

I appreciate Uber sharing this approach openly because while most companies say that they want to be “AI-powered” – however cliche that claim might be – not all provide as much transparency.

It’s worthwhile engineers internalizing how leadership views AI. These folks, in general, see a tool that can bring efficiency everywhere. My take is that in some ways, AI is seen similarly to the cloud, which has been perceived as a means to reduce costs and improve the flexibility and elasticity of hardware resources. Today, AI is seen as the way to increase efficiency and lower costs, such as customer support, software development, the finance function – or any function.

Uber is focusing not on automating everything possible in engineering. Instead, it wants to:

1. Eliminate toil: helping AI do “boring” work like upgrades, migrations, trivial bug fixes, etc.

2. Free up engineers to focus more on creative work.

As Anshu Chada, Engineering Director on Uber’s Dev Platform, puts it:

“What we found is when we push some of the boring stuff to AI – upgrades, migrations, bug fixes – not only does it result in much higher satisfaction from our engineers, but they’re able to push our product and create features for end users in ways that we didn’t even think were possible.”

1. Agentic layers & systems

Uber’s “agentic system” for software engineering is actually made up of several systems:

Categories of systems:

• Internal AI platform. Built on top of Michelangelo, Uber’s ML/AI platform. This layer provides things like a model gateway to proxy to frontier models or internally hosted models.

• Internal Uber context: accessing Uber’s source code, engineering documentation, Slack information, JIRA tickets, etc. These all serve as “memory” for agents to use.

• Industry agents: Uber’s approach is to enable the “latest & greatest” AI agents for engineers, so they support several tools like Claude Code, GitHub Copilot, Codex, and other clients.

• Specialized agents: Uber’s background agent platform, the test generation platform, code review agents, and more.

• Engineering enablement: measuring the efficiency of agents, controlling costs, and educating engineers about which tools to use.

2. Internal tooling: MCP Gateway, Uber Agent Builder, and the AIFX CLI

MCP – the Model Context Protocol – has quickly become the standard way to connect agents and data sources with one another. A frequent analogy is that MCP is like the “USB-C interface for AI agents.” We published a deepdive on the MCP protocol and covered real-world MCP server use cases.

Uber put together a “tiger team” (a temporary unit that gets things done fast) to design the MCP strategy and build the central MCP gateway, which looks like this:

This MCP gateway allows:

• Proxy internal endpoints to MCPs: any internal Thrift, Protobuffer, or HTTP endpoint can be exposed as an MCP server with a simple configuration change. Uber uses the Apache Thrift protocol and Protobuffer protocols extensively for backend service communications

• First-party MCPs: these are exposed as a single, consistent interface

• Third-party MCPs: external MCP servers are also exposed via the gateway, which handles all authentication and authorization tasks.

• Platform concerns: the gateway takes care of authorization, telemetry, and logging in one central place. Plus, it conveniently offers a unified interface to interact with any MCP.

The MCP gateway also provides:

• A registry: to look up MCP servers, and for devs to be able to register their own.

• A sandbox: for devs to experiment with MCP servers without long-winded setups.

Uber Agent Builder

Uber’s Agent Builder product is a no-code solution to build agents that can access Uber’s internal data sources (both MCP servers and Uber data sets), and hand off work to other agents:

The platform includes a tool called Agent Studio, where multi-agent workflows can be visualized, debugged, traced, versioned, and evaluated. This is how it looks:

The agents built in Agent Builder become discoverable through a registry:

Accessing agents via the AIFX CLI

Uber’s Developer Experience platform team had a few issues with deploying AI agent tooling at scale:

• How does the company update all clients when a new version comes out? For example, if a new version of Cursor is released, how can they ensure all devs use the latest one?

• How can the clients be configured with helpful defaults? Uber’s Dev Platform team might have found better defaults for tools, like more helpful models. How are these rolled out to all devs?

• How can devs easily discover MCP servers and configure them for agents?

• How can agents connect to Uber’s background task infrastructure?

The Dev Experience team built the AIFX CLI, which is the AI tooling command line used by all engineers there. Here’s what it looks like:

This tool supports:

• Provisioning AI agents (client tools like Claude Code, Codex, Cursor, and others)

• Finding and using MCP servers

• Running background agent tasks

• Updating AI agents and clients to the latest versions

3. How AI changes developer workflows

The traditional way of building software:

Devs spent some time planning, most time writing code (code authorship), and then some time in code review.

The first AI agent-based workflows were single-threaded: devs worked with a single agent in the command line or inside their IDE:

At Uber, the latest workflows which many software engineers follow are pretty different, involving parallel agents, each kicked off with their own tasks:

As Ty explained, running multiple agents comes naturally to most devs:

“[Once you start using agents] as an engineer, you’re giving a prompt and waiting for something. While it’s running and you’re waiting you’re thinking: ‘What am I going to do? Have a coffee or browse Reddit? Might as well kick off another background agent.’

And so, engineers get into this mode of running several agents at once, right? Both us [at Uber] and a lot of the industry is trying to push towards this.”

4. Minion: running background agents at scale

Read more

This article provides a high-level overview of those findings from our latest AI tooling survey. Thank you to everyone who participated!

There are plenty of interesting details, most notably, validation of just how much Anthropic and Claude Code have risen to dominate tooling usage. Claude Code is today nearly as widespread as GitHub Copilot was in our survey three years ago (in the spring of 2023) – which shows how fast the AI market moves.

In today’s issue, we cover:

1. Interesting findings: Claude Code has rocketed to #1 in just eight months, AI is fully mainstream with 95% weekly usage among respondents, and more.

2. Most-used AI tools: Claude Code leads the pack, followed by chatbots and GitHub Copilot. Cursor’s rising fast, and newcomers like Codex and Antigravity are gaining traction. Most engineers juggle two to four tools at once.

3. Popular models: Anthropic’s Opus and Sonnet models dominate coding tasks by a wide margin, with more mentions than all others combined.

4. AI trends: mainstream adoption achieved. 95% of respondents report using AI tools at least weekly, 75% use AI for half or more of their work, and 56% report doing 70%+ of their engineering work with AI.

5. AI agent usage rising: 55% of respondents now regularly use AI agents, with staff+ engineers leading adoption on 63.5% usage in the survey results. Agent users are twice as excited about AI as non-users are.

6. Company size and tool usage: smaller places overwhelmingly favor Claude Code (75% at the tiniest businesses), while large enterprises default to GitHub Copilot. This popularity is probably down to enterprise procurement preferences – and Microsoft’s enterprise marketing efforts.

7. Tools engineers love: Claude Code is the most loved tool at 46%, far ahead of Cursor on 19% and GitHub Copilot at 9%. Senior leaders are especially enthusiastic about Claude Code.

8. Demographics: Overview of who took part in the survey.

See also the follow-up parts:

• The impact of AI on software engineers in 2026: key trends. Part 1. Concerns about mounting AI costs, more engineers hitting usage limits, and AI tools having uneven effects upon different types of engineers

• The impact of AI on software engineers in 2026: key trends. Part 2. Tradeoffs of AI tooling, why adopting AI at company-level is hard, what’s changed in two years, and more.

Full subscribers also have access to a more detailed, 65-page report on our findings.

1. Interesting findings

Here are my ten personal, most-interesting findings from this survey:

1. Claude Code has gone from zero to be the #1 tool in only eight months. Released in May 2025, it’s already the most-used AI coding tool, overtaking GitHub Copilot and Cursor.

2. AI is now mainstream. 95% of respondents use AI tools at least weekly, or more often, and 75% use AI for at least half their software engineering work. Among readers of The Pragmatic Engineer, it seems the question is no longer whether to use AI in day-to-day work, but which tools to use.

3. Cursor is catching up fast on GitHub Copilot. As much as we hear about companies dropping Cursor for Claude Code, Cursor is doing more than fine, growing in mentions 35% since our previous survey nine months ago.

4. Most engineers juggle multiple AI tools. 70% use between two and four tools simultaneously, while 15% use five or more.

5. Staff+ engineers are the heaviest agent users. 63.5% use agents regularly; more than regular engineers (49.7%), engineering managers (46.1%), and directors/VPs (51.9%).

6. Codex is seeing explosive early growth. Despite not existing during the last survey, OpenAI’s Codex already has 60% of Cursor’s usage (!!)

7. Using agents correlates strongly with being positive about AI. People using agents are nearly twice as likely to feel excited about AI while non-users are twice as likely to be skeptical.

8. Company size influences tool choice more than preference. Huge companies (10K+) more likely to use Copilot (56%), tiny startups mostly go with Claude Code (75%) and Cursor (42%). It seems like enterprise procurement, not individual preference, is behind this divergence.

9. A tight chatbot race. ChatGPT, Gemini, and Claude as standalone chatbots have nearly equal numbers of mentions, suggesting there’s no clear winner outside of coding-specific tools among software engineers.

10. Directors and senior leaders are especially into Claude Code. The survey finds this tool is twice as popular with these folks as it is at less senior levels, while Cursor gets less love as seniority increases.

Let’s jump into some of the data:

2. Most-used AI tools

Just eight months after its release, Claude Code is already the most-used tool, overtaking both GitHub Copilot and Cursor:

Tools mentioned, in order of popularity:

1. Claude Code: a terminal-first coding agent from Anthropic. We cover its history in the deepdive, How Claude Code is built

2. Chatbots: ChatGPT, Claude, Gemini, and others

3. GitHub Copilot: launched in 2021, it’s the “oldest” AI coding tool on this list

4. Cursor: agent-powered coding IDE. Learn more about it in our deepdive, Real-world engineering challenges: building Cursor.

5. Codex: OpenAI’s AI coding agent. We recently covered how Codex is built

6. Gemini CLI: Google’s command line agent

7. OpenCode: the most popular open source coding agent, where you can swap out the model being used, sidestepping vendor lock-in

8. Antigravity: Google hired the original team behind Windsurf and launched its own agentic IDE

9. JetBrains Junie: the AI coding agent by JetBrains

10. Zed: a fast editor with agentic workflows

11. Windsurf: when Google acquihired the team, it did not buy the product. An AI IDE, now owned and operated by Cognition

12. Amp: a model-agnostic coding agent. A free version is supported by ads

13. Augment Code: an enterprise-focused coding agent

14. Factory: agent-native software development, which calls agents “droids.”

It’s interesting to compare how people answered the same question just nine months ago, last May:

Notable trends:

• Claude Code has massive momentum and is already the market leader. Since being released in May 2025, it has become the most-used AI coding tool among survey respondents. Anecdotally, I’ve heard several teams that have been Cursor power users are introducing Claude Code with great success (while still keeping Cursor in-place.)

• Cursor has grown circa 35% in nine months and now threatens GitHub’s popularity. In this survey, 35% more respondents mention using Cursor than in our previous research, last May. Such growth is impressive: at this rate, Cursor will have more users than GitHub in 6-9 months!

• Chatbot usage remains high. Combined mentions for ChatGPT, Claude, Gemini, Perplexity, and others used outside of coding apps are still higher than for any other tool except Claude Code. For context, the most-mentioned chatbot (ChatGPT, 107 times) only has as many as Gemini CLI (also 107).

• GitHub Copilot usage stable. Nine months ago, 46% of respondents said they used GitHub Copilot. Since then, it’s barely risen.

• Explosive growth for Codex. OpenAI’s Codex wasn’t available during our last survey. But it now already has 60% of Cursor’s usage(!), and could be growing even faster since this survey.

• Up-and-coming tools: OpenCode, Gemini CLI, Antigravity. None of these had launched nine months ago, but today they’re used by around 10% of respondents – no small feat!

• Tools growing more than before: The likes of Zed, Windsurf, Amp, Augment Code, and Factory appear less often in these results, but all are mentioned more by respondents than they were nine months ago.

Chatbot usage: ChatGPT leads – just about

Here’s how mentions of chatbot usage line up in our survey:

Tool usage: multi-tool

Most tech professionals use between two and four AI tools. An interesting detail is how many different ones are mentioned by respondents:

Noteworthy details:

• 70% of survey participants mention using between two and four tools

• 15% use a single tool

• 15% are using 5 or more

3. Popular models

Anthropic’s Opus and Sonnet dominate the ranking of models used for coding.

This is not even a contest: Opus 4.5 and Sonnet 4.5 (latest models at the start of our survey) come up more often than all other models, combined. Anthropic has become the go-to model developer for coding-related work – for now, that is. When this survey launched, Opus 4.6, Sonnet 4.6, and GPT-5.3 were not yet out.

Around 1 in 8 respondents say they just use whatever model is the default at their company. This is interesting to note: these are likely folks who might not bother changing default settings, and just go with whatever’s available. If the default model is powerful enough, that’s fine, but if the company’s default is a cheaper, less capable model, then these people could face a more frustrating experience than those who get to choose what they use.

In the “other” category of models, some other mentions include:

• Cursor’s custom Composer/Composer-1 model, and its “Cursor Auto” auto-select model

• Kimi/Kimi K2.5 — Moonshot

• DeepSeek model variants like R1, V3.2, Coder

• Alibaba’s Qwen/Qwen3

• xAI’s Grok

• Various Mistral models

4. AI trends: Mainstream adoption achieved

How often do people use AI tools? Very often, as it turns out; 95% of respondents are using them weekly, at a minimum:

It’s worth reflecting on this data in relation to insights presented by Laura Tacho at the recent Pragmatic Summit in San Francisco:

It seems that AI is now mainstream in software engineering. Anecdotally, this has been my sense since the beginning of the year: everyone whom I talk with is using AI tooling on a roughly daily basis. Now, there’s data to prove it.

How much software engineering work gets done with AI?

This year, we asked readers to estimate the percentage of their software engineering work that’s done using AI. The results:

The data show that AI is embedded in the workflows of participants in our survey:

• Only 25% of respondents use AI for less than 40% of their work

• 56% do 70% or more of their work using AI

• 75% use AI for at least half of their software engineering work

5. AI agent usage rising

Eighteen months ago, AI usage was mostly for code generation and tab completion. There were one or two respondents who experimented with early AI agents in March 2024, as something equivalent to a junior software engineer.

This year, 55% say they regularly use AI agents. This is 507 people, a massive jump!

The most common use cases for agents:

• Code review and code validation

• Automating manual / annoying tasks

• Bug fixing / investigating bugs

• Code investigation

• Debugging

• “Crafting” code, or “weaving” it together with the agent

Below is a typical-enough comment from one software engineer who uses agents at a smaller company:

“I use agents for pretty much all coding work, mostly prompting with Cursor Chat. I use it for code investigation, bug investigation, creating commits and pull requests. It’s my tool for reviewing code, I am always still in the loop. Almost all of my AI-written code is still reviewed and ‘crafted’. When using it for code review, I find it helpful to chat with and gain understanding, rather than letting it loose on code review. So, I use it for everything, but I am still very much in the loop.”

A common arrangement in the survey is the split-screen setup: a terminal with Claude Code open to drive work, and an IDE also open to review changes made by the agent.

The tools mentioned by those who regularly use AI agents split almost identically to the broader survey results:

Staff+ engineers are the heaviest users of agents. Here’s the data on agent usage by experience level:

This data point is slightly amusing because it shows there’s not much difference between the lowest and highest levels: 46% of leads and engineering managers say they use AI agents regularly, while for Staff+ engineers it’s 63%. Does it suggest the most experienced engineers are also the most curious?

The more someone uses AI, the more they also use AI agents. We segmented the data by the percentage of software engineering work that respondents do with AI:

• Heavy users have AI for 80% or more tasks, or use AI on an hourly basis

• Moderate users have AI for 30-80% of tasks, or use AI on a daily basis

• Light users employ AI for 30% of tasks or less, or use AI on a weekly or monthly basis

The more positive someone is about AI, the more likely they are using agents on a regular basis. In contrast, those negative about AI barely use agents:

One question is whether this finding indicates correlation or causation: that is, does starting to use AI agents more, actually cause people to feel more positive about AI?

A couple of details:

• Using agents seems to make people nearly twice as enthusiastic about AI (61%) as those who do not use them (36%)

• Respondents who don’t use agents are twice as likely to be skeptical (22%) about AI than those using agents regularly (11%)

It looks like that if you don’t use AI agents on a regular basis, you may have a negative opinion about the tools, in general, which could come at the cost of not experiencing what the technology has to offer.

6. Company size and tool usage

In our results, company size and tooling choice correlate for some tools; for example, the smaller a team or company is, the more likely it is to use Claude Code or Codex:

Claude Code is used by a whopping 75% of the smallest companies and teams. This is a big number, far ahead of any other tools. At the smallest places, the most-used tools are, in order:

1. Claude Code: 75%

2. Chatbots: 55%

3. Cursor: 42%

4. GitHub Copilot: 35%

5. Codex: 26%

6. Gemini CLI: 14%

7. OpenCode: 13%

GitHub Copilot overtakes Claude Code at large companies. This confirms what was known: Microsoft is very good at enterprise sales, and at bundling GitHub Copilot in its suite of products:

Cursor and OpenCode usage drops at massive companies with similar usage patterns. Across the board, usage is roughly the same, regardless of company size. We only see a drop at the very largest of companies with 10,000+ employees:

One theory for such a drop at massive companies could be that large companies often build their own internal coding agents that engineers use; e.g., at fintech, Block, the agent is called Goose, Meta has its own agent, as does Google with Jetski (its version of Antigravity) and Cider.

Google’s tools are evenly used across the spectrum of company size. Google’s Gemini CLI and Antigravity are the only tools in this survey whose usage is notably stable, regardless of company size. Both tools hover at around 10% from the smallest to largest workplaces:

The feeling among survey respondents that they can experiment at work, correlates with Claude Code being available. We asked if readers experiment frequently with tools. Below are the “yes” responses by company size:

When these responses are mapped to the percentage of people using Claude Code, there’s a very similar distribution. My theory is that Claude Code is new enough at 9 months old to have not yet been approved at companies with bureaucratic processes for approving new tools, and this is partly why some respondents feel their chance to experiment with the range of tooling is being thwarted.

Engineers at places with lots of red tape for tooling are less empowered to experiment with new AI tools; not just Claude Code, but any new, interesting tool.

7. Tools engineers love

We asked respondents: “Which AI tools do you love using the most, and why?” Below are the tools which respondents enjoy most, in descending order of number of mentions:

Claude and Cursor stand out in terms of how much they are loved. A whopping 57% of respondents mention either Claude Code (46%) or Claude models (11%) as the tools they are attached to. Cursor was at a respectable 19% – double GitHub on 9%.

Other notable tools with two or more mentions:

• Warp: a terminal for building agents

• Zed: a fast editor with agentic workflows

• Amp: a model-agnostic coding agent. A free version is supported by ads

• Cline: an open source coding agent

• RooCode: an open source, AI-powered coding assistant running in VS Code

• Continue.dev: AI checks on every pull request

Claude Code is especially loved by Director-and-above folks. Segmenting the “most loved” responses by level (engineers up to the senior levels of staff+ engineers, leads/eng managers, and Director+ folks):

Both Claude Code and Cursor become less loved – or used! – as seniority goes up, but it’s notable that folks in senior engineering leadership positions are obsessed with Claude Code, but not Cursor.

GitHub Copilot is equally loved by engineering managers as Cursor is — and this is surprising:

Both OpenCode and GitHub Copilot are surprisingly loved by Staff+ engineers. Another unexpected detail is that, despite OpenCode being used about a quarter as much as GitHub Copilot, it rivals the “loved” mentions. For Staff+ engineers, it matches GitHub on those terms:

Finally, when segmenting based on company size, we see the familiar pattern of Claude Code vs GitHub Copilot. Claude Code is less frequently mentioned as a “loved” tool as company size increases.

GitHub Copilot sees the opposite trajectory: it’s more loved within larger companies where it’s likely to be harder to experiment with alternatives.

8. Demographics

In closing, below are details about who took the survey, and how the 906 responses came together.

Engineers comprise 55% of respondents, and engineering leadership another 34%:

Respondents are experienced professionals. The median respondent has 11-15 years of experience:

Company size is also a fairly even split across this group:

Region-wise, most respondents are in Europe or the US:

Takeaways and the full report

We have compiled additional findings from this survey which did not fit in this article: it’s a 35-page article:

Read more

It was well-timed, considering that right now seems like a period of change for tech that’s unfolding faster than before. That was a consensus opinion at both events, also held by veterans like Martin Fowler and Kent Beck, who said things haven’t shifted so rapidly during their 50+ years in the industry.

At the very start of this year, I predicted AI will write almost all code, going forward, and several others have said the same. But at the Pragmatic Summit, I met an embedded engineer writing Assembly and C code who is still writing more of his code by hand than with AI agents – and was the only person I spoke with in San Fran who was not yet “giving in” to AI agents.

Even so, today, this engineer has between a third and a half of their own low-level code being generated by AI agents since the launch of Opus 4.5, and this share keeps on rising. Their view was an interesting counterpoint to the prevailing trend.

This article shares some thought-provoking ideas and conversations from both events, covering:

1. Data vs hype: how orgs actually win with AI. Laura Tacho’s keynote at The Pragmatic Summit. Exclusive data reveals 92% of devs use AI coding tools monthly (!!), “unhealthy” orgs see 2x more incidents, healthy ones have 50% fewer, and other new data.

2. Building world-class engineering orgs in the AI era. The closing session of the summit dug into what AI-native teams look like, in a fireside chat with GitHub’s former CEO and the CTO of Atlassian.

3. The Future of Software Development with Martin Fowler. Laura Tacho, distinguished engineer Annie Vella, Martin Fowler, and myself, look back on the event in Utah. Nobody has the AI shift all worked out, which is reassuring!

4. Mid-level engineers’ quiet crisis. Something I heard that engineering leaders talk about behind closed doors a lot is that mid-career engineers are being left behind by the AI wave. New grads are more productive with the tools, while seniors have more of that all-important experience. Advice on how to catch up fast.

5. Future of refactoring in an AI-native world. Refactoring legacy code has long been an important tool for keeping software maintainable. With AI generating most code now, is there any point? One answer is “hell, yeah!”

6. Future of Agile. It was only fitting to consider the future of Agile on its 25th anniversary, in the company of two of the writers of the Agile Manifesto. Amusingly, one major trend is the return of Extreme Programming (XP) practices which predate Agile.

1. Data vs hype: how orgs actually win with AI

At the debut Pragmatic Summit in SF, Laura Tacho delivered the keynote as a well-known expert on developer productivity, and former CTO of DX. She presented new and exclusive data on AI usage, and what currently works with AI adoption. Her session was one of the most popular at the conference – and you can watch the whole thing here. Full subscribers also get access to Laura’s slides.

New data points on AI adoption shared at The Pragmatic Summit, provided by DX:

Usage:

Time management:

Onboarding impact:

There’s no typical experience with AI, said Laura:

“AI is extremely different in every company because each place has its own problems and its own culture.

Organizational performance is multidimensional and these organizations are just going off to different extremes, based on what they were doing before. AI is an accelerator, it’s a multiplier, and it is moving organizations in different directions.

Some organizations are facing twice as many customer-facing incidents. At the same time, some companies are also experiencing 50% fewer incidents. So, for some companies that have used AI and have a really healthy system, it has amplified that system. They are seeing fewer incidents, moving faster, accelerating with higher quality, higher code maintainability, and higher change competence.

On the other side, we have organizations which were dysfunctional already and are now more dysfunctional: they’re dysfunctional faster. Similarly, organizations are seeing really uneven results economically from using AI.

There are a lot of steep drop-offs when it comes to using AI in a pilot context with production, and then trying to tie it to profit. This is from an MIT study in July 2025, The Gen AI Divide”.

Good analogies for the age of AI might be the space race of the previous century and the age of exploration in the 15th-17th centuries, said Laura.

• Wonder of space: space exploration includes immense hope and grounded scepticism.

• Defining moments: even though the space race was incredibly expensive – or utterly wasteful – landing on the moon was a big moment for humanity.

• Delivering but not on the grandest ambitions: in the 1960s, going to the moon carried the expectation of human habitation on the lunar surface. As yet, that hasn’t happened but landing on the moon created progress back on Earth:

There’s a rare opportunity to channel the energy around AI into practical improvements, said Laura in closing remarks:

“I want to urge you to find that balance between a sense of wonder, aiming for Mars, and a moon colony, but also understanding we need to solve problems here on Earth and we have to live in this reality.

So, please stay grounded, stay skeptical, stay human. Most of all, stay pragmatic”.

Watch Laura's keynote

Access Laura's slides

2. Building world-class engineering orgs in the age of AI

Also at the Pragmatic Summit, Thomas Dohmke (founder & CEO of Entire, and former CEO of GitHub) and Rajeev Rajan (CTO of Atlassian) took part in a fireside chat to explore what a modern, world-class engineering team looks like today. Thomas is building a new, AI-native startup with Entire – an “AI-native GitHub” – while Atlassian has more data than anywhere else on developer productivity after acquiring DX.

Watch the closing session

A number of timely, thought-provoking topics were covered by the pair on stage:

Topic #1: What does an AI-native team look like?

Rajeev (Atlassian):

“It starts with the mindset. You have to really believe in doing AI-native work. Some teams at Atlassian have engineers basically writing zero lines of code: it’s all agents, or orchestration of agents. As a result, teams are not necessarily getting smaller, but they’re producing a lot more, sometimes 2-5x more, and creativity is up.

Efficiency framing is missing the point, it’s more about what you can create now with AI which you could not before”.

Thomas (Entire.io):

“AI native is the new ‘cloud native’ and what we are calling AI native now is going to be very different from AI native in a few years’ time.

There’s a lot of BS out there about how all day-to-day tasks are now ‘AI native’, and using agents for everything. I’m a startup founder: most of the time, I’m still dealing with things like old school HR school systems”.

Topic #2: AI makes being distributed an advantage again

Thomas:

“A remote team is kind of lonely, right? But agents basically gave us these sparring partners, these experts that I can just ask “bring some help in for this topic, I need to write an ADR or solve a problem.

This help has taken it to a new level for remote work. I now have my code review agent, my coding agent, my brainstorming agent, my research agents. In some ways, agents have given an advantage again for being remote-first.”

Rajeev:

“Agents have changed things a lot. I remember the days when I was in the office at 2:00AM writing code and would get stuck. All you can do is read the code, then read the code some more until you figure it out.

Well, now you can ask an agent to explain it, or even better, do the code for you. So, with agents it’s gotten even better and easier for us to be able to be distributed.”

Topic #3: Why restrictive corporate IT gives startups a massive advantage

One hilarious exchange between Rajeev and Thomas was about how large companies are slow with AI due to restrictive IT:

Rajeev: “Over the holidays, I bought a laptop because sometimes you can’t install things because of IT. So, I bought my own personal laptop at the Apple store, installed Claude Code, built an app, a bunch of Python scripts, and things like that. As someone who has grown up writing code to my personal bar, it’s super high – as it is for all my engineers. Now, I can write code much faster with agents and things like that.”

Thomas: “To all startup founders: when an investor asks how you’re preventing the incumbent from doing the same thing you’re doing: just tell them the CTO of Atlassian had to buy a laptop on his own money to start coding!

That’s the best answer I can give because it shows you how even relatively agile companies [like Atlassian, a promoter of Agile] that actually develop agile tools, can’t really do what you can do in five minutes!”

Topic #4: Why CTOs are rolling out agents with top-down mandates

Thomas discussed how engineering leaders introduce agentic tools after playing with them as side projects.

“What happened in the last two years through coding agents like Copilot, Cursor, and Devin, is that many CTOs and CIOs, even in the largest banks, realized they can go back to coding. It no longer takes hours to install all the packages and figure out all the problems.

I actually have friends in these spaces who tell me stories of how every night they’d give the agent a task and say: “build me this.” Then, in the morning they would check in with the agent and instruct it to keep going. And then they have their normal day-to-day as CTO of a big bank.

You do that for two weeks and you realize everything is going to change – and that it has to change in my organization. And in contrast to the Twilio founders – in their case, they had developers decide, bottom up, what tools to use – these CTOs and CIOs say: “I don’t want to hear any excuses. We’re going to roll out agents.”

Still, many banks are on very archaic systems and have developers log into Citrix, and jump through all sorts of hoops to use agents.”

Claude Code was introduced in a similar way at Canadian fintech, Wealthsimple, as we covered two weeks ago.

The fireside chat touched on more subjects, including the changing role of software engineers and engineering leaders, and the biggest changes Thomas and Rajeev expect this year.

Watch the closing session

You can now also watch the other keynote session: How AI is reshaping the craft of building software with OpenAI and the Codex team. Full subscribers can already check out all sessions at The Pragmatic Summit, Q&A included. The other sessions without Q&As will be released next week for everyone.

The remainder of this article covers an event hosted by Martin Fowler.

3. ‘Future of Software Development Summit’ with Martin Fowler

Back in February of 2001, a group of 17 software developers gathered at a ski resort in Snowbird, Utah, where they drafted and signed the famous Agile Manifesto.

Almost exactly 25 years later, Martin Fowler and Thoughtworks organized a retreat at almost the same place in Deer Valley, Utah. He invited around 50 tech leaders from large enterprises and nimble startups, researchers, authors, and experienced, hands-on software engineers. Held earlier this month, it was “The Future of Software Development,” and I was delighted to attend the two-day gathering featuring Annie Vella, Kent Beck, Steve Yegge, Gene Kim, Laura Tacho, – and of course, Martin himself.

This was no attempt to draft a second landmark manifesto; rather, it was an opportunity to share notes and consider where the tech industry might be headed. It was an intense day of back-and-forth about what we’re seeing and what to make of it.

One declaration was drafted. Kent Beck, Laura Tacho, and Steve Yegge, came up with an apposite statement on inflated expectations upon AI to magically improve everything in workplaces:

“Organizations are constrained by human and systems-level problems. We remain skeptical of the promise of any technology to improve organizational performance without first addressing human and systems-level constraints.

We remain skeptical and we remain human”. – Kent Beck, Laura Tacho, and Steve Yegge.

Personal notes

I have my own thoughts on the topics covered:

Everyone, everywhere is adopting AI rapidly, which in some ways is very surprising. Looking back on tech innovation over the last 20 years, such as mobile, cloud, or even crypto – adoption was gradual. For example, native mobile progressed like this:

1. Indie devs: in the first few years, startups, experimental individuals, and teams onboarded to the new technology. For native mobile apps, 2008 was when iOS and Android launched their app stores. Between 2008 and 2010, it was mostly indie developers, small teams, and companies that built their apps.

2. Startups and forward-thinking companies: a few years later, more startups saw the opportunity to jump in. For example, Uber was founded in 2010 and Snap in 2011. Some forward-thinking places also launched in this period, like Starbucks with an iOS app in 2009.

3. Larger tech companies: Years on, some forward-looking, larger tech companies adopted the technology. For example, Bank of America launched an app in 2011 (3 years into native mobile), Yahoo started to go all-in on native mobile apps from late 2012.

4. “Traditional” tech companies finally catch up: Eventually, slow-moving companies also move over when it becomes a baseline expectation to use a new technology. For example, Ryanair launched its native iOS app in 2014, six years after native mobile apps emerged.

Contrast that to today, when it seems like everyone is moving all at once. I talked with folks from companies like John Deere (agriculture), 3M (industrial products), Cisco (networking hardware), Wealthsimple (finance), AWS (cloud), and startups, and none could be called “behind.” Every company is rolling out agentic AI tools; for example, Wealthsimple did a global Claude Code rollout just a week earlier.

Based on past experience, you might expect at least some traditional companies with no exposure to AI to wait and see what happens. But I found none doing so.

Traditional, “old school” companies don’t seem to be lagging behind the pace. A traditional company with 10,000+ developers has more than 50% of software development outsourced. This place founded an AI platform team with a mission to eliminate outsourcing activity via a mix of efficiencies from AI tools and targeted hiring.

They have already built an AI debugging agent popular across the whole business that’s connected to all internal systems like monitoring, logging, and internal data stores. It helps a dev pinpoint any and all errors and bugs. This platform team is building more internal tools and hiring aggressively.

It’s a similar story at other trad places: most are already using AI agents, some are experimenting with AI used for large-scale refactors, and others are seeking use cases for agent swarms.

Smaller engineering teams are a certainty. Around 20 engineering leaders – many at large enterprises, some at tiny startups – said that engineering teams are shrinking. One Head of Engineering at a 200-year-old company in the agriculture sector put it like this:

“We are already seeing the end of two-pizza teams (6-10 people) thanks to AI. Our teams are slowly but surely becoming one-pizza teams (3-4 people) across the business”.

To emphasize, this is one of the most traditional companies which sells physical goods and hardware, and whose business has no disruption to fear from AI. But they’ve also adapted!

Observations from Laura Tacho

Laura was kind enough to share her notes on this event for this article:

Read more

Personally, I’ve been warming up to Codex since doing an interview for The Pragmatic Engineer podcast with Peter Steinberger, the creator of OpenClaw, in which he revealed that he writes all of OpenClaw with Codex, preferring longer-running agentic loops. Update: on Monday, Peter announced he is joining OpenAI to work on building next-generation agents. It’s a major win for OpenAI and the Codex team, while OpenClaw remains independent and open source. Check out my podcast with Peter in his first in-depth interview, around when OpenClaw (back then: Clawd) was getting massive momentum.

To find out how Codex was built, how teams at OpenAI use it, and what effect it’s having on software engineering practices at the ChatGPT maker, I spoke with three people at OpenAI:

• Thibault Sottiaux (Tibo), head of Codex.

• Shao-Qian (SQ) Mah, researcher on the Codex team who trains the models that power it.

• Emma Tang, head of data infrastructure, who isn’t on the Codex team but whose team uses Codex heavily.

This deep dive covers:

1. How it started. From an internal experiment in late 2024, to a product used by more than a million devs.

2. Technology and architecture choices. Why Rust and open source? In-depth on how the agent loop works.

3. How Codex builds itself. Codex itself writes more than 90% of the app’s code, the team estimates. Also: interesting engineering practices like tiered code review, Codex self-testing itself, and onboarding new engineers to the team via pairing.

4. Research. Training the next Codex model with the current one has parallels with software engineering. Running evals, A/B testing and internal dogfooding.

5. Codex usage at OpenAI. Emma’s data team built an internal “data agent” in two months — that would have probably taken over a year, before. The number of PRs is so large that the traditional PR flow is starting to crack, though.

6. How software engineering is changing at OpenAI. The “30/70 rule:,” some engineers returning to tab-complete, and the importance of “taste.”

7. Next steps. GPT-5.2 step change, the capability overhang, and where Codex goes next.

Last week’s debut Pragmatic Summit, in San Francisco, featured a fireside chat with Tibo, myself, and the audience, and featured new details about how Codex is built. Paid subscribers can watch this recording now. Free subscribers will get access to all videos from the Pragmatic Summit in a couple of weeks.

Longtime readers might recall a deepdive entitled How Claude Code is built, based on interviews with the founding engineers at Claude Code. Some comparisons with today’s topic are obvious: Codex and Claude Code have each made bets that seem to be paying off. I was initially skeptical when I talked with the Codex team last October because the cloud-first, long-running task approach didn’t click with me. But I’ve now changed my mind.

1. How it started

In 2024, OpenAI was experimenting with various approaches for building a software agent. That fall, the company declared that building an aSWE (Autonomous Software Engineer) was to be a top-line goal for 2025. This vision came from the top: Greg Brockman and Sam Altman believed they should have an autonomous software engineer working alongside teams. Tibo describes the thinking:

“Greg and Sam had the strong conviction: ‘eventually, we should have an autonomous software engineer working alongside us and with the capabilities seen from o1-preview, the time is now to have a group absolutely dedicated to making this a reality’”.

A number of folks who’d worked on earlier prototypes were pulled into the effort, which featured:

• Michael Bolin: tech lead for the Codex open source repository.

• Gabriel Peal: who subsequently built the VS Code extension, mostly solo, and built the foundations of the Codex desktop app.

• Fouad Matin: who led the initial release of the Codex CLI, and is responsible for Codex’s safety and security approach.

OpenAI had two teams tackle different segments of the problem space: Codex Web would focus on an async, cloud-based solution, while Codex CLI targeted iterative, local development. Both products would launch in the spring, with Codex CLI being announced in April 2025, and Codex in ChatGPT introduced in May.

2. Technology and architecture choices

An obvious difference between Codex and Claude Code is the programming language. Claude Code is written in TypeScript, “on distribution”, which plays to the underlying model’s strengths. Meanwhile, the Codex CLI is written in Rust. Tibo explains why:

“We debated TypeScript, Go, and Rust. All three seemed like solid contenders for different time horizons. In the end, our reasoning came down to a few layers:

• Performance: We want to eventually run this agent at a massive scale where every millisecond matters. Performance is also important when running locally in a sandboxed environment.

• Correctness: We wanted to choose a language that helps eliminate a class of errors with things like strong typing and memory management.

• Engineering culture and engineering quality: There’s this interesting thing that language choice does: it gets you to think about the engineering bar you set. We decided to pick Rust because it’s extremely important for our core agent implementation to be extremely high quality”.

There was also a practical concern about dependencies. Choosing TypeScript means using the npm package manager. Using npm often means building on top of packages that may not be fully understood – which could clearly be problematic. By going with Rust, the team has very few dependencies and can thoroughly look through the few dependencies there are.

They also want to eventually run the Codex agent in all sorts of environments – not just laptops and data centers – and even places like embedded systems. Rust makes this more achievable from a performance perspective than TypeScript or Go.

Tibo tells me that while Codex’s early performance was less standout with Rust than with TypeScript, they expected the model to catch up. Plus, choosing Rust gave them one more engineering challenge to work with. The Codex team also hired the maintainer of Ratatui – the Rust library for building terminal user interfaces (TUIs). He’s now full-time on the Codex team, doing open source work.

The core agent and CLI are fully open source on GitHub.

How Codex works

The core loop is a state machine, and the agent loop is the core logic in the Codex CLI. This loop orchestrates the interaction between the user, the model, and the tools the model uses. This “agent loop” is something every AI agent uses, not just Codex, and below is how Codex implements it, at a high level:

1. Prompt assembly: the agent takes user input and prepares the prompt to pass to the model. On top of user input, the prompt includes system instructions (coding standards, rules), a list of available tools (including MCP servers), and the actual input: text, images, files, AGENTS.md contents, and local environment info.

2. Inference: the prompt is converted to tokens and fed to the model, which streams back output events: reasoning steps, tool calls, or a response.

3. Response:

   1. Stream the response to the user by showing it on the terminal.

   2. If the model decides to use a tool, make this tool call: e.g. read a file, run a bash command, write code. If a command fails, the error message goes back to the model, the model attempts to diagnose the issue, and may decide to retry.

4. Tool response (optional): if a tool was invoked, return the response to the model. Repeat steps 3 and 4 for as long as more tool calls are needed.

5. Assistant message: the “final message” intended for the user which closes one step in the loop. The loop then starts again with a new user message.

Compaction is an important technique for efficiently running agents. As conversations grow lengthy, the context window fills up. Codex uses a compaction strategy: once the conversation exceeds a certain token count, it calls a special Responses API endpoint, which generates a smaller representation of the conversation history. This smaller version replaces the old input and avoids quadratic inference costs. We covered how self-attention scales quadratically in our 2024 ChatGPT deepdive.

Safety is an important consideration because LLMs are nondeterministic. Codex runs in a sandbox environment that restricts network access and filesystem access by default. Tibo reflects on this choice:

“We take a stance with the sandboxing that hurts us in terms of general adoption. However, we do not want to promote something that could be unsafe by default. As a dev, you can always go into your configuration and disable these settings if you want.

We made this default setting because many of our users are not that technical. We don’t want to give them something that could have unintended consequences”.

There are several releases per week. Internally, the team ships a new version of Codex up to three or four times a day. Externally, new releases are cut every few days and are distributed via package managers, Homebrew, and npm.

Michael Bolin’s recent blog post, “Unrolling the Codex Agent Loop,” lays out the internals of how the agent loop works.

3. How Codex builds itself

More than ninety percent of the Codex app’s code was generated by Codex itself, the team estimates, which happens to be roughly in line with what Anthropic has reported for Claude Code, according to what its creator Boris Cherny told me. Both AI labs share the meta-circularity of using the coding tools to write their own code.

Tibo tells me that a typical engineer on the Codex team runs between four and eight parallel agents, which do any one of a number of tasks:

• Feature implementation

• Code review

• Security review

• Codebase understanding

• Going through plans and summarizing

• Going through what team members have done and summarizing changes

• Bugfixes

• … and more.

Codex engineers are now “agent managers” and no longer just write code. Tibo says it’s common for an engineer to walk into the office with several tabs open on their laptop: a code review running in one, a feature being implemented in another, a security audit in a third, and a codebase summary being generated in a different tab. He says:

“Codex is really built for multitasking. There’s this understanding that most tasks will just get done to completion.

People on our team have figured out what Codex is and isn’t capable of. There is a tricky thing in all of this, though: we have to relearn these capabilities with every model”.

Frequently-used “skills”

“Agent Skills” are ways to extend Codex with task-specific capabilities, which is pretty much the same concept as Claude Code’s skills. Internally, the Codex team built 100+ Skills to share and choose from. Three interesting examples:

• Security best-practices skill: a comprehensive write-up of all security practices adopted by the team. When invoked, Codex goes through each practice, checks the code, and generates patches for anything missing.

• “Yeet” skill: takes any code change, writes up the PR title and description based on the original plan, and creates a draft PR in one step.

• Datadog integration skill: Codex connects to Datadog, reviews alerts and issues, finds problems, and tries to generate a fix for them.

The way Tibo thinks about skills is that they help steer the model to more specific behaviors, and they can also be combined. Skills are continuously published internally and team members copy from each other.

Tiered code review

The team set up AI code review and it always runs. The team trained a bespoke model for code review, optimizing it for signal over noise. Around nine out of 10 comments point out valid issues, says Tibo, which is equal to or slightly better than human reviewers. AI reviews are run automatically whenever a pull request moves from “draft” state to “in-review” state, and are automated via a GitHub webhook.

Following an AI review, there are two possible next steps:

• Non-critical code: can be merged with no further human review. For non-critical parts of the code where speed matters more than perfection, the engineer running the agent can decide to merge the code following an AI review, when they decide it’s ready.

• Key parts of the codebase: mandatory human review. For key code parts such as the core agent and open source components, the team insists on careful human code reviews.

Other engineering practices on the team

Here are more practices that Tibo rates as helpful for the Codex team:

• AGENTS.md. Instructions stored inside a repository. On the Codex team, these files tell the agent how to navigate the codebase, which commands to run for testing, and how to follow the project’s standards. These are a bit like README files, but written for AI agents instead of humans. Agents.md has become a de facto standard across agents, and the only major agent not to use it is Claude Code. See agents.md for more details.

• Structure code for agents: have tests! The team has deliberately structured their codebase “to make it inevitable for the model to succeed”. Structuring means having tests in-place, clear module boundaries, and instructions on how the model should run validation (eg: tests, linting, etc). When the model implements something incorrectly, a test fails, the agent notices, and it tries to figure out what went wrong. Since the model is trained to be persistent, it keeps trying until it gets it right. Tasks can run for 20 to 30 minutes, or sometimes an hour.

• Codex running the test suite to test itself. In what is an interesting meta-testing capability, Codex can fully test itself, using a specific skill. This skill runs tests for all of Codex’s own features. Amusingly, Codex seems to invoke this skill more often than expected.

• Overnight, Codex runs to generate suggested fixes. The team set up nightly runs of Codex, instructed to look for issues across Codex. Every morning, engineers review issues that Codex identified with the codebase, with fixes waiting for review.

• Onboarding to the Codex team via pairing. New joiners are asked to keep an open mind about how the Codex team does development, and are warned that the “how” of building software is different than at most places. A new joiner is paired with an engineer on the team and shadows them for the first part of the day, observing how they develop with Codex. They are then assigned a task in the second half of the day, and are expected to ship it to production on the same day.

• Unlimited Codex usage. To put everything in this deepdive in context, OpenAI employees – just like at Anthropic – have no usage cap on LLMs. This makes sense given they are building a tool they want customers to use as much as possible. However, it’s in contrast to non-AI labs. Keep this in mind as we go through how the team operates.

Using Codex to debug Codex

The Codex team holds meetings to discuss Codex, during which it’s common for engineers to fire off a thread within Codex to see if it can come back with some information. This January, something interesting began happening, Tibo told me:

“When we asked questions related to why Codex runs in a certain way, Codex decided to debug its own systems. It connected to logs, SSH’d into research dev boxes, and analyzed ML instabilities.

It figured out a lot of useful information, and wrote a little report we could present on the screen. So, we ended up going through this doc in the meeting”.

This feels like another meta-circularity of Codex debugging itself – or at least systems that power it!

4. Research

Codex is built by researchers as well as software engineers, of whom one is SQ Mah. He managed to move into research from software engineering by competing in the Vesuvius Challenge, of reading millennia-old carbonized scrolls from the ancient Roman town of Pompeii which were buried by a cataclysmic volcanic eruption of Mount Vesuvius in AD 79. SQ finished second in the contest by renting GPUs on Google Cloud, training models, taking research ideas and turning them into useful algorithms.

So, what is “research” like at OpenAI? SQ’s take:

Read more

The next podcast episode with Kotlin’s creator, Andrey Breslav, will be out on Thursday instead of tomorrow (Wednesday). Regular scheduling, including The Pulse, resumes next week. We’ll release session recordings from the Pragmatic Summit to paid subscribers first, and later to everyone.

Please wish me luck with this event tomorrow, it’s the first ever in-person event by The Pragmatic Engineer, and there’s a full house of 500 attendees. If you’re there, come and say hi!

Steve Yegge has been a software engineer for over 40 years, with stints at GeoWorks, Amazon, Google, and Grab. He’s known for provocative, entertaining blog posts, of which the most famous might be Stevey’s Google platform rant.

Steve recently published the book “Vibe Coding”, has also built Gas Town, an open-source AI agent orchestrator, and created documentation site and community hub Gas Townhall around the project. A year ago, we did a podcast focusing on his career, including how he used AI for coding. Last week, Steve and I sat down again in Salt Lake City and explored what’s changed in terms of using LLMs for coding, and also the wider tech industry.

In this article, we cover:

1. LLMs, the end of coding by hand, and the S-curve. Initially skeptical of LLMs, Steve became a convert after trying out Claude Code. He now argues the industry is headed into a steep exponential curve that shows no signs of stopping.

2. “50% dial” and deep cuts. Could big companies cut up to 50% of their staff in order to make way for AI tools? Such a trend might dwarf pandemic-era layoffs.

3. Eight levels of AI adoption. A spectrum of AI adoption, from “no AI”, to “building your own orchestrator”. Steve fears engineers stuck at the lower levels will be left behind.

4. The Dracula effect: the drain of using AI. Vibe coding at full speed can be physically draining, and Steve argues that employers shouldn’t reasonably expect more than three hours of AI-augmented work from engineers, per day.

5. Big companies are doomed – but don’t know it. Innovation at large companies is effectively dead, and the real action will come from small, AI-augmented teams – like when cloud computing shifted the balance of power.

6. Redundant values in the AI era. Knowing how to code doesn’t make engineers special any more, but demand for software will keep growing.

7. Debugging and programming languages. Agents may not need good debugging tools, and language choices have never mattered less.

8. Grieving the demise of coding by hand. Steve endured a grieving process when he realized many skills are obsolete, BUT building software is now more fun than ever.

9. Predictions. Steve’s non-coder wife could be the top contributor to their video game, Wyvern.

Related The Pragmatic Engineer articles featuring Steve:

• Amazon, Google, and Vibe Coding with Steve Yegge (podcast episode in 2025)

• The full circle of developer productivity with Steve Yegge (deepdive in 2022)

Originally, I had audio quality issues with this recording. Fortunately, I managed to improve the audio and remove echos, and the episode is also available as a podcast now:

Listen to the podcast version

1. LLMs, the end of coding by hand, and the S-curve

One thing about you is that you’re pretty pragmatic. You were always into compilers, debugger tools, and worked on hard problems at Amazon and Google. When AI came out, I don’t remember you saying, “this is amazing!” What was your initial reaction to LLMs?

Steve: “I was pretty blown away that they could write fairly coherent Emacs and Lisp functions. The original ChatGPT in December 2022 could already write code in a weird language, right? Not very much of it, and it was janky; but for me, that was the beginning. I’ve had friends who were saying for 20 years, “any minute now, any day now” about AI. But 2022 was the first time that it was like, “okay, I see now.

I was still skeptical when the rumors emerged about Claude Code at the start of last year, that Anthropic had a tool internally that was writing code for them and it was a command line tool. Like many others, my reaction was “no, it’s not!” Then I used it and was like, “oh, I get it. We’re all doomed.” It was then that I wrote The Death of the Junior Developer.

So, was I a skeptic? Yes. But did I pay attention to the curves from the very beginning? Also yes. It was at that point that I thought we’re on a curve, this is a ride, and it’s not stopping, so let’s get on board and see where it goes.”

I dived in. I was feeling behind because I didn’t know AI, I didn’t know its fundamentals, or the lingo. I spent a year doing nothing but reading papers and catching up”.

Your book, ‘Vibe Coding’, says on its rear cover: “the days of coding by hand are over”. When did you decide this?

Steve: “It was a year ago [early 2025] and that wasn’t even my quote. It’s Dr. Erik Meijer, the inventor of many things in the programming world, and one of the most important compiler people in the world. He spent his life building technology for developers to be able to write code, and he’s saying developers aren’t going to write code anymore. That’s what caused Gene Kim (author of The Phoenix Project and former CTO of Tripwire) and I to both say that if someone this prolific in the tech world says ‘we’re done writing code’, then what does he see that we don’t?

He sees the curves; it’s that simple. It’s like exponential curves, they get real steep, real fast, and we’re heading into the steep part this year”.

One thing about being an engineer is that you can draw curves, but you never know when they end, or if they flatten. What makes you believe that this curve will continue? There’s also the question of how long they will scale for.

Steve: “The world is filled with unbelievers; people who think the S curve goes up and then flattens, and that we’re at the hump stage right now. They have thought that ever since GPT-3.5 came out. They’re like, ‘it’s not going to get any better’, but then GPT-4 comes out and they think that’s as good as it gets. Now Opus 4.5 is out and most people haven’t played with it, and don’t realize what’s there. But that thing is already two months old.

The half-life between models has gone from about four months at the beginning of last year to two months from Anthropic at the beginning of this year, and we’re going to see another model from them soon, which will be so much further up the curve that people will really freak out. All the bugs, all the mistakes they’re complaining about get fed right back in as training, so it doesn’t make them next time. This is what people aren’t understanding.

Also, time continues, and there’s this inevitable collision of curves: there will be societal upheaval. It’s already started and people are justifiably mad. And I’m mad too. I’m mad at Amazon for laying off 16,000 people and blaming AI without an AI strategy. Many of those people are not going to be able to find jobs, by and large, and they’re the first of many to come. Nobody has a plan for this”.

2. The “50% dial” and large cuts

Why do you think Amazon did those layoffs if they don’t have an AI strategy?

Steve: “People will hate me for saying this, and me saying it doesn’t make it true: it was true already. Every company has a dial [of the percentage of staff they can lay off] that they turn from zero to a hundred. It just has a default setting of what percentage of your engineers you need to get rid of in order to pay for the rest of them to have AI – because they’re all starting to spend their own salaries on tokens.

And so, at least for a while, if you want engineers to be as productive as possible, you’re going to have to get rid of half of them to make the other half maximally productive. And as it happens, half your engineers don’t want to prompt anyway and are ready to quit.

So what’s happening is that everybody on average is setting that dial to about 50%, and we’re going to lose around half the engineers from big companies, which is scary”.

That would be more than during the Covid-19 pandemic and afterwards.

Steve: “It’s going to be way bigger. It’s going to be awful. But at the same time, something else is happening, which is that AI is enabling non-programmers to write code. It’s also enabling engineers who have seen the light and believe the curves are going to continue to go up to actually get together in groups and start to do things that rival the output of big companies that are tripping over themselves.

So, we’ve got this mad rush of innovation coming up, bottom up. And we’ve got knowledge workers being laid off by big companies because clearly big businesses are not the right size anymore.

Even Andy Jassy [Amazon’s CEO] is saying they’re going to do the same thing with fewer people. So, does this mean we’re going to have a million times more companies? Is there going to be a massive explosion of software, or are people going to get out of software altogether and do other stuff? I’m very curious about where all this goes”.

3. Eight levels of AI adoption

Elsewhere, you’ve said something that might trigger many people, that if you use an IDE today, then you’re a bad engineer.

Steve: “Well, you’ve got to be a little provocative! I’m not going to say you’re a bad engineer because I know some very good engineers – better than me – who are still at level one or two in my chart. But I feel sorry for people who are good engineers – or who used to be – and they use Cursor, ask it questions sometimes, review its code really carefully, and then check it in. And I’m like: ‘dude, you’re going to get fired [because you are not keeping up with modern tools] and you’re one of the best engineers I know!’”

Tell us about your chart and these eight levels.

Steve: “I drew it on a board in Australia for a big group of people to show what happens, as I saw them all as being at different phases. Some had their IDEs open, some had a big, wide coding agent, and others had a really narrow coding agent. I put them all on a spectrum just to show what’s going on. Here’s the levels:

• Level 1: no AI

• Level 2: Coding agent in your IDE, permissions turned on

• Level 3: Coding agent in IDE, “YOLO mode.” Your trust is going up.

• Level 4: you’re starting to not look at the diffs anymore, but at what the agent is doing. You’re not reviewing as much, you’re letting more of it through, and you’re really focused on the conversation with the agent.

• Level 5: your approach is: “I just want the agent and I’ll look at the code in my IDE later, but I’m not coding with my IDE”.

• Level 6: several agents. You’re bored because your agent’s busy and you want to do something, so you fire up another agent, then another. And you find yourself just multiplexing between them, and you can’t “leave” [you start to get addicted to using more agents.]

• Level 7: 10+ agents, managed by hand. This is where you typically say “oh gosh, I’ve made a mess! I accidentally texted the wrong agent and didn’t realize. How do I coordinate all these agents? What if Claude Code could run Claude Code?”

• Level 8: you build your own orchestrator to coordinate more agents”.

4. The Dracula effect: the drain of using AI

One thing you’ve mentioned is that AI can really “drain” people.

Steve: “There is something happening that we need to start talking about as a community. There’s a vampiric effect with AI, where it gets you excited and you work really hard, and you’re capturing a ton of value.

I find myself napping during the day, and I’m talking to friends at startups and they’re finding themselves napping during the day. We’re starting to get tired and cranky.

Companies are set up to extract value from you, and then pay you for it. The way all companies have always been set up is that they will give you more work until you break. People have to learn the art of pushing back. Let’s say an engineer can be a hundred times more productive, who gets to capture all that value? Well, if the engineer goes to work and works eight hours a day and produces a hundred times as much, the company captures all that value, and that is not a fair exchange.

There’s another group of people who are capturing all the value for themselves. They go in and work for 10 minutes a day and get a hundred times as much done, they don’t tell anyone, and they’ve captured all the value. And that’s not really ideal either, right?

I think that the answer is that each and every one of us has to learn how to say “no” real fast, and get real good at it.

I seriously think founders and company leaders and engineering leaders at all levels, all the way down to line managers, have to be aware of this and realize that you might only get three productive hours out of a person who’s vibe coding at max speed. But they’re still a hundred times as productive as they would be without AI. So, do you let them work for three hours a day? The answer is yes, or your company’s going to break [because people get burnt out]”.

5. Big companies are doomed – but don’t know it

With non-technical people getting into building software, could there be jobs for engineers to take over maintenance? For example, with websites you don’t need an engineer to build one because there’s Wix, WordPress, etc. But there are frontend engineers at big businesses where the website actually matters, and you want all these advanced bells and whistles. Could this be a thing with AI?

Steve: “There are going to be a lot of engineers doing software engineering, and I just think we’re all going to be doing it with AI. I think it’ll be quite some time before companies are comfortable trusting their code to be written and deployed by AI without any human involvement at all. The important point that the naysayers and the skeptics are missing is that AI is not coming to replace your job. It’s not a replacement function. It’s an augmentation function. It’s here to make you better at your job. And that’s not a bad thing, actually”.

Assuming AI makes engineers and teams more efficient, why don’t we see more and better software being produced? Could it be that engineering teams are focusing on building internal tools around AI, not products?

Steve: “I’ll turn it around and ask whether what we’re seeing is innovation dying at large companies? What if we are only going to see innovation in small places? This is kind of what happened when the cloud came out.

Also, look at Meta. It feels like the biggest company in the world right now, but back in the day it was one dude. When a new enabling platform technology substrate appears, you’re going to see innovation at the fringes because of the innovator’s dilemma [when successful companies fail by ignoring innovation elsewhere]. Big companies can’t innovate. They’re all running into this problem.

Big companies may have hyper-productive engineers who are producing at a very, very high rate, but the company itself can’t absorb that work. Downstream, they’re just hitting bottlenecks and these engineers are getting shut down and they’re quitting. So, I think what’s happening is that we’re all looking at big companies going, “When are you going to give us something?” And the answer is we’re looking at big, dead companies. We just don’t know they’re dead yet”.

6. Redundant values in the AI era

What are some beliefs from your earlier career which no longer apply because of AI?

Steve: “‘Engineers are special’. There’s one. Sure, we learned how to do something by hand that computers can do now. Kind of cool, I guess”.

But what about the “engineer mindset”? It’s not just coding that we do, right?

Read more