Listen and watch now on YouTube, Apple, and Spotify. See the episode transcript at the top of this page, and timestamps for the episode at the bottom.

Brought to You by

• Antithesis – turbocharge testing of your systems by running your whole system under aggressive fault injection. There’s good reason teams like Jane Street, Fly.io, and the etcd community rely on Antithesis. Learn more.

• Buildkite – the CI platform trusted by OpenAI, Anthropic, Cursor, Meta, Uber, NVIDIA, Airbnb and many more. When CI volume becomes an architecture problem, you deserve better CI. Engineered to reliably manage whatever your coding agents throw at the build queue: today, next year, and beyond. Learn more.

• WorkOS – make your app and agents Enterprise Ready, with SSO, SCIM, RBAC, and more. Get started.

In this episode

In 2025, it was rational to be skeptical about AI, but in 2026 it’s clear that AI is changing all of the industry, and there’s less and less place for skepticism. This take is from one of my favorite voices in software reliability and observability: Charity Majors, CTO and cofounder of Honeycomb, co-author of Observability Engineering. (Note: the second edition of Observability Engineering is out, and it’s pretty much a full rewrite of the book, I recommend grabbing it if you’re building reliable systems)

In this episode, I sat down with Charity to discuss how her thinking on AI has evolved, why she believes it is becoming a foundational part of software engineering, and what that means for how teams build, review, and ship software.

We explore how AI is changing the economics of code generation, why reliability and verification are increasingly the bottlenecks, and why the rise of non-deterministic systems requires more engineering discipline. Charity shares her views on code reviews, observability, DevOps, leadership, and why both AI skeptics and enthusiasts are getting important things right.

Takeaways from the conversation with Charity

Here are 13 parts I found especially interesting, talking with Charity:

1. In March 2025, Charity told the audience at SREcon to try vibe coding, and back then, the response was grumbling. Charity’s point was that people who are skeptical of AI should still learn to use it, because you can complain better if you’ve learned it. At this time, Charity still saw AI having a bigger impact than a new programming language, but was skeptical that it would have a generational impact.​

2. Charity’s turning point in seeing AI as a generational change was in November 2025. This was due to Opus 4.5, but Charity argues that the coding harness (Claude Code) made the bigger difference. Because thanks to Claude Code, harnesses went from being more of a shell script to serious infrastructure.​

3. The impact of AI on the industry in 2025 was similar to the impact of the cloud in 2010. Looking back, Charity is comfortable saying this: in 2010, it became clear that cloud computing was certainly going mainstream and would change the infra-layer. After 2025, it’s also clear that AI will have a similar impact on the infrastructure of building software.

4. Engineers who were skeptical of AI up to 2025: they had good reason to be so. This was because we’ve seen plenty of technologies and innovations in the past that all promised to transform the software industry, but later fell short. Examples include COBOL (a technology promising that programmers would no longer be needed to create software), neural nets, no-code and low-code tools.​

5. The question engineers need to answer: what would it take for you to be fully comfortable shipping code you have not read? Charity believes it is a “when” and not an “if” that professional software engineers will ship code they never looked at – and thus do not understand – to production. Engineering is building the systems that validate this code, and allow shipping with full confidence.

6. AI could have the software industry go through the “pets” to “cattle” change that compute infra went through in the 2010s. Up to now, writing software from scratch was far more expensive than editing existing software. But now, generating hundreds of variants of a function can be done faster than how long it would take you to hand-write it once.

Charity believes that we might be at the beginning of the transition from “pets” to “cattle” that happened at the hardware infrastructure layer. Before the 2010s, configuring and repairing individual servers was commonly done. But with tools like Terraform and Kubernetes, individual servers having issues are no longer fixed up: they are re-created instead. Charity thinks the same might happen with code, sooner rather than later. When there’s an issue with the code, generate new code that solves it, and is verifyably correct.​

7. Her contrarian take: code review is overrated, and the least valuable part of what humans add to software engineering. Charity says that humans are good at conversations and deciding what to build, not reading code to check for correctness, syntax and bugs.​

8. Charity’s verdict of 20 years of DevOps: it failed. The DevOps feedback was about trying to create a feedback loop that connected people writing the code to the code running in production. She thinks that the “ops people: learn to code!” wave worked, but the “software engineers: understand your code in production” failed, to this day.

9. Non-deterministic systems require more engineering discipline versus before. With code written by AI, we’re reducing the trust in the code (because we no longer wrote it), so we need to increase trust at the other part of the development process. Specifically, at validation: with things like tests, evals, and conformance testing.​

10. Charity’s career advice for engineering directors: run towards the waves, and get AI on your resume, immediately. It’s an anxious time to work in tech, thanks to all the change, driven by AI. Charity reminds us that anxiety and excitement are physiologically almost the same, but the difference is agency. When you have no agency, you’re more likely to get anxious, and when you do, you’re more likely to get excited.

So her advice to anxious engineering directors: consider going back to IC work, where you’ll have far more agency. IC work is well-respected, getting back to it has never been easier, but the window to do so is closing. As she put it:​

“The next time you’ll have a job interview, you’ll be filtered out if you don’t have AI experience.”​

11. On AI fatigue: take back control with small acts! We talked about various types of AI fatigue: reviewing AI slop, getting tired of the AI hype, and getting worn down by “doom trolling” by AI CEOs. Charity finds small acts of taking control back in your work from AI tools help. For example, none of the Honeycomb team uses AI on Wednesdays.​

12. Charity would like to see both the “AI-pilled” and the “anti-AI” camps tell the stories better. As she put it:

“There are some really incredible things happening in software right now, for example, with rewrites and with automating away toil. Not a single person that I’ve talked to would give up using AI.

But half of the people are seeing the wins, and they’re not connecting it to the cost, which makes them think that their coworkers are just afraid of getting automated out of existence.

So that’s my beg to everyone who listens to this: tell the whole story! Talk about the costs as well. We’re all in it together.”

13. Charity’s rule on AI writing: do not send any message/email to a human that you yourself have not read in full. She also says that it would take them longer to read whatever you send than it took you to produce it: it’s probably slop!

The Pragmatic Engineer deepdives relevant for this episode

• Shipping to production

• Deepdive: How 10 tech companies choose the next generation of dev tools

• Why is Meta destroying its engineering organization?

• When AI writes almost all code, what happens to software engineering?

• Are AI agents actually slowing us down?

• Observability: the present and future, with Charity Majors

• The third golden age of software engineering – thanks to AI, with Grady Booch

Timestamps

00:00 Intro

02:56 How Parse led to Honeycomb

06:00 The limits of individual productivity metrics

09:08 How Charity’s perspective on AI has evolved

13:50 Rewriting code vs. editing code

19:20 Production as a stage of development

22:14 Code reviews

26:56 Non-deterministic systems

31:11 Sensible uses of AI

37:41 The two AI camps

44:40 Why AI works so well for building software

49:42 DevOps

55:13 Modern observability

1:00:40 Handling context overload

1:01:56 What’s new in Observability Engineering’s 2nd edition

1:07:45 What effective leadership looks like

1:10:25 Engineering management: what is changing?

1:16:31 Junior engineers

1:18:01 AI fatigue

1:21:39 Book recommendations

References

Where to find Charity Majors:

• X: https://x.com/mipsytipsy

• LinkedIn: https://www.linkedin.com/in/charity-majors

• Website:

charity.wtf

observability, tech advice, honeycomb.io, etc

By Charity Majors

Mentions during the episode:

• Observability Engineering, 2nd Edition: https://www.oreilly.com/library/view/observability-engineering-2nd/9781098179915

• Honeycomb: https://www.honeycomb.io

• Linden Lab: https://lindenlab.com

• Second Life: https://secondlife.com

• Parse: https://en.wikipedia.org/wiki/Parse,_Inc.

• Scuba: https://research.facebook.com/publications/scuba-diving-into-data-at-facebook

• Can You Really Measure Individual Developer Productivity? - Ask the EM: https://blog.pragmaticengineer.com/can-you-measure-developer-productivity

• Let’s Talk Agentic Development: Spotify x Anthropic Live: https://engineering.atspotify.com/2026/4/anthropic-agentic-development

• Questionable Advice: Can Engineering Productivity Be Measured?:

charity.wtf

Questionable Advice: Can Engineering Productivity Be Measured?

I follow you on Twitter and read your blog. I particularly enjoy this post: https://charity.wtf/2019/05/01/friday-deploy-freezes-are-exactly-like-murdering-puppies/ I’m reaching out looking for some guidance…

Read more

6 years ago · Charity Majors

• 2025 was for AI what 2010 was for cloud:

charity.wtf

2025 was for AI what 2010 was for cloud

I was at my very first job, Linden Lab, when EC2 and S3 came out in 2006. We were running Second Life out of three datacenters, where we racked and stacked all the servers ourselves. At the time, we were tangling with a slightly embarrassing data problem in that there was no real way for users to delete objects (the Trash folder was just another folder…

Read more

8 months ago · 43 likes · 2 comments · Charity Majors

• AI demands more engineering discipline. Not less:

charity.wtf

AI demands more engineering discipline. Not less

A few days back I wrote a piece called “AI enthusiasts are in a race against time, AI skeptics are in a race against entropy…

Read more

2 months ago · 202 likes · 52 comments · Charity Majors

• The Phoenix Architecture: https://aicoding.leaflet.pub

• The third golden age of software engineering – thanks to AI, with Grady Booch: https://newsletter.pragmaticengineer.com/p/the-third-golden-age-of-software

• Software architecture with Grady Booch: https://newsletter.pragmaticengineer.com/p/software-architecture-with-grady-booch

• TypeScript, C# and Turbo Pascal with Anders Hejlsberg: https://newsletter.pragmaticengineer.com/p/typescript-c-and-turbo-pascal-with

• David Poll on LinkedIn: https://www.linkedin.com/in/depoll

• Intercom: https://www.intercom.com

• AI is approving our pull requests: Here’s how we made it safe: https://www.intercom.com/blog/ai-is-approving-our-pull-requests-heres-how-we-made-it-safe

• How AI will change software engineering – with Martin Fowler: https://newsletter.pragmaticengineer.com/p/martin-fowler

• HackerRank open sourced its ATS. My resume scored 90/100. Oh wait 74. No – 88: https://news.ycombinator.com/item?id=48713832

• AI enthusiasts are in a race against time, AI skeptics are in a race against entropy:

charity.wtf

AI enthusiasts are in a race against time, AI skeptics are in a race against entropy

I recently attended a talk where one of the presenters made some pretty…astonishing claims about what they had achieved by the pure, uncut power of vibe coding. Difficult engineering problems solved, backlogs cleared. Rewrites that would have taken a year or more in the beforetimes, now whipped out in a few short weeks of prompting. Afterwards, wanderin…

Read more

2 months ago · 204 likes · 41 comments · Charity Majors

• Ep. #89, Software is the Killer App with Bryan Cantrill of 0xide Computer: https://www.honeycomb.io/resources/podcasts/ep-89-bryan-cantrill-software-is-the-killer-app

• Eric Riddoch’s post on LinkedIn: https://www.linkedin.com/posts/eric-riddoch_the-observability-engineering-book-has-share-7475807056285814785-pw4J

• Why traditional observability misses AI agent failure: https://www.dataiku.com/blog/traditional-observability-misses-ai-agent-failure

• Charity’s LinkedIn post on effective leaders: https://www.linkedin.com/posts/charity-majors_the-most-effective-leaders-are-kind-caring-share-7477160924928233472-qcLw

• Catastrophe Ethics: How to Choose Well in a World of Tough Choices: https://www.amazon.com/dp/0593471970

• More Everything Forever: AI Overlords, Space Empires, and Silicon Valley’s Crusade to Control the Fate of Humanity: https://www.amazon.com/More-Everything-Forever-Overlords-Humanity/dp/1541619595

—

Production and marketing by Pen Name.

The Pragmatic Engineer is back from our summer break. We resume with a detailed deepdive about the trading industry, and interesting engineering challenges that come when working at a company that has no external customers, but where a single, unfortunate enough software bug could wipe out the whole company.

In tech recruitment, proprietary trading companies have a particularly high bar and typically offer compensation on a par with, or even exceeding, Big Tech; right at the top of the market. That’s because for these market makers, success is all about gaining a competitive edge over rivals. Such competitive advantages today includes software that is superior to that at their competitors.

Software engineers tend to know little about trading companies – and this piece aims to change that. Trading companies build bespoke hardware stacks and have larger platform engineering teams than most workplaces. For software engineers, it’s a lucrative niche in terms of compensation, full-stack (hardware to software) work and for engineering challenges, and so we decided to go deeper in this interesting area.

In order to find out more, The Pragmatic Engineer sat down with a leading proprietary trading firm, Optiver. Headquartered in Amsterdam, they also have a large engineering presence in the US and globally. We met engineers and engineering leaders to learn in depth how engineering works in a modern trading business, with contributions from:

• Alex Itkin: CTO, Optiver US

• Pat Cooney: Head of Global Platform Engineering

• David Gross: Technology Lead, Options

Thanks to everyone at Optiver for taking part in this report which lifts the lid on how software engineering is done when even nanoseconds can count. In this article, we look into a software engineering environment that’s distinct from what you expect at most startups and Big Tech. For example:

• No external customers. Usually, companies have consumer customers (B2C), business customers (B2B), or both. But not trading houses like Optiver, where their own business is the customer. This is a different reality: there’s no external deadlines and related pressures, but personal motivation to improve is highly valued.

• Latency: “enemy number one”. Nearly every major engineering decision at Optiver is made in the interest of minimizing latency – the amount of time between a request and response. This approach is present across the software stack and in kernel-level work. It’s why Optiver manufactures its own hardware.

• Today, latency is the floor, and AI models are becoming a differentiator. Gone are the days of having lower latency than the competition allowing for arbitrage opportunities to make risk-free profits. Instead, information models are becoming a differentiator: slow models with a fast trigger sending signals to execute trades, and fast models running at the edge of the network making trade decisions realtime.

• Haunted by a bug that nearly killed a business. Among trading houses, there’s a cautionary tale of when a peer company, Knight Capital, nearly went bankrupt after a single bug in a high-frequency trading system triggered a $440M loss.

• Different incentives. The business is incentivized to move very fast, but with a high premium on caution in order to avert potential financial disasters on the market. This cautious attitude to risk in concert with chasing speed feels pretty distinct in tech.

I this deepdive, we cover:

1. Overview of trading & hedge funds. Categories of trading companies, high-frequency trading (HFT), plenty of ML & math, and AI labs poaching HFT talent

2. Engineering organization. How trading-specific roles work together, platform engineering, the “build and own” culture, and more.

3. Software tech stack. The three-layer tech stack, languages and tools, CI/CD stack and the data layer.

4. Hardware engineering, FPGAs and Silicon. Latency progression, custom FPGAs, custom hardware, AMD hardware partnership, and more.

5. Network & physical infrastructure. Physical infrastructure, dedicated fiber & wavelength leasing, optical cable, radio, data centers & co-locations, and why AI models matter more than ever before.

6. Engineering practices. Risk vs speed, knowledge-sharing culture, testing culture, monitoring & incident detection, risk management.

7. AI at Optiver. AI tooling stack, future of agentic coding, details about adoption, and how it all looks in practice.

8. Hiring, career development & culture. Engineering levels at Optiver, going from hiring mostly juniors to hiring experienced engineers today, competition during hiring, and the onboarding feedback loop.

We’re delighted to publish this report, including details never shared before. Let’s dive in!

1. Overview of trading & hedge funds

Here’s a summary of the world of ‘prop shops’; another name for firms like Optiver that invest their own funds in trading financial assets. Below are some useful mental models for understanding the sector.

How trading operates

Buy side/sell side

• Buy side: companies invest money and earn returns. Examples: hedge funds, asset managers, pension funds.

• Sell side: firms sell services or products such as advice, underwriting, research, execution, etc. These are usually investment banks and broker-dealers.

Optiver is on the “buy side”, as a prop shop.

Sources of capital

Based on whose money is being traded, there are three main capital sources:

1. Investment banks serve corporate and institutional clients by raising capital, advising on deals, and executing trades on their behalf. Examples: Goldman Sachs, JPMorgan, Morgan Stanley.

2. Hedge funds raise money from external investors and trade it on their behalf, charging management & performance fees. Examples: Citadel, Millennium, Two Sigma, Bridgewater.

3. Proprietary trading firms trade only their own capital, with no clients or external funding. Examples: Optiver, Jane Street, Jump Trading, DRW, Hudson River Trading.

Trading eras

Optiver’s CTO US Alex Itkin pictures the evolution of trading as having unfolded across four eras to date:

1. Pre-electronic (pre-1990s). Trading was done face-to-face on noisy trading floors and by phone. Prices were shared on reels of ticker tape and printed in newspapers. Investors contacted brokers to place orders.

2. First wave of electronification (early/mid 1990s). Financial markets moved onto computer screens but orders were still entered manually.

3. Automated trading (late 1990s to ~2015). Computers did the same as human traders, but faster and at scale. This was the “mechanical” automation era of building automated workflows without data-driven decision-making.

4. Quantitative trading (~2015 to present). Data-driven decision-making with machine learning models and inference compute, with human decision-making in some key areas.

Each era “weeded” the market. Some companies excelled at automated trading but never made the leap to quantitative trading. According to Itkin, competition has got tougher over time, while the number of serious players has decreased. Today, there are only a handful of really big firms, and one reason for this is cost: investment in research clusters – which serious prop shops all do – requires hundreds of millions of dollars.

Optiver at a glance

Optiver turned 40 years old in March 2026, launching in 1986 at the European Options Exchange. Today, the company has:

• ~2,200 employees

• ~950 engineers and ~1,000 traders and researchers

• 11 offices: Amsterdam (HQ), Chicago (US HQ), Austin, New York (2025), London, Sydney, Shanghai, Hong Kong, Singapore, Taipei, and Mumbai.

• 10M+ trades executed per day, across 100 exchanges

• €4.5B ($5.1B) in trading income, and €1.7B ($1.95B) profit, as per 2025 financial results

Optiver is a mix of:

• Market maker: providing liquidity on exchanges by quoting ‘buy’ and ‘sell’ prices of financial products and earning the spread between the two.

• High-frequency trader: executing automated trading strategies at very low latency

High-Frequency Trading (HFT)

High-frequency trading involves placing high volumes of orders at lightning speed in an effort to take advantage of extremely rapid market movements. In this domain, speed is the biggest advantage, and achieving it obviously involves high-performance computing. The basic trading loop is run millions of times a day. It’s made up of three steps:

1. Watch the market for new information like price changes

2. Decide what the information means and the right trade to make

3. Send a trade to the exchange before competitors do

In trading, timing is everything, and for some types of trade even nanoseconds count. Optiver’s fastest trading system operates in the realm of sub-nanosecond, where measurement noise becomes a challenge in itself. Software, hardware, and physics are all involved, along with microwave and shortwave links between data centers, and custom-manufactured chips.

We go deep into this in the “Hardware Engineering” section below.

However, in this niche, even ultra-low latency is no longer a competitive moat in itself. As competitors have squeezed performance out of their systems, focus has shifted towards fine-tuning of trading strategies. Today, Optiver invests substantially more in building better models than it does in lowering latencies. More on this in the “Network and physical infrastructure” section below.

HFT evolves faster than other industries. Profitable strategies don’t last long, opportunities are fleeting, and innovation is a constant. In this environment, a tool like AI is relatively straightforward to implement because trading houses like Optiver are well used to change in their daily business environment. More on this topic in the ‘Optiver & AI’ section.

Plenty of ML & math

There’s a big role for machine learning (ML) and mathematics in quantitative trading. A good chunk of Optiver’s business is the buying and selling of options, and the pricing of these rests on mathematical theorems like the Black-Scholes model. Traders, quants, and even software engineers building option-pricing strategies must understand the math of this problem space.

Over time, machine learning is becoming more important than math models, but it’s worth keeping in mind that trading is not purely an ML pursuit.

AI infra providers are heavily involved. NVIDIA, Groq, and Cerebras are actively courting trading firms, due to how much money they spend on GPUs. For example, see Hudson River Trading discussing Blackwell deployments at NVIDIA’s GTC conference, or Jump Trading being among the first to deploy next-gen Vera Rubin systems. HFT companies have very clear monetization paths for GPUs and spend large sums on hardware, hence why NVIDIA and other suppliers are keen to partner with them.

AI labs poach trading talent

One new trend is AI labs like Anthropic and OpenAI recruiting from prop shops, defying the assumption that AI labs mostly recruit from Big Tech. There are a few reasons why AI labs seek out talent from the trading world:

• Infra expertise. Prop shops like Optiver have spent decades operating their own data centers and deploying on-prem hardware at co-location facilities.

• Custom, high-performance hardware. Prop shops also often build their own hardware and their kernel stacks achieve very low latencies. That’s a talent AI labs seek!

• Skillsets. The highest-paying destinations for CS majors out of standout colleges are often prop shops, paying top-of-market compensation for standout talent. Outside of select colleges prop shops recruit from, however, there tends to be little awareness about these companies for new grads, or across the industry.

2. Engineering organization

Two eras of Optiver tech

Optiver’s history can be seen as two distinct ages:

1. Regional systems (“unblock yourself”: 1986-2020): internal systems and platforms were built to serve local needs, such as building support for a market. Systems built exclusively for the US, Europe, or Asia were common.

2. Global platforms (“build for the whole company”: 2020-present): Optiver recently started to build new systems to work globally across their platform. This global focus is also why the company is investing a lot more in its platform engineering arm. A globalization push started around 2023, and its momentum has been growing.

The benefit of the old “unblock yourself” approach of local teams building whatever they needed, was that it enabled them to move fast and not get held up by dependencies. But this became problematic because of fragmentation and duplication, and the downsides became more visible over time:

• Fragmentation: different teams use different technologies, frameworks, and infrastructure

• Duplication: teams in different parts of the business independently build the same or very similar services

The career trajectory of Pat Cooney, Optiver’s head of platform engineering, mirrors the shift to a global platform: he was the CTO of Optiver in Europe in the mid-2010s when the business was split by region, and was appointed head of platform engineering in 2025 when that approach was replaced.

Optiver’s approach to continuous integration (CI) has also evolved. Previously, the company had several regional CI services, but from 2025, it started to rebuild its CI system with two new goals:

• Build for scale: create a CI system built to scale across regions and stand the test of time

• Use from any region: standardize deployment pipelines, so that code built in one location can run anywhere without friction

How roles work together

At Optiver, there are three main areas for tech roles:

• Engineering: build and own the full trading-platform stack

• Research: quantitative scientists who build models and predictive signals to create and improve trading algorithms. Typically, their background is in math, physics, economics, and statistics

• Trading: quantitative traders who watch live markets, adjust trading system parameters in response to conditions, and build tools to automate decisions

In reality, the boundaries between these areas are porous. Yes, people do the job they were hired for, but it’s common to also see researchers roll up their sleeves and take part in implementing a trading strategy, or software engineers conducting research.

Cross-functional collaboration between roles is very common. For example, when developing market signals and associated trading strategies, it’s normal for engineers, researchers, and traders to collaborate on most, if not all, projects.

End-to-end ownership, plus autonomy, is a given. Engineers have autonomy in how they get things done, and they own and solve problems from the ideas stage through to implementation. There is a limited amount of guidance for trading, and it’s down to engineers to find the right solution.

In many ways, this approach to software engineering is pretty similar to startups’: software engineers get limited guidance and lots of autonomy. In order to succeed at tech startups, engineers typically need to understand the business, as well as being excellent at building production-ready software. It’s the same at Optiver, where understanding the business means understanding markets.

Platform engineering

Before Optiver’s globalized platform efforts started seriously in ~2023, regions duplicated effort:

• Multiple implementations of identical core logic

• Each region had its own systems, frameworks, and infrastructure

• Local teams built whatever they needed in an “unblock yourself” culture

But that’s all changed. An obvious sign of global platform efforts is the appointment of Optiver’s first global CTO, Lance Braunstein, who joined with a mandate to scale the platform.

Roughly 30-40% of Optiver’s 950 engineers work on the platform. In contrast, a more typical ratio at other large tech companies is for 15-20% of engineers to be dedicated to platform work.

Prior to the global platform, there was a lot more tolerance of development experience friction; new engineers could spend weeks checking out the codebase and getting their build system to work. This mindset has changed, with the platform team stressing user empathy and reducing friction on engineers’ journeys, like by setting up build pipelines for their software.

Now, the platform is beginning to reimagine itself as built for AI. As agents proliferate at Optiver, users are both humans and automated systems. The goal of this shift is to empower people to decompose work into workstreams and orchestrate agents. Two projects were launched earlier this year by the platform team for agentic work:

• AI gateway: gives Optiver engineers access to models

• MCP hosting platform: makes it easy for engineers to access internal systems and tools via agents

How trading teams are organized

Trading teams at Optiver have three roles:

1. Traders decide strategy and make risk decisions

2. Researchers and quantitative analysts (“quants”) build hypotheses, pricing models, and run evaluations

3. Engineers build production systems

In practice, these roles overlap. This was true before the AI era, but it seems to be accelerating with AI adoption. Most traders and quants have STEM backgrounds without recent production coding experience. AI enables quantitatively-minded people to automate workflows with agents and to implement strategies.

Trading teams are organized by asset class and strategy. For example (asset classes in italic):

• A large team is focused on a broad area like options

• A team focused on cash markets and building strategies for exchange-traded funds (ETF) and stocks.

• A team focused on machine learning (ML) and trading in the cash market.

Within larger teams, there are horizontal and vertical sub-teams. Horizontal teams take on challenges that impact any trading desk; for example, pricing is a horizontal team as the underlying mechanism is the same whether a soybean or an index fund being priced.

Vertical teams are similar to “tiger teams”, accelerators, and program teams at other companies. They focus on short-term goals attached to a few different desks in a location like the US, Amsterdam, Mumbai or Sydney.

Each team has a trading or research lead and a tech lead, who identify work for the team to do. The overall direction is set by a partnership structure, similar to an investment bank, but partners are not necessarily in charge of teams. At Optiver, partners are collections of senior people responsible for overall strategy.

Regardless of asset class or vertical, every trading team builds a version of a trading loop with four components.

1. Retrieval of market-related information

2. Collecting signals to work out which trades to execute

3. Execution of strategies (sending orders to market)

4. Intervention via a feedback loop, enabling a trader to monitor the system.

“Build and own” culture

Optiver runs on an ownership culture, with the principle that the best engineers take work personally and care deeply about Optiver’s systems, decisions, and outcomes. Leaders want engineers to treat their projects as if they were CEOs of a company, and be responsible for design, build, rollout, shipping, or support. There is no notion of throwing work over the wall to a QA team.

Optiver’s ownership model:

1. Traders and engineers define problems together. Engineers design, build, test, deploy, and monitor a solution. There are hundreds of production changes daily

2. Design reviews for architectural decision-making. When an engineer has a project that entails architectural change to the stack, the engineer is responsible for bringing multiple options with the pros and cons to the team for consultation. The goal is to share information and knowledge, and to make decisions

Optiver pushes new hires and interns to develop ownership. From day one, engineers have something they own and are assigned a real project with mentoring support. Production code changes are an expectation for new hires. Within a year, a new hire becomes the experienced person in their domain, ramping up the next engineer. This is explicitly emphasized in Optiver’s onboarding materials:

Ownership is also baked into the interview process, with explicit questions about problem-solving, talking through trade-offs, and implementation.

Case study: the Options Org

Optiver started life with options trading. The word ‘Optiver’ is actually a Dutch portmanteau of “options” and “trader”, so it’s unsurprising that the options team is among the most developed parts of the operation, with engineers split across multiple locations. The organization is composed of both vertical and horizontal teams.

One of the technical systems for which the Options organization is responsible is the retreat system. When Optiver trades an option, that trade itself changes the price of the next quote on offer. The retreat system has to reprice the entire option surface (i.e., all options related to the one just traded). This is called a ‘retreat’.

In the case of S&P options, the option surface can consist of thousands of options that have to be updated. Ten years ago, the retreat process took seconds; now, through optimizations at every level of the stack, it’s down to nanoseconds.

Retreat speed matters because everything changes as soon as a trade occurs: the original quote is stale and a trader needs to remove the bid from the exchange before anyone can exploit it. Faster firms can take advantage of others’ stale prices, leading to an adversarial market dynamic.

Horizontal vs vertical team structures

Vertical teams work on specific tactical problems related to local trading desks with a focus on immediate impact. But they are not short-term or temporary teams, even if they work on short-term problems. They’re empowered to solve the most important current problems, end-to-end. On the other hand, horizontal teams serve most desks, and have longer time horizons because they work on cross-cutting problems like pricing, market connectivity, or auto-trading.

3. Software tech stack

Basic trading loop & three-layer tech stack

Most trading software applications or services (aka “apps”) at Optiver can be simplified to the basic trading loop. The exchange where the trading takes place is part of the outside world from which signals are extracted:

Signals

This is the information-gathering phase where services collect market data such as prices and order book information, and also run various data calculations, such as pricing algorithms and machine learning pipelines. These signals are made available to strategy applications/services which decide how to trade.

Strategy

A single trading strategy typically focuses on a particular class of assets and trades, and many different strategies run concurrently. The strategy sets what and how to trade, but doesn’t execute the trade; that’s the next step.

All strategies are enveloped by a risk management system that can block trades and stop individual strategies. To be effective, it has a broader view of the combined risk level of multiple strategies.

Risk mechanisms can include human oversight, with traders tweaking strategy parameters, and also automated monitoring that checks if apps are outputting orders within expected parameters, regardless of what the algorithm wants. The latter approach is essential in low latency strategies where faster-than-human reaction speeds are needed.

Execution

The execution step involves executing trades on exchanges. There’s a hard ‘separation of concerns’ principle where execution steps are only permitted to execute the trade. No additional logic is meant to run there.

Ultra low-latency loop

In some market-making use cases where nanosecond-level latencies matter, much of this process may run within a single chip (FPGA or ASIC) where the strategy part can be memoized with precomputed responses for all expected input patterns. This is then burned into the hardware to minimize latency from when market information arrives until a trading order is issued.

The tech stack’s three layers

All apps implementing the trading loop sit on top of a multi-layered internal platform:

• Basic infrastructure layer: the stuff you’d see at most tech companies (CI/CD pipelines, k8s, Kafka, Postgres, etc), but they’re also customizing their stack. They run their own data centers, have custom hardware, custom Linux kernels, customized CI tooling, and databases.

• Domain-specific infrastructure contains core trading-specific services such as trading data dictionaries, metadata on securities, and the trade booking system.

Historically, most of this infrastructure was duplicated at each local office level when teams prioritized moving fast and independently over avoiding duplication. A centralized platform team has started consolidating these efforts in recent years.

Roughly 30-40% of the engineering headcount is allocated to the Platform team. This level of investment in the platform is beyond what you’d typically see in a tech company. That’s likely to remain the case for a while longer as they focus on improving the development experience, consolidating duplicated functionality, and catering to the specifics of their tech stack.

Languages and tools

At a glance:

Language choices at Optiver are fairly standard for a financial institution: C++ for low latency applications, and Python for modeling, prototyping and internal tooling work.

However, looking closely at Optiver’s contributions to the Python ecosystem reveals that this language is not just a prototyping tool:

• optiver-asyncpg: Optiver’s fork of a performance-focused async Python lib for Postgres

• vulcan-py: Optiver’s own dependency manager for Python allows more granular control over indirect dependencies

• opti-napalm: Optiver’s fork of a library for automating and simulating various network equipment

Optiver’s internal tooling also has strict performance requirements because traders use internal dashboards and tools to make time-sensitive trading decisions. Avoiding hand-offs between traders and engineers for reimplementation in C++ saves time, and empowers non-engineers to solve their problems directly, in line with the “unblock yourself” ethos.

Rust is starting to play a significant role in research tooling and service orchestration, likely driven by the performance requirements. It’s interesting to see Rust used in areas such as Python, as opposed to it replacing C++, which would be obvious given its focus on performance. It’s likely due to Optiver’s decades’ worth of investment in the low-latency C++ ecosystem, its deep integration with existing internal hardware, and being able to directly control things like memory allocation with C++.

Other languages used in some niche use cases include:

• C# for building data-intensive trader-facing GUIs,

• VHDL and SystemVerilog for FPGA development.

CI/CD stack

Much of the software that Optiver builds interacts with custom hardware, custom Linux kernels, and requires predictable compute performance for predictable results in performance tests. These are all constraints that the CI/CD stack has to operate within.

Optiver’s CI/CD runs on bare metal machines, with custom hardware installed, the right OS tweaks, and a well-understood performance profile. Interestingly, this means Optiver needs to plan capacity in advance for its CI/CD clusters in the same way as it plans capacity for production systems. This is tricky since AI-coding tools started boosting the number of builds an average engineer does in a day.

They chose GitHub Actions as their CI Platform for the seamless development experience with GitHub. Unfortunately, Actions doesn’t provide overall, system-level metrics like queue times and utilizations, which are critical information for planning CI cluster capacity. Therefore, they had to build a bespoke observability layer over GitHub Actions pipelines with GitHub webhooks.

Data

When it comes to databases and storage systems in general, Optiver is a big user of Kafka, Postgres, and Databricks (the company built its entire data platform around this).

A few interesting details show the role of Postgres:

• They contributed a new timestamp type to Postgres, allowing timestamps to be expressed with nanosecond precision. Few Postgres applications care about nanosecond-level precision, and this wasn’t available “out of the box”.

• They built their own internal version of the NOTIFY - LISTEN mechanism called ‘PG Feed,’ based on Postgres’ write-ahead log. This is used for distributing high-fanout, latency-sensitive messages to clients like pricing and configuration data, whereas using something like Kafka may involve additional disk reads and writes, which imply unwanted latency.

Optiver generally picks industry-standard tooling, but heavily tweaks it to fit their specific performance needs. Not many tech companies of this size tweak Postgres or GitHub Actions, let alone Linux kernels!

4. Hardware engineering, FPGAs and Silicon

Read more

Listen and watch now on YouTube, Apple and Spotify. See the episode transcript at the top of this page, and timestamps for the episode at the bottom.

Brought to You by

• Antithesis — Turbocharge testing of your systems by running your whole system under aggressive fault injection. There’s good reason teams like Jane Street, Fly.io, and the etcd community rely on Antithesis. Learn more.

• WorkOS – make your app and agents Enterprise Ready, with SSO, SCIM, RBAC, and more. Get started.

• turbopuffer – A vector and full-text search engine built on object storage. It’s fast, cheap, and extremely scalable. I met their team in San Francisco, and am a fan of their “hardcore and whimsical” engineering culture, and how pragmatic their engineering philosophy is. Check them out.

In this episode

There’s a popular theory that AI will finally make formal verification mainstream because mathematical proof of correctness will be needed when machines write most or all of the code. But will this happen? Today, I’m talking with one of the best people to tackle the prediction. Hillel Wayne is a formal methods consultant, educator, and author (his most recent book being Logic for Programmers), who’s deeply interested in software history.

In this episode of The Pragmatic Engineer podcast, I sit down with Hillel to compare software engineering with traditional engineering, discuss where formal methods fit into modern software development, and we explore why they are essential for some of the world’s most complex systems. We cover the formal specification language, TLA+, walk through several formal verification tools, examine why distributed systems are so difficult to reason about, and look into whether AI will make formal methods accessible to more engineering teams.

Takeaways from the conversation with Hillel

1. Are we “real” engineers? After thorough research, Hillel has an answer. For The Crossover Project, Hillel interviewed ~20 people in different fields of traditional engineering and software engineering, and found plenty of similarities and differences. He concluded that the rigor needed in software engineering means we earn the right to the title of “engineer.”

2. Version control is unique to software engineering. Other fields of engineering have change management, but “traditional” engineers wish the concept of version control in software engineering existed in their fields because it’s far more sophisticated.

3. TLA+ is a formal specification language created by Leslie Lamport for designing and verifying systems. Lamport is a mathematician and creator of LaTeX, who wanted to create a language for modeling complex systems. The language represents the state machine of the system and every possible state it can transition to. From the initial state, the system enumerates to get to every reachable state and checks whether properties defined upfront apply to those states. In this episode, Hillel walks us through a demo with TLA+.

4. Amazon used TLA+ to find a bug almost impossible to locate without formal methods. In the paper How AWS uses formal methods, the AWS team shared that they’d found a complicated bug for which the shortest error trace to exhibit was 35 steps (!!). The bug passed unnoticed through extensive design review, code reviews, and testing. AWS concluded they wouldn’t have uncovered it if they’d stuck to conventional testing approaches.

5. Lack of practice makes most engineers bad at dealing with concurrency problems and race conditions. When a system has a race condition due to your code, you usually don’t find out until a few months later – if ever! In contrast, a system modeled in TLA+ can tell you about race conditions as soon as the tool is run, making it a fast feedback loop.

6. Why not use formal verification for everything, then? It’s because specs in the real world are a nightmare to write. Even a simple problem like “find the file in a directory that has the most lines” gets complicated when modeled with formal methods. We would have to answer questions like: ‘do we look at ASCII or UTF-8 new line characters, what about unreadable files, and Symlinks?’ Without formal methods, we can write a simple verification that is right in 99%+ of cases. Formal methods require a lot of extra effort for the less than 1% of exotic use cases!

7. Hillel recommends most engineers adopt property-based testing, and stop there. Property-based tests mean defining properties which the test then throws thousands of inputs at, in order to stress test a system. Hillel is convinced that formal methods are a niche tool for most engineers, whereas property-based testing is the most practical approach for building robust software with this lightweight formal method.

8. AI won’t make formal verification mainstream, but will increase its use. As Hillel says, “AI bringing formal verification up from maybe 0.1% to 0.3% across the industry would still be huge!” He also finds that people who succeed at using AI to generate formal specs are often formal verification experts.

9. Hillel worries about the time-of-check vs time-of-use bug. It makes Hillel want to pull his hair out when he sees a time gap between the time of checking something (e.g., whether a bank account contains sufficient funds for withdrawals), and the action itself (e.g., withdrawing money). This category of bug is hard to defend against and can cause annoying issues in real-world systems.

10. Hillel worries less about job losses from AI and more about software becoming an “ordinary” job. Revisiting his 2025 predictions of the impact of AI on the tech industry, one of Hillel’s concerns is that software engineering in the future will be lower-paid and lower-prestige than today. At present, the range of software careers available is pretty magical, especially compared to “traditional” engineering roles. But will this last?

11. One of Hillel’s coolest projects: verifying train transponders. Beyond databases and distributed systems, he has also formally verified device firmware. One cool project was working on the electric beacons between rail tracks that pass traffic information to the control system. He found a really odd bug in one transponder system, and fixing it made the real-world system more reliable and safe.

12. One thing that software engineering could take from “traditional” engineering: books on “the fundamentals” which every professional in the field should know. One of Hillel’s favorite books is The First Snap-Fit Handbook, a nearly 500-page tome on those little clips that hold battery covers in place. He observes that while most industries have copious documentation for the most mundane topics, within software engineering there’s not even a book on how to version an API! We could learn from other fields about the value of documenting our own craft.

13: The “materials” in software engineering are freakishly consistent. All other engineering professions have to worry about the consistency of their materials; for example, electrical engineers work with resistors that offer resistance within 20% of 100 ohms across a thousand units, and only when operated within a given temperature range. In contrast, a program runs identically on any given computer in software engineering. Hillel argues that the variability we deal with in software, like versions, APIs, bugs with integrations, etc, are largely battles of our own making.

The Pragmatic Engineer deepdives relevant for this episode

• How to debug large, distributed systems: Antithesis

• How AWS S3 is built

• Paying down tech debt

• How Big Tech does quality assurance (QA)

• Bug management that works

• Resiliency in distributed systems

Timestamps

00:00 Intro

04:32 The Crossover Project

11:37 What software engineering does better

15:30 What traditional engineering does better

18:17 Formal methods

29:32 TLA+: what it is and demo

36:58 TLA+ at Amazon

38:10 Ways distributed systems break

41:03 Formal methods and systems thinking

46:20 The value of learning math

50:23 What TLA+ is good for and isn’t

52:50 Alloy: a declarative language for software modeling

58:53 Other formal methods tools

1:01:24 Property-based testing

1:05:31 AI and the need for formal verification

1:12:29 Logic for programmers

1:14:35 Hillel’s 2025 prediction on AI’s impact

1:21:30 Book recommendation

References

Where to find Hillel Wayne:

• LinkedIn: linkedin.com/in/hillel-wayne

• Newsletter: https://buttondown.com/hillelwayne

• Website: https://www.hillelwayne.com

Mentions during the episode:

• The Crossover Project: https://www.hillelwayne.com/tags/crossover-project

• Blog Series: Real Software Engineering: https://vanderburg.org/blog/series/real-software-engineering

• Software Art Thou: Glenn Vanderburg — Real Software Engineering:

• New Austrian tunneling method: https://en.wikipedia.org/wiki/New_Austrian_tunneling_method

• The Design of Everyday Things: https://www.amazon.com/dp/0465050654

• The First Snap-Fit Handbook: Creating Attachments for Plastics Parts: https://www.amazon.com/dp/1569902798

• NuSMV: https://nusmv.fbk.eu/

• TLA+: https://github.com/tlaplus

• Use of Formal Methods at Amazon Web Services: https://lamport.azurewebsites.net/tla/formal-methods-amazon.pdf

• Common Sense Computing: From the Society of Mind to Digital Intuition and beyond: https://link.springer.com/chapter/10.1007/978-3-642-04391-8_33

• Alloy: https://alloytools.org

• Time-of-check to time-of-use: https://en.wikipedia.org/wiki/Time-of-check_to_time-of-use

• P: Formal Modeling and Analysis of Distributed Systems: https://github.com/p-org/P

• Quint: https://quint.sh

• PRISM: https://www.prismmodelchecker.org

• NuSMV: a new symbolic model checker: https://nusmv.fbk.eu

• I formally modeled Dreidel for no good reason: https://buttondown.com/hillelwayne/archive/i-formally-modeled-dreidel-for-no-good-reason

• Formally modeling dreidel, the sequel: https://buttondown.com/hillelwayne/archive/formally-modeling-dreidel-the-sequel

• Event-B: https://eventb-soton.github.io/en-us

• MCRL2: https://mcrl2.org/web/index.html

• KeYmaera X: https://keymaerax.org

• Dafny: https://dafny.org

• JML: https://www.openjml.org

• Frama-C: https://frama-c.com

• Ada SPARK: https://www.adacore.com/languages/spark

• The Coming AI Revolution in Distributed Systems: https://zfhuang99.github.io/github%20copilot/formal%20verification/tla+/2025/05/24/ai-revolution-in-distributed-systems.html

• CRAQ.tla: TLA+ specification of CRAQ (lamport-agent): https://github.com/zfhuang99/lamport-agent/blob/main/spec/CRAQ/CRAQ.tla

• My EuroSys 2026 paper is obsolete: https://claudiacauli.com/2026/03/08/my-eurosys-2026-paper-is-obsolete

• Situated Software — Clay Shirky (2004): https://gwern.net/doc/technology/2004-03-30-shirky-situatedsoftware.html

• Lamport Agent - AI-assisted Formal Specification: https://zfhuang99.github.io/github%20copilot/formal%20verification/tla+/2025/11/14/lamport-agent.html

• LLMs are bad at vibing specifications: https://buttondown.com/hillelwayne/archive/llms-are-bad-at-vibing-specifications

• Logic for Programmers: https://logicforprogrammers.com

• Engineering a Safer World: Systems Thinking Applied to Safety: https://www.amazon.com/dp/0262533693

• The following can all be true: https://www.linkedin.com/posts/hillel-wayne_the-following-can-all-be-true-1-vibe-coders-share-7341040573711073281-3V8C

• The third golden age of software engineering – thanks to AI, with Grady Booch: https://newsletter.pragmaticengineer.com/p/the-third-golden-age-of-software

• Data and Reality: A Timeless Perspective on Perceiving and Managing Information in Our Imprecise World: https://www.amazon.com/dp/1935504215

• Debugging: The 9 Indispensable Rules for Finding Even the Most Elusive Software and Hardware Problems: https://www.amazon.com/Debugging-Indispensable-Software-Hardware-Problems/dp/0814474578

—

Production and marketing by Pen Name.

So, I visited the two leading AI labs to see how teams and engineers do things day to day. In this article and in an upcoming follow-up, I’ll share what I learned about how AI is reshaping software engineering principles many of us are accustomed to – and what’s stayed mostly the same despite the AI wave.

In a later article, we’ll compare findings from Anthropic and OpenAI to see what their ways of working might mean for the overall direction of software engineering.

Thanks to Anthropic for showing me inside their lab in San Francisco. I talked with four people:

• Katelyn Lesse, Head of Engineering for Claude Platform, whose organization owns the infrastructure that Claude runs on

• Jarred Sumner, creator of Bun, now at Anthropic on Bun and Claude Code

• Thariq Shihipar, who works across Claude Code engineering and education

• David Hershey, at Anthropic’s Applied AI organization in a role resembling a sales engineer, working with customers like Cursor, Cognition, and Perplexity

Thanks to them, I got a sense of where things are headed at the leading AI lab – and possibly for the wider industry.

Before we continue, The Pragmatic Engineer will be on summer break for the next week and a half. This means no Thursday article this week, and no articles next week. I appreciate your understanding and support!

Back to today’s deepdive, we cover:

1. Complex & long: Claude Managed Agents. One of the most complicated projects took the Claude Platform team six months to ship, and created a new primitive to use at the agent infra level. Infra projects still need re-architecting mid-way through and take time to get right.

2. Twelve-month project done in 11 days: Bun rewrite to Rust. Migrating a 500K+ line project to another language used to take a small team a year, making it impractical. With Fable and $165K of tokens, it recently took the creator of the project less than two weeks.

3. Changing engineering practices. Inside the AI lab with more than 3,500 employees, prototyping is more fluid, verification is more time-consuming than implementation, code review and testing are increasingly done by AI.

4. Team-level changes. Design is more ongoing and less upfront, teams work on more projects, a maximum of two engineers per project, and more.

5. Still the same: two-pizza teams, planning is important, PRDs are relevant in complex projects, context switching is a challenge, the ratio of time spent on coding vs testing not changing that much.

6. Changing the “standout” software engineer archetype? Deep understanding, including of a layer below what you work on, is valuable, along with the ability to coordinate work.

7. Will AI replace software engineering? The more hands-on software engineers get with AI at the lab, the less they fear their jobs are going away.

1. Complex & long: Claude Managed Agents

The Claude Platform team’s most complex project in the past year was building Claude Managed Agents, a pre-built harness for production agents that runs in the cloud on infrastructure managed by Anthropic, or on your team’s own infrastructure, with any sandbox you choose. The project took around six months from idea until launch in April. Katelyn Lesse, head of engineering for Claude Platform, shared the story.

Claude Platform

This team sits between the model/accelerator layer (Claude models operate on GPUs) and the product/application layer (with products like Claude Code and Claude Cowork):

Katelyn on what the Platform team does:

“We’re on the ‘token hot path.’ The prompt comes in, then we tokenize it. Then, things like safeguards and billing all happen within our layer.”

What the Claude team calls “Platform,” I think more of as “API.” Claude Platform operates the API, and owns responsibilities an API would have. Of course, the team does more than that, and Claude Managed Agents is one case we cover here.

The platform layer is being migrated from Python to Rust. Originally, this layer was written for Python for the “usual” reasons at AI companies: it’s a convenient language and AI researchers use Python already, which enables quick iteration. But Python is single-threaded, and at scale, when the API is under high load, it’s not as performant as Rust.

Harness infrastructure demand

The project came together due to customers wanting their own “harness infrastructure”, says Katelyn:

“We started with a model where you get an API to define an agent, then you get an API to start a session with an agent. The reality of what the world wants and needs right now is people running their own infrastructure. So, we started to build a self-hosted sandbox.

But then, what we started to hear from lots of customers is that they’re trying to hack harnesses together, running their own “harness infrastructure,” and this gave us the idea for Claude Managed Agents.”

The largest part: planning

In this project, Katelyn said the single biggest matter was planning:

“There are products you can jump straight to prototyping, but then there are ones where you need to start by architecting it properly. For example, if we build a TypeScript CLI – which is pretty trivial for what needs to be built – we could go straight to prototyping. But with Claude Managed Agents, we needed to first figure out what we are doing.

Of course, we did some upfront prototyping for Managed Agents: hacking and spiking things. But prototyping itself was more about understanding the requirements.

Our planning process looked more like a typical pre-AI planning process. You know how every team has the project, where everyone comes up with some version of the same idea and people keep floating and circling it around until you finally do it? Managed Agents was this for our team. When we started the project, we had documents dating back up to two years about ideas and suggestions.

Post-planning, when the project officially kicked off, a PRD (product requirements document) was created:

“In the end, it was the Product Manager and the Tech Lead on our API Agents team who decided to pull the trigger and kick off this project. We’d get in a room, go through it, and get aligned. But it wasn’t just us: we’d have to align with teams around the business, other cloud providers, and other engineering teams. For example, we have a sandboxing team inside of the Platform org: and so this team was consulted on the design of Managed Agents, given this product would spawn a lot of sandboxes.

Just like before, we had a PRD, it was a Google Doc. We used a Google Doc because we needed to coordinate all interested people. This has not gone away.

Similarly, my product counterpart and I run product reviews.”

Some processes from before AI, like the PRD, are still useful in complex projects today, for getting large groups of people on the same page.

Build for an internal customer first

With planning complete, the team decided to do a “spike” and stress-test the idea and architecture, by building the backend of Claude Code on the web. Remote execution of code with an agent harness was a similarly shaped problem to the one they wanted to solve for customers. The thinking was to start by solving it for the Claude Code team before tackling it in a more generic way for customers.

Internal teams are more fluid than before AI. Katelyn:

“Pre-AI, we might have hit the Claude Code team up with a bunch of big requirements documents, and they would have then hit us back with another set of documents. Now it was much easier: someone on our team built a few components, took it over to the Claude Code team, and they started to hack around it. We could figure out how this component plugs into this part of their product, and the other way around. It was just a faster and easier process, getting this first internal version of the product up and running.

Aligning with other teams on interfaces remains important, and it’s easier. Back in the day, you’d have to come with a fully spec’d interface to use. Now, we could do it a lot more fluidly: we could stand up a stub service that shadowed traffic to start with, and iron out the interfaces with the Claude Code team as we went. They did some hacking on it and gave feedback, we made changes while building out the service under the interface, then went back to make it work.”

They launched a service for Claude Code’s mobile app to spin up a sandbox, boot up Claude Code and run it. The service went to production and the Claude Platform team took the learnings.

Re-architecting midway through

It’s likely a familiar scenario many engineers can relate to, that after planning a project and getting underway, you see that you’re going to need to change the architecture. It happened on this project, too.

The platform team ended up re-architecting Managed Agents based on learnings from the Claude Code “spike.” Re-architecting meant decoupling the “brain” of Claude and its harness from the “hands” (sandboxes & tools that perform actions) and the “session” (the log of events). Each became an interface that made few assumptions about each other.

The team also built an abstraction around vaults and credentials. Credentials can safely be stored inside a vault. All calls using credentials are made via a proxy which has a session token. It is the proxy that fetches the right credentials from the vault: the credentials are never seen by the agent, sandbox, or session. Credentials are only injected at the egress boundary when the service is invoked:

Internal “dogfooding” helped surface hard problems to solve. A few examples:

• Reliability and scalability: these are really hard to do well for agents because if connection to the sandbox is lost, the whole agent dies and you lose state

• Credentials and access control: also hard and problematic, especially when first building the service

The Managed Agents team shared more about this re-architecting project.

The project took about six months, by no means a rapid process. Katelyn emphasized that pre-AI, a project like this would have probably been in the realm of two years. Managed Agents is one of the biggest projects the Claude Platform team has built, and more complex than it looks: for example, adding support for running agents on AWS, GCP and Azure.

2. Twelve-month project done in 11 days: Bun rewrite to Rust

As covered before, Jarred Sumner is the creator of Bun, a popular JavaScript runtime with 22 million monthly downloads currently and Claude Code as a dependency.

Bun is written in Zig, a performant, productive language. However, it’s not memory safe and memory issues kept coming up. Jarred thought that rewriting the project to an also-performant, memory-safe language like Rust could be an option – except that rewrites like this turned out as follies in the past. Jarred (emphasis mine:)

“Historically, rewrites are a terrible idea. Excluding comments, Bun is 535,496 lines of Zig. A rewrite in another language would take a small team of engineers a full year. It would mean freezing bugfixes, security fixes or feature development for that time. The least risky approach to getting something shippable would be a mechanical port from Zig to Rust, with the minimal number of behavioral changes, using the exact same test suite we already use for testing Bun.

Fortunately, Bun’s own test suite is written in TypeScript which means it doesn’t depend on the runtime’s programming language.

A year of zero user-facing impact was not an option we could consider. So, enforcement through code style to fix stability issues was our best bet, and was our plan when we added Rust-inspired smart pointers to Bun’s codebase.

But honestly, I didn’t want to do it. Homegrown smart pointers offer worse ergonomics than Rust, with none of the guarantees.”

But then, Jarred asked if AI could do the heavy lifting and wondered how much the migration could be sped up. In the end, he completed the rewrite from start to merge in 11 days, using 64 parallel agents and $165,000 in tokens at API price. Here’s Jarred on how his AI-heavy rewrite compared:

“By hand, I think this would’ve taken three engineers with full context on the codebase about a year, during which time we wouldn’t be able to improve Node.js compatibility, fix bugs, fix security issues or implement new features. We never would’ve done that. The realistic alternative was to do nothing and keep fixing the bugs at the top of this post forever.”

There was a lot more to the project than typing out the “...make zero mistakes” prompt:

• Jarred made a detailed plan and style guide on how to migrate

• He set up the project so agents would not use Git worktrees which he found slow, but worked on different files in the same codebase

• He created an orchestration system where each AI agent came up with suggestions of what to change, but did not make a change to the file to avoid conflicts; an orchestrator AI agent created the commits

• The most time and tokens went on fixing the compile bugs, tests, and verifying that things worked

• Bun itself has a very robust test harness: when all tests pass, it’s a high-confidence signal that the rewrite works

• Crucially, Jarred is the ultimate domain expert in Bun: he created the project and knows the codebase better than anyone

The rewrite has been shipped to production and powers Claude Code today.

We cover a lot more on this in What can we learn from Bun’s rapid Rust rewrite with AI?

3. Changing engineering practices

So, what has changed in how teams build software at Anthropic, compared to the pre-AI days? That’s the question of this article, and it seems that many things are different. Let’s go through it:

AI lab-specific practices

Some things as normal as breathing at AI labs like Anthropic stand out as different with an outside perspective:

• Everyone runs multiple AI agents all the time. Running 3-10 parallel agents is a given. Folks I talked with had their agents running in the background or cloud.

• No token budget, usage not tracked. One major difference between AI labs and everyone else is that there really is no token limit or token leaderboards that promote tokenmaxxing; people already use agents all the time.

• Very high autonomy. Work is becoming more structured inside AI labs, but there’s still massive autonomy compared to Big Tech and most startups. When everyone has unlimited tokens, it’s pretty easy to prototype any idea.

Prototyping and “spiking” is far more fluid

It was several times faster to prototype early approaches for Claude Managed Agents. Similarly, “spiking” the Claude Code mobile backend implementation was much faster than pre-AI, Katelyn told me.

Verification takes longer than implementation

Jarred made a point about the split between implementation and validation in his 11-day rewrite to Rust. Roughly, it was:

The “implementation” part of rewriting the code from Zig to Rust took about 15% of the time, while 85% went on fixing things up: getting it to compile, fixing tests, verifying that it worked.

Most tokens no longer spent on implementation

Thariq:

“We see that few tokens are spent on actual implementation. Most are spent on discovery of unknowns, prototyping, mocking, and then in verification and testing.”

Jarred’s Bun rewrite echoes this: he spent more tokens on fixing up the implementation and verifying that it worked than on the implementation itself!

Code review and more testing by AI

Jarred:

“Critiquing the code and testing it with agents is a new approach we do a lot more of. I think a lot about trust when you merge a lot of code. How do you merge 100+ PRs a day, and make sure the code works? At this pace, you need to trust the code without the ability to read it all yourself. And I think it’s a few things:

• Code review: it needs to be really good and automated. I’m clearly tooting our own horn here, but I find Claude’s code review to be really good. Claude’s code review catches bugs that would take me an hour of closely reading the code to figure out. The caveat is that it’s expensive!

• Security scanning: for this Rust rewrite we did 11 runs of the Claude Security Scanner.

• Fuzz testing: we’ve also been doing different types of fuzzing (fuzz testing), where we had Claude write a fuzzer for things like parser fuzzing.

Running out-of-process testing, where it happens in a different process/session from coding, is one way to build trust in the code. I expect more of this.”

New pattern: fanning out work to AI

Jarred described a new way he works:

“A new approach I’m using is fanning out a lot of the work to many Claudes at the same time. I did this with the Bun rewrite, but I use it for other work. This approach works very well for me, and I feel it’s pretty underused.”

Time-saving automations powered by agents more widespread

Jarred listed several time-saving automations set up by the Bun team to run an active open-source project with a small team, while the team works on Claude Code:

• Every time someone files an issue, Claude runs to try and reproduce the issue. If it succeeds, it starts another container, which then tries to fix the issue and submit a PR.

• The agent tasked with submitting a PR has to write a test that fails in the system version (the one without the patch) of Bun, and passes in the debug build with the patch, before it is allowed to submit a PR

• There are other automations, like if there is no test, the PR is auto-rejected; all linters are run: Claude Code review is run, CodeRabbit’s code review is run, and the agents go back and forth on the GitHub pull request

Auto-merge of pull requests: coming soon?

Pull requests are merged manually when all quality gates pass, but this could become automatic at some point. Once all the above checks pass, all (AI) code review comments are addressed, tests are added to new code, etc. As an interesting aside, a lot of GitHub activity is Claude talking to Claude!

But manual merging may vanish in low-risk cases, at least for the Bun project. Jarred told me:

“Today, a person presses ‘merge’ but within a few months, I expect:

• Automated reviewer LGTMs

• → another Claude with a fresh context window judges if it’s simple and low blast-radius

• → if it is: auto-merge!”

Test assumptions with each model generation

Inside Anthropic, the team keeps testing their priors. Thariq gave an interesting example:

“The thing with agents is that you have to revisit any assumptions you have made because it can change with a new model generation. For that reason, we deleted 80% of the Claude Code system prompt recently because the model has gotten smarter.

Using HTML is another assumption we needed to re-examine. HTML is one of those things which Claude is a lot smarter at than many of us expected. I’ve started preferring HTML as an output format over Markdown, and see this being used by others on the Claude Code team.

HTML can convey much richer information compared to markdown, HTML documents are easier to read and share.”

4. Team-level changes

At Anthropic, there are also changes in how engineering teams operate, compared to pre-AI.

Read more

Today, we cover:

1. Moving my video podcast off Spotify due to constant reliability issues. Spotify’s podcast platform has become chronically unreliable since the company’s leadership started boasting about high AI adoption. But competitors haven’t had similar issues, and so I have offboarded from Spotify.

2. Will Kimi K3 trigger US push for closed-source AI models? Moonshot AI’s latest open model, Kimi K3, is on par with Anthropic’s Fable 5. Could it lead to the US government regulating or banning Chinese open models to protect US labs?

3. AWS laughs off “heart attack” billing error. AWS customers were billed trillions more than they should have been, due to what was likely a conversion error. But instead of sharing an incident report, AWS saw the funny side.

4. Industry pulse. OpenAI’s unreleased model tried to hack HuggingFace to improve its test scores, X took more than a year to develop its new Android app, Google’s new AI model flops, and more.

1. Moving my video podcast off Spotify due to constant reliability issues

Read more

Subscribe now

After I recently interviewed Simon Eskildsen, co-founder and CEO of turbopuffer, on the main stage at the AI Engineer’s World Fair, many people at the event told me they found him relatable and inspiring for his choices to stick with one company for close to a decade, his belief in the power of “napkin math” to reveal why products run slowly or cost too much, and for his insight about how too much of VC funding is about ego, not business needs.

This article contains the most interesting parts from that conversation; in particular, the concept of “napkin math” – doing quick calculations to get rough answers – as a way to challenge existing systems to improve. The full 55-minute-long video of the discussion at the AI Engineer’s World Fair is available to watch:

Watch the full interview

Today, we cover:

1. Algorithmic programming speedrun. While in high school, Simon competed in the International Olympiad for Informatics (IOI) which pushed him to learn about writing correct programs that are fast and memory-efficient, and more.

2. Eight years of infra at Shopify. There are many upsides to longer tenure: Simon learned infrastructure concepts, dug deep into databases running across regions, and learned to write software that ages well.

3. “Napkin math” as a superpower. Simon became obsessed with finding the theoretical limits of compute operations, such as sending over data and reading bytes. This held him in good stead at Shopify, and then at his startup.

4. Origins of turbopuffer. When ChatGPT took off, context windows were small, and so stuffing them with the right information was key for AI applications. Fast search was needed, but the search solutions were surprisingly expensive. Using “napkin math”, Simon discovered they were far more expensive than necessary.

5. A new product without VC funding & Cursor as customer no.1. After raising $8M in seed funding, Cursor rolled the dice on the new turbopuffer team after Simon helped with their search & database needs.

6. Reasons to raise venture capital. Fund R&D, fund growth, stroke founders’ egos, and more.

Disclaimer: turbopuffer is a season sponsor of the podcast, but as with all our deepdives, this article is independent of podcast sponsorships.

1. Algorithmic programming speedrun

A self-taught professional, Simon skipped college to work at Shopify and spent nearly a decade there building a variety of systems. His interest in computers started after initially getting into building websites aged 12. Growing up in Denmark, he dabbled in HTML with tools like Microsoft FrontPage and Adobe’s Dreamweaver.

While still a teenager, Simon “hit a wall” by exhausting the Danish-language part of the internet for learning programming, and got into the World of Warcraft MMO game, which helped him acquire English.

After discovering the International Olympiad for Informatics (IOI), he decided to enter, despite a very competitive, multi-stage selection process open to all Danish high schools. IOI problems are pretty similar to Leetcode problems: algorithmic challenges that value correctness, speed, and memory usage.

Simon cleared the online qualification round and was invited to the Danish Nationals. This was more than a competition: a weekend-long bootcamp to teach participants more advanced programming techniques such as recursion (which Simon already knew), the divide-and-conquer algorithm, and dynamic programming; one of the more tricky concepts to master for algorithmic programming.

I tip my hat to the organizers for creating a challenging bootcamp and offering the opportunity for people like Simon, who wasn’t aware of what “NP complete” meant when he entered. He recalls:

“The routine was that every four hours we’d be introduced to a new “programming concept”, and receive ~2-6 tasks where this, combined with previously introduced concepts, had to be applied. All the solutions had to be submitted to the same site I submitted my qualification solutions to, as it was all part of the final evaluation. The tasks were incredibly challenging, like nothing I had ever tried before.

Sometimes in extreme desperation combined with tiredness from the trip, I’d think about taking the next train home. This feeling would disappear with the utter joy and confidence that arose whenever I would finally solve a task, and creep back once again when I found myself still struggling after an hour on a new problem. But this kept me going. By Saturday afternoon, I had almost managed to get up to speed with the others, and was doing the same tasks as them.”

Ultimately, Simon claimed one of six spots in the Danish national team, making it through to the regional finals.

Aware of how little he knew about programming, Simon doubled down to catch up. He realized that most other participants were better prepared and that he had to do something to survive the next qualifying round. So, he got to work:

As Simon recalled:

“I armed myself with a borrowed copy of “The Art of Computer Programming”, worked through the exercises, read up on common algorithms on Wikipedia, completed tasks on USACO, and memorized the critical parts of my Vim config for the competition computers. I managed to create quite an intense training weekend for myself.”

The regional finals were even more challenging than the Nationals, and participants struggled to write performant solutions to the problems. Despite putting in the effort, Simon was pretty sure he was out of the competition, and recorded his learnings:

“[From the programming competition] I learned that you must avoid digging holes. Repeatedly, I found myself so fixated on getting a particular idea to work that I’d get absolutely nowhere. Sometimes, you have to bite the bullet, delete your program, find a new sheet of paper, and start from scratch. A good case of this is when you start working around a general solution to solve specific edge-cases. I learned that there is almost always a simple way to solve a problem without explicitly handling edge-cases. If there are two edge-cases, there’s almost certainly two more. The simple solution will handle edge-cases automatically – even those you might not have considered.”

But as it turned out, two months later he was told he had been selected to represent Denmark in the final phase of the competition. Simon continued to push himself out of the comfort zone of web development, using HTML, a bit of PHP, and getting into algorithmic programming.

Another decision that would later bear fruit career-wise was starting a blog while at high school – though he didn’t know it at the time. In 2010, Simon launched his English-language blog but posted only one or two articles per year; mostly short ones describing problems he’d solved:

• A five-line script to take screenshots and upload them to Imgur, all from the command line (2010)

• Setting up a Ruby HTTP server on NGINX using the library Unicorn (2010)

• Why Simon favors SQL over ORMs: Object-Relational Mappings (ORMs) are frameworks to abstract SQL, these were very popular around that time (2011)

Two posts he published would go on to have an impact on his life. One was about his IOI experience and learnings. The other only came about after he suffered the disaster of fatally dropping his iPhone during his final year in high school.

2. Eight years of infra at Shopify

With his smartphone dead, Simon was forced to switch to an old-school Nokia “dumb” phone. He wrote a short article about the experience titled Why I’m glad my iPhone broke, which went viral on Hacker News, making it to the front page of the site with many comments.

It seems that someone at Shopify, in Canada, read the article and others by him – including the summary of his impressive performance at the International Olympiad for Informatics – as a recruiter from Shopify flew Simon out to interview in Ottawa, Canada, where he was offered a software engineering position at the company.

In his new job on the infrastructure team, Simon made notes about things he didn’t understand and read up on new concepts. As he told me:

“When I started at Shopify, I was insecure about having not studied computer science and my biggest exposure to programming had been the IOI. If nothing else, preparing for the IOI taught me that you can sit down, read a paper, and figure it out if you spend enough time on it. So, I did that repeatedly.

In my first year at Shopify, every time I heard something I didn’t know, I noted it down on a piece of paper. Then, that evening, I would read up about it.

For example, if someone at work mentioned TCP, I assumed that surely they would know exactly what’s in the three-way handshake and how TLS is layered on top. And I also assumed they’d looked at Wireshark and all of that. I don’t think that’s true, but that’s what I thought at the time! So, I dug deep into everything I encountered.”

Working in infrastructure meant solving interesting engineering problems at a time when Shopify was growing 120-140% in load year-on-year; Simon was exposed to problem domains like:

• Ruby on Rails and databases: Shopify was already one of the world’s largest Ruby on Rails monolith applications, and working on infrastructure meant being close to the database layer

• Sharding and cutting over: as Simon’s manager used to say, “you cannot cache writes”, so Shopify had to move from running on a single group of machines to a shard (partitioned) setup of machines. The team did the cut-over (moving to the shard) just a week before Black Friday, the busiest time of the year.

• Multi-data centers: expanding database footprint from one data center (DC) to multiple DCs

• Splitting up key services: Shopify had a 128GB machine (massive for the time) running Redis as a key-value store. Nobody dared touch it until one day the service went down. Then, the team split responsibilities up into separate services.

Simon built a framework to simulate networking conditions called toxiproxy. The idea came to him while attempting to write a more thorough, systems-level test to see how Shopify’s application held up during partial outages. He created a matrix of Shopify’s services, and wanted to write a complete test suite for this matrix, to see if the system was resilient enough to handle some parts of the system being down. For example:

As Simon wrote about this problem at the time:

“Having tests for the matrix was a must; otherwise we couldn’t guarantee the state of the matrix wouldn’t degrade over time. Since the tools mentioned previously require root access, we investigated proxies to simulate latency and downtime at the TCP level, but didn’t find one that suited our needs. We needed an online API to edit proxies and to support deterministic latencies, which made it suitable for integration testing.”

This was the inspiration that led to toxiproxy. As Simon told me on stage:

“Toxiproxy is a proxy that sits in between the application and the databases. With the proxy in place, you could do things like make an API call to the proxy, instructing it to simulate taking the database down, or making it slow.

Over time, we added a bunch of other ways to inject failures. With this proxy, we did not have to mock low-level drivers, but we could test failure handling really well.”

Toxiproxy was open sourced in 2014, and apparently still runs in Shopify’s CI system 12 years later, to Simon’s knowledge!

The biggest benefit of a long tenure at a company was learning to write software that ages well. Simon told me that this was the lesson that made it so worthwhile, and how often the simple solution someone put together in a week or two outlasted the big, multi-team RFC-driven solutions. That was a lesson he still uses today.

3. “Napkin math” as a superpower

While at Shopify, Simon became interested in figuring out the theoretical limits of certain computer operations. For example, how much bandwidth does DRAM have for memory transfer? How long does a round-trip operation to AWS S3 take, and how much does it cost? What does a gigabyte of memory cost?

He sought the actual numbers, wrote a script to collect them, and created a table with the data:

Simon started to memorize key numbers with flash cards. He wanted to be able to instantly recall all important numbers. As he told me:

“Napkin math was essentially just this table that I maintain on GitHub. There’s probably like 50 of these numbers and then a script that generates them all.

For example, what does a gigabyte of memory cost? $2. What does a gigabyte of S3 cost? Two cents. What does a gigabyte of this cost? 10 cents. What does it cost on spot? What does it cost on a three-year commit? I had a massive table, then I created flashcards for almost every single cell so I know all these numbers.

This was a project I started taking on at Shopify because I would review projects and these numbers would be helpful.”

Napkin math enabled him to challenge design decisions based on benchmarks that had issues. As he told me:

“When reviewing a project, a product team would tell me that they chose Database A over Database B because they benchmarked both, and Database A was better. I hate benchmarks because making design decisions based on benchmarks is not a satisfying answer to me.

For example, you’re saying that with Database A, as per the benchmark, it takes 10 seconds to do this one thing. But it should take 10 milliseconds if you do the napkin math. Say, it’s a search query. Okay, you’re searching for three terms then. Each term has this many documents that match it. That’s this many megabytes. We intersect this many lists. You have DRAM bandwidth on multiple cores, at 100 GB/sec. So if you do the math, it should be 10 milliseconds.

So, now you’re telling me that the benchmark for the same thing takes 10 seconds, which means one of us is wrong! Either there’s a gap in my understanding – which is possible! – or the benchmark measures the wrong thing.

Often, it would be things like the person doing the benchmark and not realizing that the query would be a distributed one, running across a hundred different nodes. And in this case, of course the p99 is going to be very high!”

Simon went even deeper into “napkin math” after leaving Shopify. He investigated whether MySQL’s maximum transactions per second is equivalent to fsyncs per second (the number of file writes per second a system can handle). He discovered that MySQL can handle more writes than the operating system could sync to the file, which was a surprise:

“It takes ~3 seconds to perform 16,000 insertions, or ~5,300 insertions per second. This is 5x more than the 1,000 fsyncs per second our napkin math told us would be the theoretical maximum transactional throughput!

Typically, with napkin math we aim for being within an order of magnitude, which we are. But when I do napkin math, it usually establishes a lower bound for the system, i.e., from first-principles, how fast could this system perform in ideal circumstances?

Rarely is the system 5x faster than the napkin math says. When we identify a significant-looking gap between the real-life performance and the expected performance, I call it the “first-principle gap.” This is where curiosity sets in. It typically means there’s (1) an opportunity to improve the system, or (2) a flaw in our model of the system. In this case, only (2) makes sense, because the system is faster than we predicted.”

Simon started to investigate, and learned that MySQL does grouping of transactions (not doing an fsync for every write), and it also does smart merging of multiple fsync operations that would be processed in parallel, effectively doing a “group commit” to further improve performance. This is a learning he’d later use: grouping writes together on top of S3, to reduce latency and improve cost.

Personally, I find it amazing how much you can learn with some measurements and by asking questions about how a system achieves results that go against what the “napkin math” suggested should be its limit!

4. Origins of turbopuffer

“Napkin math” also played a part in the creation of turbopuffer. Simon told me that three things had to combine for the project to come to life.

1. Search was a difficult project at Shopify. Building search was one of Simon’s final projects at the company, and he did not have a good time with it. He used a popular database vendor but struggled to have it perform napkin math. The query plans were not exposed easily by the database, so he could not figure out what was missing and why it wasn’t performing the napkin math without reading massive amounts of source code. Also, the search infra was difficult to operate.

2. Napkin math became a surprisingly efficient reasoning tool. As a toolkit, napkin math gave Simon a way to reason about what could be achievable with a machine, if utilized perfectly.

3. His friends’ startups needed fast search badly because of AI. In early 2023, Simon helped a few friends with infra at their startups, many of whom had the same problem: LLM context windows were very small (4-8 KB), and they needed to fill them with the right parts of documents, but this demanded very fast search. One startup budgeted that their new search vendor would cost $30K/month, while the existing infra bill was only $5K/month! For a bootstrapped Canadian company, the cost was too high, so they didn’t ship the AI feature.

Simon could not stop thinking about why search was so expensive, and why it didn’t line up with what the “napkin math” predicted the cost should be. Eventually, he laid out a basic architecture to make it fast and cheap:

1. Store the data to search in AWS S3

2. Do some clustering and organize the files

3. Get latency down. S3 is cheap and reliable, but has high latency

Driven by curiosity about how that could work, he set out to build it. As he told me:

“I just became fully obsessed that summer [of 2023] with building it. The first version was the simplest possible thing. I’m a very pragmatic person, so I didn’t get buried in detail. I barely read the literature on log-structured merge (LSM).

The simplest way to do this is to run a clustering algorithm on the vectors. You get the clusters, and then put the clusters in files. The files are called ‘cluster_01’, ‘cluster_02’, ‘cluster_03.’ Then you have a file called centroids of the clusters. Then, you search by downloading centroids, looking into them, then downloading the closest clusters.

There were a few optimizations around merging clusters that were adjacent in files, just to control costs and boost performance. But this was the core of it.”

What about performance? Simon:

“For the first version, I didn’t even implement a dedicated caching layer. I just put the reverse proxy (NGINX) in front of S3, and that was it! The performance improvement came from caching all of the S3 objects. It was the simplest possible layer at the NGINX level.”

5. A new product without VC funding & Cursor as customer no.1

I first heard of turbopuffer last year, when I interviewed Cursor cofounder Sualeh Asif about how they built the AI coding harness. From our previously published Cursor deepdive:

“The ongoing need to re-shard as usage grew became error-prone and frustrating. Sualeh told me the biggest lesson the team learned was to avoid sharding where possible in future. So, they looked for a vector database that could support multi-tenancy without requiring manual sharding.

Turbopuffer was a startup that promised this, so Cursor tried it and migrated a good part of their vector search use cases over.”

So, how did a billion-dollar startup come to bet on a tiny, unproven infra product with zero customers? There were a few factors:

First, Cursor was not yet a billion-dollar company in the fall of 2023, but a relatively fresh startup. The company raised $8M in seed funding in the same month. In 2024, the company was valued at $400M, and then its valuation surged to $29.3B. It was sold to SpaceX for $60B this year.

Secondly, it involved a tweet Simon posted after having had enough of building all summer, and wanting to test the waters. As he told me:

“I was so sick of working on [turbopuffer]. I’d been working on this all summer and didn’t know if anyone cared. I only wanted to work on it if anyone cared. So, I decided to put it on Twitter.

At that point, I had a single TMUX instance running on an 8-core node somewhere in GCP. I was thinking: “if someone goes to prod, I’ll set it up properly on multiple nodes.

But first, let’s see if anyone cares.

Anyone who’s worked in the internals of databases would’ve had too much pride to ship anything like that; I was just releasing it like a SaaS project. Why can’t you work on a database like it’s SaaS?”

So, he put out this confident-sounding tweet:

Simon had the confidence to launch his product because he knew it was rock-solid and scalable:

“I knew turbopuffer was reliable. It upheld its invariants. For example, you shut down all the VMs and no data is lost. All the writes are committed directly to blob storage.”

Cursor reached out. The company was an eight-person team at the time, fresh from seed funding and growing fast, but with a search problem that they were looking to partner with a startup on. Simon recounts:

“Knowing Cursor’s founders now, I’m sure they must have sat at the dinner table one day and were like: ‘the unit economics of what we have right now, where all the vectors are in DRAM, are not working.’

They probably were asking why someone had not built a solution where you can put all the vectors from the codebase into S3 (to make it cheap), and move the part of the codebase used in memory (to make it fast). Then, everything sits in blob stores and you just hot load it all into cache. When you open the codebase, after a few seconds it’s all in RAM and the queries are as fast as everything else.

Aman (one of the cofounders) was already talking about using S3 as a key-value cache, which at the time, barely anyone was thinking of for unit economics.”

Simon knew nothing about B2B sales at that point, and was not trying to “sell” anything to Cursor. He exchanged a few emails with the Cursor team about their use case of searching many small local codebases. He wanted to help Cursor with the unit economics, while also proving that turbopuffer works. So, he hopped on a flight from Canada to San Francisco, arrived at Cursor’s office and got down to debugging their database provider:

“When I showed up at Cursor’s office, they were having some Postgres problems. I asked, ‘do you guys have pganalyze?’ And they didn’t, so I was like, ‘Okay, let’s get that going. Let’s look at it. “

And the problem was the same thing as it always is with Postgres: autovacuum hadn’t run enough. And so they had all of these going to heap, when they should be doing index scans, etc. So, we were talking about all of that.

I was just helping them: my “database genes” just kicked in. I think this built enough trust with them to believe that if I know enough to help them with their database, maybe I also know how to build one.”

It was around when turbopuffer’s other cofounder, Justine Li, joined that the work with Cursor kicked off. Simon describes Justine as the best engineer he ever worked with at Shopify, and together, they made more performance optimizations. Cursor migrated their local code-base search over to this brand new product.

Simon promised to reduce Cursor’s bill by 95%, and delivered – thanks to napkin math! Cursor was spending around $80K/month on indexing and search; with turbopuffer in place, this dropped to $4K/month. Simon was confident about making this prediction thanks to napkin math! He did the calculations based on the fundamentals; only counting resource usage that Cursor was using, and not taking a margin for turbopuffer’s operating cost at the time.

6. Reasons to raise venture capital

After securing Cursor as the first customer, Simon was not convinced that turbopuffer should raise VC money. He explained his thinking:

“I understood that if you take venture capital, no matter how many smiles there are in the room, everyone’s expecting to earn a big return on some timeline that makes sense to everyone involved. And ‘everyone involved’ are pension funds in Canada.

But at the time, I did not know if turbopuffer could be a billion dollar company. It felt like a very niche kind of search engine. And that was completely fine with me!

So, I just looked at what we invoiced and I looked at my GCP bill. And I was making this equation that:

Customer Invoices >= GCP bill [the turbopuffer costs]

Justine and I were going to optimize the system until these numbers were roughly equal. And if we could get some other workloads over to turbopuffer, we could then start paying ourselves.

But either way, I didn’t know if I could even go and raise a bunch of money. I didn’t have any relationships. I was an outsider who grew up in Aarhus, Denmark, and moved to Canada.”

A few months later, turbopuffer ended up raising a total of $700K, just to be able to hire two engineers until the end of the year on real salaries. Simon found that most VCs did not take them seriously for wanting to raise too little money, and assumed this signified a lack of ambition! Two years later, with less than $1M in initial funding raised, turbopuffer crossed $100M in annual run rate. So, there evidently was ambition!

Since our conversation, Simon has reflected on the topic of VC capital and built a mental model of six reasons that justify raising external funding:

Reason #1: Research and development. This is what turbopuffer raised for: they needed the money to hire two more engineers until the end of the year, in order to build out more of the platform. Then, they would either get more customers and generate enough revenue to not need funding, or shut down the project if it failed to gain traction.

Reason #2: Growth. When you’ve built something and want to tell the world about it, often, you have to spend money on doing that.

Reason #3: Massaging founders’ egos. Simon:

“This is a very, very popular reason! You see big numbers, you get lots of press. But I think it’s a really dangerous reason to raise money.

I wish this reason was talked about more, because you are diluting all of your employees when you do it. And for some people, it can become a status game. It’s not what we are about: we’re here to build a big business together.”

Reason #4: Employee rewards. A startup is a long journey, and everyone wants to work with the best people. But there are necessarily few of these folks, so you want to reward the standout ones by raising money so that employees can sell their shares. This was why, in December 2025, turbopuffer raised a round where they let employees sell some of their equity: it meant they would not have to wait for a future financial event like an IPO.

Reason #5: Strategic partnerships. It might be the case that raising from a VC whose network of connections is key to your business succeeding, or that taking funding from a strategic investor gives you access to their platform or services.

Reason #6: Mergers & acquisitions (M&A). Purchase another company with the funding in order to expand the business.

Simon encourages founders to be honest about why they raise capital funding – and to watch out if the reason is egocentric!

Takeaways

Watch the full interview here.

It’s inspiring to see how far it’s possible to get with “napkin math” and by understanding the bottom layers. It didn’t sit right with Simon to make decisions about vendors based on hastily written benchmarks that might measure the wrong thing. By understanding the constraints of data transfer latency and storage cost, he found a way to estimate the theoretical lower bounds of the system.

This helped lead to more informed design decisions at Shopify, and showed him there was an opportunity to build a faster, cheaper search product than the status quo.

Getting “lucky” in business requires a lot of skill, and in-person greatly helps with first impressions. Turbopuffer’s first customer being Cursor sounds almost too good to be true. But the account from Cursor cofounder Sualeh Asif – and now from Simon – reveals the ingredients in more detail:

• Spotting a new business need: AI-native startups like Cursor saw their search bills explode, but needed search functionality to offer usable AI products

• Offering a product with a magnitude of lower pricing: what people made suddenly pay attention to turbopuffer was the promise of not “just 20-50%” cost savings, but a seemingly radical, 90%+ reduction in costs. When you’re late to enter a market (like search), you need major differentiation, then deliver on it! This is also what Cursor’s attention.

• Building trust before a sale: Simon helped the Cursor team fix their existing database before discussing using his product

• In-person impressions: if Simon had not flown to San Francisco to meet the Cursor team in person, would they have taken a bet on turbopuffer?

• Launching at the earliest opportunity: none of this would have happened if Simon did not announce the first version of the product when he knew it could work, but was still in a pretty unpolished state!

It’s always helpful to be aware of the dynamic of venture funding – some of which are rarely mentioned. Simon knew investors expect returns and growth on timelines which they set. Raising money is helpful in many situations, but problems arise from egotistical reasoning, and Simon believes too many founders prioritize them – knowingly or not.

Today, we cover:

1. Grok’s CLI uploaded all your local files to the cloud, then got caught. Just as devs were starting to use the Grok CLI – thanks to the capable Grok 4.5 coding model – they discover…

Read more

Listen and watch now on YouTube, Spotify, and Apple. See the episode transcript at the top of this page, and timestamps for the episode at the bottom.

Brought to You by

• Antithesis — with Antithesis, you can use AI agents to work on critical systems without worrying about correctness. Going far beyond code review, you can run your complete system in a hostile environment, analyze its behavior, and reproduce every issue perfectly. Teams like Jane Street, Fly.io, and the etcd community use Antithesis to ship better code, faster. Learn more.

• Buildkite — the CI platform trusted by OpenAI, Anthropic, Cursor, Meta, Uber, Ramp, Nvidia, Airbnb and many more. Buildkite was stress-tested at the largest scale inside companies solving some of the hardest engineering problems. It’s built to reliably manage whatever your coding agents throw at the build queue, today, next year and beyond. Learn more.

• Sentry – application monitoring software built by developers, for developers. Sentry’s Seer AI agent is one of their new, neat tools, as a way to debug errors faster. Same with Sentry’s MCP server. Check out Sentry.

In this episode

Knowing how LLM contexts work and how to work around context limitations – aka “context engineering” – is becoming more important for software engineers working with LLMs. Let’s look into what works and what doesn’t, today.

In this episode of The Pragmatic Engineer podcast, I sit down with the CEO and cofounder of HumanLayer, Dex Horthy, who coined the term “context engineering”. We discuss the ideas behind this context engineering, harness engineering, loop engineering, software factories, why his approach to AI-assisted software development has evolved, and how HumanLayer is helping engineering teams automate more of the software development lifecycle without sacrificing code quality.

Key observations from Dex

Here are 12 useful points Dex made in our conversation:

1. Dex talked with ~100 “real” AI Engineers and wrote the popular ‘12-Factor Agents - Principles for building reliable LLM applications’ based on what he learned. Around August 2024, he started to build AI agents when the common approach was to use frameworks like LangChain and CrewAI, and also talked with around 100 AI engineers doing tangible work such as taking on $100K+ contracts to ship AI solutions within enterprises. They tried those frameworks and discarded them in favor of building their own pipelines. Dex shared the learnings from the conversations in his hit book, 12-Factor Agents. It’s a great read!

2. Lesson learned: Shipping unread code spells disaster within months. Dex experimented with having the model write the code and humans not reviewing anything in July 2025. Four months later, they shut things down and threw the whole system out. Production broke, and no matter how much the team prompted Opus 4.1, the model could not find the root cause.

It took days of wading through spaghetti code to discover the primary key wrongly routed through the complete codebase. Once fixed, it took three weeks to re-onboard to a codebase no human had ever read. Today, Dex thinks this problem wouldn’t even take four months to develop, now that newer models produce code a lot faster than a year ago.

3. Today’s coding models are most likely trained in a way that makes codebases worse over time. Dex believes that the reason we see LLMs “degrade” existing codebases is because they are optimized to do well on SWE-bench-style benchmarks. These benchmarks reward reproducing a known fix in codebases like Django, but cannot measure poor architecture decisions. This is because the cost function of bad architecture and bad program design cannot be evaluated by running a unit test. It’s a tricky problem to solve: Dex’s best guess is an eval of a model building 20 features in a row in a codebase without knowing what’s coming next.

4. Context engineering 101: figure out where the “dumb zone” begins. As a rule of thumb, the less of the context window that is used, the better the outcomes are. This is because the attention mechanism is quadratic: the more that goes into the context window, the more compute is required to process it all. We cover more about self-attention scalability challenges in our ChatGPT deepdive.

For a model with a 1M context window, Dex pushes it to around 300-400K when it feels right. For smaller models, he stops at around 100K. You hit the “dumb zone” when its performance starts to degrade because the context window fills up beyond this heuristic limit, and the model begins doing increasingly stupid things like deleting your .env file, for example.

5. A larger context window does not mean a smarter model. Models’ intelligence is behind the ability to use the tokens in the context window, by deciding which parts of the context are relevant for the next decision. You have to get a feel for how much context usage makes sense, and experiment when you hit the “dumb zone.”

6. Frequent, intentional compaction is a technique Dex uses for more complex projects. He will take a long and noisy context, compress it into a Markdown document, then start a new session fresh, pointing the model to this “compressed context” that is in the Markdown.

A workflow he uses:

• One session reads a ton of code (filling up its context window while in the “smart zone”), then emits a research document

• The next session takes tickets describing the work to be done and turns it into a design document

• The following session takes both documents to create a plan

• The human is in the loop where it really matters: in this case, reviewing the design document and architecture because Dex finds models to be weak on this

7. Don’t bother optimizing LLM usage until business is booming, there’s massive scale, or high costs. Dex suggests to always start building software with the smartest available model to solve the problem, since engineering time is almost always the bottleneck. Begin to optimize LLM usage and context usage only when at real scale and costs are high enough. That’s when it can be worth using GPT-OSS-120B (1/1,000th the cost of Opus) for the simpler steps in your process.

8. “You’re completely right!” or “you’re right to push back on that” are phrases that mean it’s time to start a new session. These responses mean the LLM session is trajectory-poisoned, and you’re wasting time and tokens to continue. Models are autoregressive, so if you get into this loop of:

• Model makes a mistake

• → user “yells”

• → model keeps making mistakes

• → user “yells”

• … the model calculates that the next most probable message is to make another mistake!

9. Only four things matter in the context window:

• Size: the bigger it is, the more space you should have before hitting the “dumb zone”

• Information quality. Once something is in the context window, every subsequent turn treats it as fact. This is why errors can compound.

• Missing information: if there’s information missing that the agent would need, the outcome will be worse, as the agent fills in the gap with guesses.

• Trajectory. Models are autoregressive, so they predict the next message in the conversation based on previous ones in a kind of thread of reasoning. “Trajectory poisoning” is when the agent gets into a pattern of doing things you don’t want. In this case, it’s time to start over.

10. Slow loops are Dex’s favorite way to do ‘loop engineering.’ The HumanLayer team started with a nightly automation setup that kicks off an agent to fix one thing in the codebase, and open a pull request. In the mornings, they woke up to a PR waiting to be merged. They tweaked it over time and now have four agents open a total of four PRs by the morning, with the focus on code quality improvements. A person still reads all of them before merging.

11. “Token harder” vs. “token smarter”: Dex is in a group chat named ‘Hyper Engineering’, where members share advice on how to max out their Claude subscriptions. This approach, he calls “token harder”. On the other side is “token smarter”: aiming to get maximum value from AI while keeping control. Smarter is harder to pull off.

12. Three ways to run a “software factory.” Here’s options Dex sees as viable:

1. “Turn the lights off:” go all-in on agentic coding, do not review the code, and pray that AI doesn’t create too much slop. Dex tried this and failed.

2. Read and review all AI-generated code. This slows things down to human speed. Dex says that this way, you should expect a 30-50% lift in productivity from AI, compared to pre-AI engineering.

3. Find leverage, but keep people in the loop. Find out where an hour spent in planning could save four hours’ worth of implementation, in terms of fewer bugs. Invest more time in areas with leverage: design, architecture, and key decisions. Then, let the agent generate code and don’t insist on reviewing all of it. In this way, Dex believes you can move 2-3x faster than when devs wrote all code by hand.

The Pragmatic Engineer deepdives relevant for this episode

• How Uber uses AI for development: inside look

• Are AI agents actually slowing us down?

• AI Tooling for Software Engineers in 2026

• Vibe Coding as a software engineer

• How Claude Code is built

• AI Engineering in the real world

• The AI Engineering Stack

• How AI-assisted coding will change software engineering: hard truths

• The creator of OpenClaw: “I ship code I don’t read”

Timestamps

00:00 Intro

03:35 Dex’s path into tech

05:36 Early work in platform engineering

07:30 Replicated

13:26 Metalytics

14:38 12-factor agents

20:29 Context engineering

25:40 Harness engineering

28:13 Context overload

32:47 Loop engineering

46:36 Software factories before and after AI

52:35 Automation limits

57:20 Three options for automating

1:01:02 RPI framework

1:06:18 Intentional compaction

1:13:50 Token harder vs. token smarter

1:18:46 AI slop

1:21:17 HumanLayer

1:31:11 Book recommendation

References

Where to find Dex Horthy:

• X: https://x.com/dexhorthy

• LinkedIn: linkedin.com/in/dexterihorthy

• Website: https://www.humanlayer.dev

Mentions during the episode:

• HumanLayer: https://www.humanlayer.dev

• Jet Propulsion Laboratory (JPL): https://www.jpl.nasa.gov

• Dykstra’s projection algorithm: https://en.wikipedia.org/wiki/Dykstra’s_projection_algorithm

• Replicated: https://www.replicated.com

• Docker: https://www.docker.com

• HashiCorp: https://www.hashicorp.com

• DataStax: https://www.ibm.com/products/datastax

• Puppet: https://www.puppet.com

• Travis CI: https://www.travis-ci.com

• Circle CI: https://circleci.com

• Randy Newman’s website: https://www.randynewman.com

• 12-Factor Agents - Principles for building reliable LLM applications: https://github.com/humanlayer/12-factor-agents

• The creator of Clawd: “I ship code I don’t read”: https://newsletter.pragmaticengineer.com/p/the-creator-of-clawd-i-ship-code

• Vaibhav Gupta on LinkedIn: https://www.linkedin.com/in/vaigup

• Tobi Lutke’s post on X about context engineering:

• Andrej Karpathy’s post on X about context engineering:

• Improving Deep Agents with harness engineering: https://www.langchain.com/blog/improving-deep-agents-with-harness-engineering

• # Skill Issue: Harness Engineering for Coding Agents: https://www.humanlayer.dev/blog/skill-issue-harness-engineering-for-coding-agents

• Harness engineering for coding agent users: https://martinfowler.com/articles/harness-engineering.html

• How AI will change software engineering – with Martin Fowler: https://newsletter.pragmaticengineer.com/p/martin-fowler

• Dex’s post on X about context reality check:

• Laurie Voss on LinkedIn: https://www.linkedin.com/in/seldo

• Everything is a ralph loop: https://ghuntley.com/loop

• Dex’s post on X about feedback loops:

• Dex’s post on X about token harder vs. token smarter:

• Dex’s post on X about AI slop:

• GitHub: https://github.com

• Paul Graham, Live from Stockholm: https://www.ycombinator.com/library/Q7-paul-graham-live-from-stockholm

• Refactoring: Improving the Design of Existing Code: https://www.amazon.com/dp/0134757599

• Clean Code: A Handbook of Agile Software Craftsmanship: https://www.amazon.com/dp/0132350882

• The Pragmatic Programmer: From Journeyman to Master: https://www.amazon.com/dp/020161622X

—

Production and marketing by Pen Name.

“I don’t prompt Claude anymore. I have loops running that prompt Claude and figuring out what to do. My job is to write loops.”

Soon after, Peter Steinberger, creator of OpenClaw, preached loop design in a post:

Elsewhere, Addy Osmani, formerly of Google, wrote an article, ‘Loop Engineering’:

“Loop engineering is replacing yourself as the person who prompts the agent. You design the system that does it instead.”

That’s three mentions in quick succession of this new approach, which is a novel one and therefore pretty abstract to me. To find out more, I turned online to some of the folks who read these articles. In replies, you told me what “loop engineering” means to you and gave some examples of loops in your work.

Today, we cover:

1. Where it began: “Ralph Wiggum” loop. A year ago, software engineer Geoffrey Huntley shared how he builds “loops.” In December, the approach went viral and “Ralph loops” were born.

2. The /goal command ships in all major harnesses. By May this year, the major AI coding harnesses added support to run a loop from a single prompt, using the /goal command.

3. Loops which devs use: triggers and cron jobs. I asked devs how they use loops. Most use cases involve responding to events or running scheduled jobs. They are useful, but don’t feel like brand new workflows.

4. Helpful loops for devs. Open PRs for newly-recorded app issues, have notes ready when an oncall joins an outage Slack channel, “babysitting” and fixing nightly end-to-end tests, and more.

5. Disappointment and “tokenmaxxing”. Several devs reject looping after trying it. Agents drifting, and the “human in the loop” having better results are some reasons. Also, at companies that pay API prices for tokens, loop engineering gets expensive fast.

6. Was looping a hack while tooling caught up? Distinguished engineer Max Kanat-Alexander believes the “loop” might have just been a temporary hack while the harnesses added the ability to do the same from a single prompt.

7. Does “context engineering” matter more for devs? Except for engineers building AI infra, there seems little benefit in going deep into loop engineering. Instead, becoming familiar with AI context windows – also part of building loops – could be more useful.

1. Where it began: “Ralph Wiggum” loop

Exactly a year ago, software engineer Geoffrey Huntley published the article ‘Ralph Wiggum as a software engineer’. The name references the naive son of the local police chief in The Simpsons, who is extremely eager to always be helpful. In engineering, “Ralph” is intended to continuously nudge the agent in the right direction. Geoff described it:

“Ralph is a technique. In its purest form, Ralph is a Bash loop:

while :; do cat PROMPT.md | claude-code ; done

Ralph can replace the majority of outsourcing at most companies for greenfield projects. It has defects, but these are identifiable and resolvable through various styles of prompts.

That’s the beauty of Ralph - the technique is deterministically bad in a nondeterministic world.”

The article expands on the idea of the Ralph loop:

• Start the agent with a prompt that captures the task and defines a goal

• Create a plan of work, each item with a success criteria

• Start the loop:

  • Take one item per loop

  • When the agent is done, check if the goal is achieved

  • If not: start the agent again, with a clear context window

    • Start loops if needed: spawn subagents as and when necessary

Geoff published the experiments he did with this approach, such as building a new programming language last summer, and said it requires skill:

“Engineers are still needed. There is no way this is possible without senior expertise guiding Ralph. Anyone claiming that engineers are no longer required and a tool can do 100% of the work without an engineer is peddling horses***.”

The “Ralph method” blew up late last year with the arrival of better models which were surprisingly capable of building ambitious projects. Software engineer Matt Pocock created a tutorial, ‘Ship working code while you sleep’ with the Ralph Wiggum technique. He said:

“One of the dreams of coding agents is that you can wake up in the morning to working code, where your coding agent has worked through your backlog. It has spit out a whole bunch of code for you to review, and it works.”

Before the Ralph loop, Matt did this in two steps:

1. Ask the agent to create a detailed plan for the work, with tasks broken out

2. Then, in a sequential order, have the agent complete each subtask in a separate run

A problem with this approach is there’s no easy way to add new tasks to the “masterplan”. Software engineers know that most plans do need to be modified, often while the work is ongoing. In contrast, with Matt’s take on the Ralph method, the “masterplan” is continuously updated in a “master PRD”. Here’s the prompt he gives the agent:

1. Choose the next feature: Find the highest-priority feature to work on and work only on that feature

2. Have tests pass: check that the tests pass (via pnpm test)

3. Update the master tracker: update the PRD with the work done

4. Log work: append your progress to the progress.txt file

5. Commit: make a git commit of the feature

This style of working is more of a “dynamic Kanban”:

The “Ralph method” is all about working around context window limitations. Back in mid-2025, the maximum size of a context window was around 200,000 tokens. That’s not enough for more ambitious tasks, so it’s necessary to break up agent runs into smaller ones and run them, one by one. In this context, here’s where the Ralph method works:

• Have a goal for a project, and keep running (or re-running) agents until this goal is reached

• Persist work done in a “compressed” manner on the filesystem (as logs or an updated plan)

• Start agents with fresh context to minimize “context rot”

• Allow each agent to add or modify the “masterplan” if needed

2. The /goal command ships in all major harnesses

For a few months, building a Ralph loop meant doing it yourself: setting up the loop, state tracking, deciding how the agent can add tasks and when to stop. But things changed once coding harnesses made it easy to run these loops.

April: Codex ships /Goals

About six months after the “Ralph technique” started gaining wider traction, Codex shipped the “Goals” feature in Codex. From the documentation:

“Goals are persistent objectives in Codex that keep a thread working toward a defined outcome across turns. A Goal gives Codex a completion condition: what should be true, how success should be checked, and what constraints must stay intact.”

Also from the docs (emphasis mine):

“A normal prompt says: do this next thing.

A Goal says: keep working until this outcome is true. In a normal request, Codex works through the immediate instruction, reports a result, and waits. With a Goal, Codex has a durable target attached to the thread. After a turn finishes, it can inspect the current evidence and decide whether the objective is satisfied. If the answer is no, and the Goal remains active and within budget, Codex can continue from the latest state.”

A visual representation:

Here’s an example of using a goal in Codex:

/goal Reduce p95 checkout latency below 120 ms on the checkout benchmark while keeping the correctness suite green

That’s a clear enough “end criteria” to just hand off to the agent, which then breaks up the task, spawns subagents, and runs until complete. So, how did OpenAI build Goals? They used files, logs, running tests, and lifecycle controls:

In this way, “Goals” feels awfully similar to a Ralph loop, except compressed into a single command! It feels like the Codex team took the idea of the Ralph loop, built infrastructure around it, took care of coordinating agents by not having them step on one another, dealt with state, running of tests, starting and stopping agents, and then added functionality such as being able to set a budget.

May: Hermes and Claude Code follow with /goal

Three days later (May 2), Hermes agent, a popular AI agent framework and OpenClaw rival, shipped their implementation of /goal. From the docs:

“[/goal] It’s our take on the Ralph loop, directly inspired by Codex CLI 0.128.0’s /goal by Eric Traut (OpenAI). The core idea — keep a goal alive across turns and don’t stop until it’s achieved — is theirs. The implementation here is independent and adapted to Hermes’ architecture.”

Less than two weeks later – on 12 May – Claude Code also shipped their /goal command. It’s identical in what it does to Codex. From Claude Code:

“The /goal command sets a completion condition and Claude keeps working toward it without you prompting each step. After each turn, a small fast model checks whether the condition holds. If not, Claude starts another turn instead of returning control to you. The goal clears automatically once the condition is met.”

A few months before, in March, Claude Code shipped the concept of scheduling an agent with the /loop command. It is basically what JavaScript’s setTimeout() function would be equivalent to: repeat a task after a given interval until the work is done.

By May, running a Ralph loop had become as simple as giving a single command in one of the major agent harnesses. It’s as if AI labs noticed user demand to do more with agentic loops which were hard to set up, and built ways to make it simpler. For instance, in open source agent harnesses like OpenCode, there are plugins like the /goal plugin. For the minimalist coding agent, Pi, the /goal command can be added as a package.

3. Loops which devs use: triggers and cron jobs

By May, we had access to the /goal primitive. So, what use cases were Boris Cherny and Peter Steinberger referring to in terms of spending most of their time on designing loops, instead of writing prompts? I asked around for examples of “loop engineering” from fellow devs. Based on ~210 replies, mostly from X and LinkedIn, it seems that triggers and cron jobs are two very common use cases for loop engineering:

Triggers / automations: an agent kicks off when an event happens. The event could be an error being logged, a new ticket created, customer support feedback received, etc.

Pre-AI, these events were typically triggered by a webhook, and kicked off things like a Slack bot posting in a channel, triggering a system, or being the starting point of a Zapier or an n8n integration.

Cron jobs: many devs see “loop engineering” as kicking off jobs that involve agents on a cadence. This is fundamentally the same as scheduled cron jobs. From director of engineering, Oded Messer:

“The idea is that strategic workflows that are repeatable and automatable can be done so with an agent. OK. But if my strategic workflow is automatable then it either becomes tactical if the AI is capable enough or it’s just a high level old-school-automation I can set up like a cron or a trigger.

The name suggests simplicity and repetition. Sometimes it feels like AI enthusiasts forgot automation was a thing before LLMs.”

4. Helpful loops for devs

Below are some workflows with AI agents that run on a regular basis as a /loop command, or when triggered:

Development-related work

Open PRs for newly recorded app issues. Software engineer Ivan Pantić:

“App encounters a problem and creates a sentry issue. Then:
→ Cron tells agent to check sentry and open PRs.
→If there’s no active PR, agent tackles the issue and creates one
→ If PRs aren’t reviewed, agent pings the devs via slack

In this flow, there is only one PR open at a time.”

Fix flakey tests. Paul D’Ambra, software engineer at PostHog:

“/loop pull the next flakey test from the trunk API. run it to check if it flakes locally, if it does open a PR with the fix... which netted me 13 PRs to stabilise some of our tests.”

Triaging issues and outages. Software engineer Ivan Abad:

“A new alert/exception pops up in a channel.
→ The agent investigates the issue
→ if it is a code change, implements the change
→ creates the PR
→ pings the human for review.

Same applies for customer tickets or incidents. By the time you get paged and connect to the incident the agent already triaged everything and often located the root cause.”

Review design plans. Artem Nikitin, software engineer at Elastic:

“What I’m finding myself doing often recently is to review design/implementation plans in a loop.

Usually, agents only find a few issues during a normal run and then find more on subsequent runs.

So I’m now asking them to run in a loop until they find 0 new major issues.”

Daily/nightly work

Daily product improvements. Jack D, founding engineer at Schematic:

“We have a loop which reads the logs for the last 24 hours, user feedback and makes PR with fixes - we still review the PRs it makes though!”

Nightly end-to-end test run babysitted by an agent. Utku K, engineering manager:

“nightly e2e runs on the frontend app. When a test fails, an agent investigates first to work out whether it’s a real regression or a false negative. If it’s a real bug, the agent attempts a fix, reruns the test, and keeps iterating until it passes or hits a retry cap and escalates... Either way it lands as a PR ready for review by morning.”

More complex development work

Build new telemetry integrations and verify they work. Lawrence Jones, software engineer at Incident.io:

“A lot of the AI runbook that we use to build new telemetry integrations uses loops such as executing the query, verifying that it executed correctly and iterating on the query plan/output format/whatever until it is happy with the outcome.”

Do a long-running migration, mostly autonomously. Startup founder Rafel Mendiola:

“Recently, I converted my startup’s codebase from a regular React app to a React Native app using Expo.

There are two ways I could have done it:

• In the traditional software engineering style, I could have created a large epic with 50-100 different tickets. I started building myself the infrastructure for that, it felt like it was way too much work.

• What I did instead was create a skill that would let an agent figure out a small to medium-sized piece of work or piece of code to convert, given certain detection mechanisms and guidelines, and then do that conversion and keep track of the migration progress. Then I put that skill on a cron job. This was a lot easier to handle cognitively than managing a large migration plan. It also ran every 30 minutes, not nightly or weekly.”

Productivity-related workflows

Daily tasks executed. Aaron Stannard, creator of Akka.NET:

Read more

Today, we cover:

1. Bun’s Rust rewrite with Fable: what can we learn? To a sceptic, spending $165K to migrate Bun from Zig to Rust sounds very expensive. But to a realist, shortening a 1-2 year migration down to 11 days opens amazing new opportunities for devs. However, a thoroughly-tested project is required to pull it off.

2. Anthropic’s Fable, OpenAI’s GPT-5.6 Sol, Cursor’s Grok 4.5, Meta’s Muse. Coding LLM wars heat up: Fable is back, OpenAI releases a comparable GPT-5.6 Sol, Cursor offers cheap & very capable Grok 4.5, and Meta is back with its first truly competitive coding model since Llama 3. But how did Gemini slip out of the top-ranked AI coding models?

3. North Korean hackers keep trying to infiltrate full-remote companies. The founder of a Canadian digital consultancy caught a North Korean dev red-handed, using an AI filter. These events are now so common that it’s hard to trust remote interviewees are who they claim.

4. Industry Pulse. Meta’s key logging exposed sensitive data, massive cuts at Xbox, Meta could not buy enough AI capacity from Google, Qualcomm acquires Modular, and memory price hikes hit Apple products.

1. Bun’s Rust rewrite with Fable: what can we learn?

Last week in San Francisco, I met Jarred Sumner, creator of JavaScript runtime, Bun, and was keen to learn more about the rewrite of Bun from Zig to Rust. But at the time, Jarred didn’t want to say too much, as the tool used for the migration, Fable, was out of action due to the US government imposing export controls.

Fortunately, the situation is now resolved and Fable is available globally, and Jarred has published a detailed post about the project. Before we get into the migration, some context:

Read more

Listen and watch now on YouTube, Spotify, and Apple. See the episode transcript at the top of this page, and timestamps for the episode at the bottom.

Brought to you by our presenting partner, Antithesis.

Verify your system’s correctness by running your whole system in a hostile simulation and finding bugs. I’ve been using Antithesis myself, and I’m impressed with their innovation in building new kinds of debugging tools. Like this:

I show more examples of this neat UI inside the episode, here. You can also check out Antithesis.

In this episode

In this special “ask me anything” episode of Pragmatic Engineer podcast, I am in the hot seat facing questions sent in by subscribers that are read out by guest Volodymyr Giginiak, CTO and cofounder of Wordsmith AI, a legal tech startup (note: I’m an investor).

I tackle your questions on the software industry, AI, hiring, engineering organizations, career growth, the business model of the Pragmatic Engineer, and more. We also discuss where software engineering is headed, and I offer advice on some specific situations. Thanks to everyone who sent questions!

Three stories & observations

Story #1: Without the COVID-19 pandemic, The Pragmatic Engineer might not exist. Prior to the global health crisis in 2020, I had no plans to get serious about writing: I enjoyed blogging on The Pragmatic Engineer blog, but intended to remain an engineering manager or software engineer for the foreseeable future.

But then, COVID-19 happened and Uber made layoffs, which led to a quarter of my team being let go, while the rest of us were disbanded into other teams. It was a tough time, and I decided it was a good moment to exit and finish writing a book I had been working on, ‘The Software Engineer’s Guidebook’. After that project was complete, I planned to try and start a VC-funded startup and build something around platform engineering; possibly a system for tracking RFCs at mid-sized and larger companies.

I gave myself around eight months to finish the book, but when that deadline elapsed, it still wasn’t ready. I did write three other books (‘The Tech Resume Inside-Out’, ‘Building Mobile Apps at Scale’, and ‘Growing as a Mobile Engineer’) and still wasn’t convinced by any startup idea. But I did discover that I like to write!

Story #2: I was on track to publish a damning exposé, until one message from an engineer changed my mind.

During the first year of The Pragmatic Engineer, I wrote about conditions for engineers at the Dutch neobank Bunq, based on accounts from disgruntled employees. I had a final draft ready, which I sent to the company to provide a right of reply ahead of publication. Then I received a message.

It was sent by an engineer originally from the Middle East. They told me they had really wanted to break into the European tech industry, but that no company would sponsor their visa, except Bunq. Yes, the company was a tough place to work at as a dev – and this engineer subsequently left for another opportunity – but they appreciated Bunq for taking a chance on them that enabled them to move to Amsterdam and learn how to build fintech with a small team. This engineer now works at Meta and attributes their success to the break Bunq provided.

Based on that interaction, I opted not to publish the article, and it also led me to adopt a new editorial policy that I have followed since: I write about what works inside companies, instead of focusing on what seems to be broken.

Story #3: Being called a “nobody” by a CEO led to my sole investigative piece, which uncovered some pretty interesting details. After I briefly noted Pollen’s poorly-handled layoffs, CEO Callum-Negus Fancey dismissed my report during a company all-hands, and compared The Pragmatic Engineer unfavorably to the BBC as just some minor publication with an agenda against Pollen (why I would have an anti-Pollen bias isn’t clear to me!) To be honest, I took it personally when I heard a recording of this sent to me by employees there, and so started digging around.

I discovered unpaid salaries, silently cancelled health insurance in the US, and the CTO deliberately triggering a $3.2M double charge to customers and never publishing a postmortem, despite engineers requesting one.

It was certainly something, and I published the findings in the article Inside Pollen’s Collapse: “$200M Raised” but Staff Unpaid - Exclusive. To ensure the CEO saw the report on a platform he deemed worthy, I also contributed to the BBC’s documentary: Crashed: $800M Festival Fail, aired in the UK during prime time. By doing all this, I also learned that investigative journalism is just not for me.

In a strange turn of events, someone at Pollen evidently wants my original article to disappear from Google’s search results, and filed bogus DMCA takedown notices a few weeks ago. Well, it’s having the opposite effect!

Opinion #1: I believe LeetCode-style interviews will stay because they self-select tolerance of corporate nonsense. The existence of data structures and algorithm (DSA) interviews is a bit of a head scratcher because these skills are rarely used at work. However, a candidate who’s willing to grind for weeks or months to prepare for an interview which bears little resemblance to the job, is likely to be someone who understands that sometimes it’s necessary to do pointless work.

This suggests they’ll probably have a much better time in Big Tech than someone who refuses to engage with meaningless tasks. It’s one reason I’ve observed for companies retaining LeetCode-style interviews. Of course, AI solves the puzzles with ease these days, and I expect larger companies to move back to in-person interviewing – all while keeping DSA interview questions.

Opinion #2: MCP became industry standard partly because Anthropic wasn’t a threat – but it couldn’t pull this off today. When MCP launched in November 2024, Anthropic wasn’t yet considered the leading AI lab. GPT-4o was seen as the top-performing multimodal model, followed by Claude 3.5 Sonnet and Gemini 1.5 Pro. At that point, Claude 3.5 Sonnet was seen as the best coding model, but it wasn’t understood how advantageous being good at coding would be for AI in general.

Therefore, OpenAI, Google, Microsoft, and other players could adopt MCP without fear of lock-in, as it came from a promising, but not a dominant lab. When Google launched its Agent2Agent protocol a few months later, no major lab adopted it due to concerns about Google’s dominant position. Today, Anthropic is the leading frontier lab and I reckon this would discourage adoption if MCP was launched in the present climate, for the same reason as nobody adopted the Agent2Agent protocol.

My answer to subscribers with questions about how to create standards is that I see them as emerging somewhat coincidentally, following a technically strong approach and with the right external conditions in place, which are impossible to entirely predict or control.

Opinion #3: My hot AI take: AI doesn’t make work easier, and be mindful of skill atrophy. If you’re using AI and life seems to be getting a lot easier, it raises the question: are you trying hard enough? Personally, using AI forces me to think just as hard, or even harder than before. I choose to use zero AI in my writing for the Pragmatic Engineer, and Grammarly is turned off as well. This is because I don’t want my writing skill to degrade, and would like to keep improving. On the other hand, with coding, I do use AI and accept my hand-coding ability will unavoidably degrade. My tip is to be mindful of the tradeoffs inherent in AI, and to keep using those skills which you value and want to keep sharp, even when AI tools are available.

The Pragmatic Engineer deepdives relevant for this episode

• State of the software engineering job market in 2026

• The impact of AI on software engineers in 2026: key trends.

• How 10 tech companies choose the next generation of dev tools

• The reality of tech interviews

Timestamps

00:00 Intro

01:56 From Uber to writing

09:22 AI-native SDLC

14:00 AI and hiring

19:06 Engineers currently thriving

22:18 Junior roles

24:44 Meta’s war mode

27:54 AI at Big Tech vs. startups

36:46 Tech debt

41:36 Types of engineering managers

44:40 Measuring AI productivity

48:30 The value of CS degrees

50:53 AI at Pragmatic Engineer

56:09 Future-proofing your career

1:01:36 The EU job market

1:03:55 Making money as a creator

1:08:20 What’s next for The Pragmatic Engineer

1:09:27 Bunq and Pollen

1:13:38 Spotting trends

1:14:33 Book updates

1:15:20 Favorite books & tech products

1:17:13 What won’t change in engineering

References

Where to find Gergely Orosz:

• X: https://x.com/GergelyOrosz

• LinkedIn: https://www.linkedin.com/in/gergelyorosz/

• Bluesky: https://bsky.app/profile/gergely.pragmaticengineer.com

• Newsletter and blog: https://www.pragmaticengineer.com/

Where to find Volodymyr Giginiak:

• LinkedIn: https://www.linkedin.com/in/giginiak

• Newsletter:

The Legal Engineer

The journey of an engineer building AI for lawyers

By Volodymyr Giginiak

Mentions during the episode:

• Wordsmith: https://www.wordsmith.ai

• Uber: https://www.uber.com

• Lenny’s Newsletter:

Lenny's Newsletter

Deeply researched product, growth, and career advice for product leaders, founders, and ambitious builders.

By Lenny Rachitsky

• Waterfall methodology: https://www.atlassian.com/agile/project-management/waterfall-methodology

• Agile: https://www.atlassian.com/agile

• How Kent Beck shapes the software engineering industry:

• Building Claude Code with Boris Cherny: https://newsletter.pragmaticengineer.com/p/building-claude-code-with-boris-cherny

• How Claude Code is built: https://newsletter.pragmaticengineer.com/p/how-claude-code-is-built

• How AI is changing software engineering at Shopify with Farhan Thawar: https://newsletter.pragmaticengineer.com/p/how-ai-is-changing-software-engineering

• Linear: https://linear.app

• Inside Linear’s Engineering Culture: https://newsletter.pragmaticengineer.com/p/linear

• Linear: move fast with little process (with first engineering manager Sabin Roman): https://newsletter.pragmaticengineer.com/p/linear-move-fast-with-little-process

• Inside Meta’s Engineering Culture: Part 1: https://newsletter.pragmaticengineer.com/p/facebook

• Inside Meta’s Engineering Culture: Part 2: https://newsletter.pragmaticengineer.com/p/facebook-2

• Stacked diffs and tooling at Meta with Tomas Reimers: https://newsletter.pragmaticengineer.com/p/stacked-diffs-and-tooling-at-meta

• Chaos Monkeys: Obscene Fortune and Random Failure in Silicon Valley: https://www.amazon.com/dp/0062458191

• Gemini: https://gemini.google.com/app

• Ramp: https://ramp.com

• Intercom: https://www.intercom.com

• Block: https://block.xyz

• Coinbase: https://www.coinbase.com

• Why Rust is different, with Alice Ryhl: https://newsletter.pragmaticengineer.com/p/why-rust-is-different-with-alice

• Building a best-selling game with a tiny team – with Jonas Tyroller: https://newsletter.pragmaticengineer.com/p/thronefall

• Bunq: https://www.bunq.com

• Inside Pollen’s Collapse: “$200M Raised” but Staff Unpaid - Exclusive: https://blog.pragmaticengineer.com/pollen

• A Philosophy of Software Design: https://www.amazon.com/dp/1732102201

• Tidy First?: A Personal Exercise in Empirical Software Design:

https://www.amazon.com/Tidy-First-Personal-Exercise-Empirical/dp/1098151240

• Granola: https://www.granola.ai

• Perplexity Deep Research: https://www.perplexity.ai/hub/blog/introducing-perplexity-deep-research

—

Production and marketing by Pen Name.

For this final part of our series on the tech hiring market during the first half of 2026, I spoke with more than 50 hiring managers, software engineers, and engineering leaders. Thank you to everyone who contributed!

For more on this topic, check out our analysis of what the data says about the market in Part 1 and Part 2 of this mini-series.

Overall, an appropriate description of the employment market as many folks experience it right now would be “weird”. This is a characteristic that’s not easy to see in the data, but is clear from talking to people and hearing their anecdotal, personal accounts of job hunting this year. I think the data in the first two articles of this series failed to capture just how unusual things are. So, in today’s issue, we attempt to shed some light on the weirdness, covering:

1. “Catch-22:” nobody finds each other. Hiring managers struggle to find experienced folks, who barely get any replies when applying for jobs. How’s that work?

2. No trust. Is AI to blame? AI-enhanced resumes read as incredible, but hiring managers often face disappointment. Some places don’t bother reading inbound applications as a result.

3. Hot market for some, but tough for most. For those in AI Engineering, ML, or FDE, the market is incredible. For everyone else, it’s much less great.

4. Higher hiring bar & lower compensation – but not for everyone. Many candidates are unhappy with offers that are the lowest in years. This doesn’t apply to AI Engineering positions or at AI businesses, however.

5. Engineering leader recruitment: also weird for senior ICs. Senior engineering leaders are struggling to find opportunities, or may turn them down in favor of fractional roles or to work on their own startup.

6. US market trends. Folks experiencing the “best market ever” are likely in the US, where a talent shortage is a bigger complaint than it is elsewhere.

7. Trends in the UK, EU, and rest of the world. “Ghosting” is more commonplace than in the US, “fake applicants” a bigger issue, remote roles are going extinct, and more.

For more details on the hiring market, see also:

Part 1: what the data says:

• Software engineering recruitment: trending up, mostly

• Big Tech and publicly-traded companies

• Who’s hiring the most software engineers?

• AI engineering: explosive demand

• Who’s hiring the most AI engineers?

• Is AI engineering replacing software engineering hiring?

Part 2, what the data says, continuied:

• Top AI labs are now more attractive than Big Tech

• Harder for new grads & interns to get hired

• Mobile and frontend demand drops, AI & FDE surges

• AI engineering comp > software engineering comp

• Management’s “great flattening” continues

• Big Tech seniority & tenure keep rising

• Interview preparation signups: what do they indicate?

• Where engineers go after Big Tech

1. “Catch-22:” nobody finds each other

The phrase “Catch 22” refers to a paradoxical problem, whose solution is blocked by the problem itself. The term originated in a famous World War II novel of the same name, and it also describes pretty accurately what I see in today’s tech job market. Hiring managers are saying that highly-skilled talent (typically senior+ engineers) is not available to be recruited, at the same time as experienced, proven professionals find their applications ignored by employers.

What seems paradoxical here is how both can be true. It’s as if recruiters and potential candidates aren’t hearing each other. Of course, there’s some nuance:

Mike Julian, CEO of DuckBill Group, which is hiring software engineers, replied to my post with this observation:

“We get about 1,000 applications a day on inbound and maybe two of them are even relevant to the posting.

I mostly no longer look at inbound seriously because it’s so c***. I’d almost certainly miss a great inbound submission if it came in.

All of our recent hires have been via network and us reaching out to folk on LinkedIn. Biggest hurdle we have to outreach is thin LinkedIn profiles and little other online presence.”

From the other hiring managers we talked with, more themes emerged:

More inbounds than ever & also more noise

“I’ve never seen so many inbounds and strong resumes. I’m hiring for lots of roles; for one software engineering position in Seattle, we have had 800 resumes inbound over a three-month period. I’ve never seen anything like this! These resumes are not low-quality either: they are people who have worked at MSFT, AWS, other large tech companies, and have solid skills.” – Head of Engineering, Series B startup, Seattle, US

“It’s difficult to find good candidates among all the noise. Have had two open senior engineer positions for months, and the only good interviews or offers we’ve extended were in network.” – Engineering Manager, late-stage startup, US

“A glut of vastly underqualified people completely drowning your hiring pipeline. This is what I’m seeing, and it’s making inbound a useless channel.” – Engineering Director, Big Tech, US

“From what I hear from recruiters, every job posted has 1,000+ applicants, and 98% of them are considered unqualified.” – fullstack engineer, Middle East

“We are a small company and gave up on inbound hiring. We got too many AI applications. Much lower signal-to-noise compared to past experience.” – Director of Engineering, Canada

Experienced engineers struggle to get interviews

“Even when you’re good, it feels like there’s a LOT of noise to cut through to get noticed. I’m quietly looking for a new role in devtools, and fit that ‘product engineer’ profile perfectly. I feel like the demand for great people is higher than ever, but I can’t figure out where on that bar I fall.” – Tech lead, seed-stage startup, US

“As an experienced engineer & successful-ish founder I have yet to get to a phone screening in this job market. There is clearly something amiss, just a wall of noise preventing any signal from getting through.” – Software engineer + founder, 8 years experience, US

A staff engineer at a late-stage startup in the US summed things up:

“It feels like everyone who has a good job is holding onto it for dear life, and THOSE are the people we want to hire.”

“Tale of two cities:” in demand or not

A director of engineering at a Big Tech in the US, identifies two distinct groups in the market:

“If you are at the top of your game, have AI experience, and are senior enough, you can write your own ticket. If not, then the job market is tough! This job market is like the tale of these two very different cities.”

Observations from some job seekers echo this: demand feels strong for “AI-adjacent” engineers (those building AI systems), especially in the US. Standout engineers with a strong network are still in demand, and we covered the story of one such person in “How to be a 10x engineer” – interview with a standout dev. For everyone else, it’s a struggle to get an interview!

Referrals: more important than ever?

Several experienced engineers currently on the job market say they only get interviews when they personally know someone at a company:

“Larger companies have to have some serious AI going on to sift through resumes because they are being bombarded. It really is the case that you don’t get your resume seen unless you know people who can vouch for you.” – Technical Program Manager, Big Tech, US

“Referrals are a lifeline. It’s impossible to get interviews for Staff or Principal Eng positions by cold applying. The only interviews I am getting from a cold-apply are Senior-level roles.” – Principal Engineer, 10 YOE, US

From a couple of hiring managers:

“The only interviews or offers we’ve extended were to devs already in our network.” – Software architect at a mid-sized company, US

“I filled an engineering manager position in record time thanks to a person in my network looking for this job. If it would not have been for my network I would probably have been looking for someone for a couple of months.” – VP of Engineering, private equity-funded company, Germany

2. No trust. Is AI to blame?

Hiring managers repeatedly tell us they no longer trust what they read in resumes, and that some candidates even turn out to be fake.

Polished CVs, weak candidates

A common gripe among hiring managers is that resumes are highly optimized by AI, but candidates turn out not to have the experience they claim:

“CVs are high-quality, but the people behind them are not. Almost every resume looks impressive. However, the quality of the conversations does not match it at all.

One recent example: I interviewed a senior candidate who had spent five years at a US-based cloud consulting company, most recently as an architect. I asked which architectural principles or patterns he had used in his projects. His answer was: “Daily standup, sprint planning, and retrospective.” I clarified that I meant from a tech perspective, not process perspective. He confidently replied: “Yes, daily standup, sprint planning, and retrospective.” – Engineering manager, large company, Berlin, Germany

“Ten years ago, we would have appreciated resumes tailored to a role that was posted, now it’s just lazily thrown together with AI.” – Engineering Manager, mid-sized company, Canada

AI-keyword stuffing is rampant, according to one head of engineering in the UK:

“Lots of people are rebranding themselves as senior AI Engineers and demanding much higher salaries. Their resumes now have lots of AI-related keywords mentioned, like RAG, evals, inference… but when digging deeper there is little substance. Many of them are seeking a senior level salary (£90k–£140k) when they are barely showcasing mid-level skills.”

Cover letters are as good as dead, several hiring managers tell us. The reason is that they’re always AI-generated, boring to read, and pointless. More than 18 months ago, we reported on how cover letters were being made redundant by AI in How GenAI is reshaping tech hiring.

An engineering manager at a UK e-commerce agency summarizes what’s happening:

“’Claude; write me a CV that matches this job spec, then auto send’. This seems like the name of the game for most applicants.”

Fake candidates

Does anything encapsulate the challenges of today’s job market for recruiters more succinctly than candidates who do not exist, even when they appear to be sitting in an interview? A year ago, we covered an “AI faker” applicant caught by a security startup, who was potentially a state agent from North Korea.

Such incidents are becoming more common at US, UK and EU companies which hire for remote roles:

• “There are a lot more fake candidates applying, leveraging AI for not just resumes, but also interviews. In extreme cases, the interviews are being outsourced so that a different person shows up for the interview. It feels a bit like playing captcha with them during interviews.” – Senior EM, private-equity backed company, Bay Area, US

• “The second person we interviewed was clearly a North Korean scammer, writing questions into an LLM, reading the response, easily tripped up, and other interviews were background noise in the room.” – Staff Engineer, UK

• “Many applicants that looked a good fit turned out to be someone else in the Asia Pacific region doing interviews with an AI in the background. It’s easy to spot because of the ‘lag’ in a naturally flowing conversation.” – head of engineering, mid-sized company, Germany

Cheating in remote interviews by using AI is also commonplace. A software engineer based in Finland told us:

“A couple of times we saw an applicant who was using AI to answer the questions, and this was sooo weird. I honestly didn’t know how to react, so we decided to cut the interview short. We then shared a note with the recruiters on how to spot this.”

3. Hot market for some, but tough for most

The market appears sharply bifurcated: amazing for AI and a few specialist roles, and a major struggle for everyone else. Some accounts from folks benefitting in the current conditions:

“The market is really good as an (AI) engineer with the right experience. I’m not actively looking but get 2–3 messages a day. When I was hiring for AI engineers at my last AI startup, the market for these folks was terrible. It’s very hard to find good talent, and people that were great on paper did poorly in interviews.” – AI Engineer, 5 YOE, New York, US

“It seems like demand for senior positions is still there, but only if your profile really matches what the company needs. For example, I’ve been working on different data pipelines for a while, and finding such a position is relatively simple. However, breaking into anything else is not! Every startup wants to hire only people experienced in that particular thing they need.” – Software Engineer, 13 YOE, Big Tech, UK

Here’s what two job seekers who are struggling at present say:

“I sent my handcrafted resume to 30–40 positions, and heard back from zero. Eventually, I got an interview after a recruiter reached out via LinkedIn; I don’t know how I would’ve found anything, if it was not for this!” – Software engineer, 5 YOE, Amsterdam, Netherlands

“As a developer without a specific “specialization,” I’m struggling to get any interviews. I’ve spent a few years as frontend, then another few as backend, and most recently working on DevEx. In this market, I just don’t get any callbacks for interviews, not even a first round interview.” – Software engineer, 6 YOE, London, UK

AI/ML/FDE market on fire

We see in the data that AI engineering is seeing explosive demand, and that forward deployed engineers (FDEs) are also seeing a massive spike in demand. For more detail on this, check our deepdive on what FDEs are, and why they’re hot right now.

Anecdotal evidence from job seekers suggests that being in these fields means the market is as good as it gets, right now:

“It’s the greatest job market I’ve ever seen. I’m an L5 former FDE, now SWE, who has worked on LLM apps for ~2 years. The inbound top of the funnel is bonkers, and I find myself saying “no” to places that I would have once killed to work at.” – AI Engineer at an AI decacorn, San Francisco, US

“I have seen huge interest in FDEs and AI Engineers as an engineer who’s been interviewing.” – Software Engineer, 5 YOE, New York, US

“It’s not hard to get interviews as an ML/AI engineer. The technical bar is about the same as late 2022. If anything, onsites seem to be of fewer rounds for the same level. Companies seem to move fast to get candidates onsite.” – ML engineer, ex-Meta, Bay Area, US

An engineer at Apple is surprised by the demand for their skills from AI companies:

“Got a lot of responses from cold applications for AI roles, and ended up with two offers at AI infra companies. I ended up getting a significant pay bump beyond what Apple offers for the same level!”

A software engineer at well-known company in the Bay Area, also finds the job market is good:

“I got offers for Senior SWE with Stripe and Rippling, was rejected from Snowflake… Anecdotally, the job market is better compared to 2025 for regular SWE positions in the Valley.”

EM and Staff+ profiles are near-impossible to fill

A repeat complaint from hiring managers in the US is how challenging it is to recruit solid engineering managers and Staff+ engineers:

“It’s extremely difficult to hire EMs and Staff+ engineers. It’s much easier to hire folks with less than 10 years of experience. This is despite us offering 90th percentile comp via Pave, and having a hybrid and good culture.” – Fractional VP of Engineering, Series B, New York, US

“My team has been looking for a new engineering manager for three months and we barely had any good applicants. It’s a super weird dynamic right now.” – Senior Infrastructure Engineer, Series D Fintech, company with offices in the US and EU

“The most desirable candidates right now are EM or staff-level folks who can keep up with uncertainty in the business. We are having trouble hiring folks who roll with the constant change that is now very typical at startups.” – CTO at a San Francisco-based startup

“Specialist” profiles are hard to recruit

A few examples:

• Distributed systems engineers: “We can’t find enough qualified distributed systems engineers, cloud infra etc. I have 20 reqs [open positions] right now.” – Recruiter at a hyperscaler, US

• Product engineers: “‘Product engineer’ has been a hard profile to find. It is also hard to find someone with a decent design eye who can also build full stack. The hardest thing to hire for has been taste + trust… I’d rather hire someone who is ‘behind’ on AI, but has great taste/judgment than someone with complex agent setups and prompt libraries.” – Tech Lead, seed-stage startup, Los Angeles, US

• Senior engineers, in general: “As a hiring manager trying to hire seniors right now, it has felt pretty difficult. Most applicants are totally unqualified, and the qualified ones do poorly in an interview.” – Engineering manager, robotics company, Canada

Silence for many

One in five respondents detail how difficult it has been to get responses from recruiters:

“Things are looking bleak. Now, applications are going into a void. Recruiters aren’t reaching out immediately as before. Of 10 or so applications sent out, I’ve gotten one interview scheduled that was then rescinded because it looks like the team was laid off.” – Frontend engineer, 10+ YOE, Southeast US

“The job market has been a desert. I’ve gotten back into applying, job agencies seem to have few roles they’re looking to fill. The companies I have applied to myself, never reply.” – Software Engineer, 8 YOE, Southern California

“Recruiters are ghosting me! Every time when I respond to recruiter reachouts, they ALL ghosted me after a few messages and have similar stories from friends.” – Software Engineer, 7 YOE, ex-Meta, Switzerland

“So far, I have a hard time even getting past the hiring manager and I’m rejected before I can do the proper technical assessment.” – Software Engineer, 7 YOE, Helsinki, Finland

Even folks at Big Tech face a lack of response. Here’s a Technical Program Manager currently at Meta who used to be very in-demand:

“I put a few feelers out when Meta announced layoffs, and it’s just radio silence. I don’t want to sound cocky, but with my resume, you at least get a chat with a recruiter, usually right away. But Google and Anthropic just ghosted me.”

4. Higher hiring bar & lower compensation – but not for everyone

In a “normal” market, when the hiring bar goes up, so does compensation. But we heard anecdotes about the hiring bar going up, with the compensation on offer trending down!

Read more

Listen and watch now on YouTube, Spotify, and Apple. See the episode transcript at the top of this page, and timestamps for the episode at the bottom.

Brought to You by

• Antithesis. If you’re using agents to code, the problem isn’t writing the code but making sure that nothing broke. Antithesis runs your whole system in a hostile environment and identifies hard-to-find bugs before users hit them in production. Teams like Jane Street, Fly.io, and the etcd community use agents safely and ship better code, faster, with Antithesis. Learn more.

• WorkOS – make your app and agents Enterprise Ready, with SSO, SCIM, RBAC, and more. Get started.

• turbopuffer. A vector and full-text search engine built on object storage. It’s fast, cheap, and extremely scalable. It’s never been better to try it out: last week, they dropped their base price from $64 per month to $16 per month. Give it a spin now.

In this episode

Few have made as big an impact on software engineering as this week’s guest on the Pragmatic Engineer podcast, Kent Beck. He created Extreme Programming, pioneered test-driven development (TDD), co-created JUnit, and is one of the authors of the famous ‘Agile Manifesto’. But these days, he’s re-examining many ideas for the age of AI, and says we’re failing to accumulate trust during this new era at the same high rate as new code is being accumulated.

Longtime readers will remember how Kent and I bonded over our co-authored article in our response to McKinsey on measuring engineering productivity, and Kent was on the podcast a year ago, talking about TDD and AI.

Today’s episode is the previously untold story of Kent’s remarkable career and how he became the industry legend he is today. We start with Kent’s journey from discovering Smalltalk in the early days of personal computing, to helping define modern software engineering practices. We explore the origins of TDD, design patterns, Extreme Programming, and Agile – along with some lessons learned at Apple and Facebook.

Kent explains why he believes software engineering is about far more than writing code, why no one yet knows exactly how engineers should work alongside AI agents, and how his “explore, expand, extract” framework can help engineers navigate major technology shifts.

Key observations from Kent

Here are 12 rarely-told stories and observations from Kent:

1. Coding is only a small part of software engineering – the rest can’t be automated. Kent rebuts the claim that coding – and eventually the whole software engineering craft – will vanish. He believes coding is only part of what we do, and a small part of it, too. Through your work, you also build confidence, make connections with other people, and develop your personal understanding of the domain. This will remain important, even if we develop our skills in other areas than coding.

2. Lesser-known story: fired by Apple. Kent joined the tech giant in 1987, drawn in by Smalltalk, whose language was taking off at the time. Xerox had handed the language to a few companies to see who could run with it, and Apple was one. But Apple’s customers wanted C and Pascal compilers instead, so the Smalltalk project went nowhere. Kent joined a team building a programming language for kids, but says he was eventually fired because he was in “punk mode” and wanted to do things his own way, instead of being a team player.

3. Kent used to keep a thesaurus close at hand. When working at Tektronix in the late 1980s with Ward Cunningham (the creator of the first wiki, and a pioneer in design patterns), they built HotDraw, a graphical editor with a boxes-and-arrows model. For naming core abstractions, Beck chose “drawing object,” “drawing handle,” etc. But Ward cared intensely about nomenclature and hunted for better vocabulary, so they kept a physical thesaurus which was well thumbed. Beck recalls they became pretty obsessed about the names of things.

4. TDD wasn’t ‘invented’ so much as ‘rediscovered’ by Kent. As a kid, Kent read one of his father’s programming books from the tape-to-tape era, when an input tape like time cards was fed through a payroll program to produce an output tape, such as one with checks. The book’s advice was to take a real input tape and manually type the expected output tape before writing the program. He read this, didn’t understand it, and forgot about it.

Years later, Kent built SUnit, a small testing framework, and randomly remembered the input-tape trick, so mapped it onto SUnit. If he followed the pattern, he’d write the test before the code. He laughed out loud at this because it seemed like such a stupid idea: why write a test that’s guaranteed to fail, when the classes and methods aren’t even defined yet? But when he did, he found his anxiety about programming vanished. This is when he became a TDD convert.

5. Kent invented Extreme Programming (XP) while on the Chrysler payroll project. Kent threw away a codebase that didn’t work and restarted the project with a new methodology. He paired with others, and used his own ideas for testing. Later, he coined the new methodology’s name by deliberately picking one that he knew would be unpopular with the tech establishment of the day: “extreme programming” was born.

6. The Agile Manifesto came together in a messy way. In 2001, a loose group of folks who rejected waterfall development gathered at Snowbird, Utah, to rethink programming. Kent recalls this summit proceeded badly as everyone pushed contradictory ideas. During a break, Martin Fowler and Jim Highsmith stayed behind, and when the others returned, they found the values written on the whiteboard. Kent’s contribution was the word “daily”: “Business people and developers must work together daily throughout the project.”

7. Calling it “agile” was an error. Kent objected to the word “agile” at the time, and still does today, since nobody claims they prefer “rigid” development, and everyone says they’re “agile”, even when they’re not. He would’ve preferred a less spacious term, like with “extreme programming”: after all, it’s hard to call yourself an “extreme programmer” without actually following that methodology.

8. The Dotcom Bust hit very hard. The day before the 9/11 terror attack in 2001, Kent had eight months of consulting work booked at top rates and was finishing work on a house in rural Oregon. The next day, everything was canceled, just as big bills fell due. He burned out into depression and was left unable to program for years, in what was a “lost decade.”

9. At 50, Kent joined Facebook and realized nobody cared about testing or TDD. At Facebook, Kent found a company that barely did any form of unit testing, while running a massive, stable, and fast-growing site. He signed up to teach a TDD class at a hackathon — he wrote the book, after all! The classes either side of his in the schedule both filled up, but the TDD class got zero signups, not even a pity one. He made the decision to forget everything he knew and to relearn software engineering as it was at Facebook. In the end, he stayed seven years.

10. Building software products has three phases: explore, expand, extract. This is Kent’s “3X” model. ‘Explore’ means trying many cheap uncorrelated experiments, ‘expand’ involves focusing on the one thing that’s working and overcoming obstacle after obstacle, while ‘extract’ is a repeatable playbook and economies of scale. How you code, hire, and organize differs across each phase.

11. Kent has always been an anxious programmer. He describes himself as chronically anxious because the more complex the code is, the more he knows it could break. This was the fuel behind testing and TDD, which are approaches designed to soothe an anxious mind.

12. Kent sees himself as a “tree shaker, not a jelly maker.” He starts things like patterns, SUnit, JUnit, TDD, XP, 3X, then pushes them until they take off, before moving on to the next thing. It’s his defining trait, and may explain his enormous output, and also why he abandoned TDD just as it peaked.

+1: The human part is the most important one in software engineering. As Kent explained:

“This is the biggest cosmic, practical joke ever. As young people, we were promised: “Okay, here’s this computer and once you’ve completely understand this computer, you’ll be fine. That’s all you need to do.”

So I set out the first part of my career just to become the best programmer that I could be because that’s what it would take to be successful. And then you realize: sorry, there’s this whole human side. Your ability to affect change in the world is gated by your ability to communicate with, to soothe, to understand other human beings. And those are exactly the skills that I thought I didn’t need to learn!

So I was promised: just understand the computer and you’ll be successful. And then someone went “just kidding, understand people!” And now I was in a position of being ten years behind.”

The Pragmatic Engineer deepdives relevant for this episode

• Measuring developer productivity? A response to McKinsey – co-written with Kent Beck

• TDD, AI agents and coding with Kent Beck

• Paying down tech debt

• The past and future of modern backend practices

Timestamps

00:00 Intro

03:47 Human engineers aren’t going away

08:00 Kent’s path into tech

13:50 Undergraduate and graduate studies

17:21 Kent’s first programming job

18:54 The rise and fall of Smalltalk

27:04 Working with Ward Cunningham

37:36 Design patterns

44:05 Working at Apple

51:08 CRC Cards

59:29 Testing tools in the language

1:04:22 The C3 project with Martin Fowler

1:09:54 Extreme Programming

1:16:25 Developing TDD

1:25:07 Writing the Agile Manifesto

1:30:00 Agile’s impact

1:32:40 Agile’s downside

1:37:32 The Dotcom Bust

1:44:30 Lessons from working at Facebook

1:59:44 Kent’s ‘Good to Great’ program at Facebook

2:06:07 Soft skills engineers need to learn

2:09:30 AI and the challenges of acceleration

2:15:53 Explore, expand, extract

2:22:33 What Kent is excited about

References

Where to find Kent Beck:

• X: https://x.com/kentbeck

• LinkedIn: https://www.linkedin.com/in/kentbeck

• Website: https://kentbeck.com

• GitHub: https://github.com/kentbeck

• Newsletter:

Mentions during the episode:

• TDD, AI agents and coding with Kent Beck: https://newsletter.pragmaticengineer.com/p/tdd-ai-agents-and-coding-with-kent

• Anthropic CEO Predicts AI Will End Coding and Software Engineering:

• Extreme Programming Explained: Embrace Change: https://www.amazon.com/Extreme-Programming-Explained-Embrace-Change/dp/0321278658

• Tidy First?: A Personal Exercise in Empirical Software Design: https://www.amazon.com/Tidy-First-Personal-Exercise-Empirical/dp/1098151240

• Test Driven Development: By Example: https://www.amazon.com/Test-Driven-Development-Kent-Beck/dp/0321146530

• Tektronics: https://www.tek.com/

• Ward Cunningham on LinkedIn: https://www.linkedin.com/in/wardcunningham

• Design Principles Behind Smalltalk: https://www.cs.virginia.edu/~evans/cs655/readings/smalltalk.html

• Kotlin: https://kotlinlang.org

• Swift: https://www.swift.org

• Prolog: https://en.wikipedia.org/wiki/Prolog

• The Timeless Way of Building: https://www.amazon.com/Timeless-Way-Building-Christopher-Alexander/dp/0195024028

• Notes on the Synthesis of Form: https://www.amazon.com/Notes-Synthesis-Form-Harvard-Paperbacks/dp/0674627512

• Larry Tesler: https://en.wikipedia.org/wiki/Larry_Tesler

• PARC: https://en.wikipedia.org/wiki/PARC_(company)

• August 1981 issue of Byte featuring Smalltalk: https://vintageapple.org/byte/pdf/198108_Byte_Magazine_Vol_06-08_Smalltalk.pdf

• Class-responsibility-collaboration (CRC) cards: https://en.wikipedia.org/wiki/Class-responsibility-collaboration_card

• MasPar: https://en.wikipedia.org/wiki/MasPar

• Cray: https://en.wikipedia.org/wiki/Cray

• JUnit: https://en.wikipedia.org/wiki/JUnit

• Erich Gamma: https://en.wikipedia.org/wiki/Erich_Gamma

• OOPSLA: https://en.wikipedia.org/wiki/OOPSLA

• C3: https://www.martinfowler.com/bliki/C3.html

• How AI will change software engineering – with Martin Fowler: https://newsletter.pragmaticengineer.com/p/martin-fowler

• Cycles of disruption in the tech industry: with software pioneers Kent Beck & Martin Fowler: https://newsletter.pragmaticengineer.com/p/cycles-of-disruption-in-the-tech

• The third golden age of software engineering – thanks to AI, with Grady Booch: https://newsletter.pragmaticengineer.com/p/the-third-golden-age-of-software

• Ivar Jacobson’s website: https://www.ivarjacobson.com

• James Rumbaugh: https://en.wikipedia.org/wiki/James_Rumbaugh

• Ron Jeffries’ website: https://ronjeffries.com

• The Agile Manifesto: https://agilealliance.org/agile101/the-agile-manifesto

• Jim Highsmith on LinkedIn: https://www.linkedin.com/in/jhighsmith

• Gusto: https://gusto.com

—

Production and marketing by Pen Name.

In recent days, I’ve visited the offices of OpenAI, Anthropic, and Cursor, in San Francisco. Onsite, I talked with software folks working on the model side to learn more about how their way of building software is changing. This article is based on observations from those visits, including some new developments that I reckon may be adopted industry-wide.

We cover:

1. Next mega-trend? Agents running in the cloud to go mainstream. OpenAI, Anthropic, and Cursor are all-in on cloud agents and expect demand for them to increase massively.

2. Mass adoption of coding harnesses by non-developers. At OpenAI, more than 95% of non-engineers use Codex, not ChatGPT. Is it a sign of things to come across tech?

3. Will the main task of engineers be to make agents more efficient? Ever more engineering work is about building environments for agents to execute more efficiently at Anthropic and Cursor.

4. Next trend? Companies aggressively optimize spend-per-token. AI spending by software engineers is so high that it makes sense for platform teams to slash per-token cost. A case study from Coinbase.

1. Next mega-trend? Agents running in the cloud to go mainstream

Last week, Andrej Karpathy employed the phrase “new paradigm” to describe using Claude Tag – a way to mention Claude in Slack and have it kick off tasks – to work with AI:

There was plenty of pushback against this claim on social media; after all, it’s just a Slack integration with Claude, right? I also thought this until I asked David Hershey at Anthropic’s Applied AI unit about it while visiting the company’s offices. He explained in detail what makes this particular Slack integration different from using something like Claude Code:

• No additional setup. For Claude Code to work well, it should be connected to internal MCP servers, with the right skills on your local machine. Of course, at larger companies this setup is at least partially automated, but devs often need to do tweaking.

• No “tool context-switching.” Just mention it in Slack! Of course, opening Claude Code is not a big effort, but it’s still more work than just typing it out in Slack, and kicking off work.

• Routine work made easier. David has “Claude playing Pokémon” as his side project. Every time a new model comes out, he kicks off a run of his script on it. Previously, this took a few minutes to set up every time, and then it ran on his machine for hours. With this new Slack integration, it’s just one command.

My sense is that the excitement here is less about the Slack integration itself, and more to do with the fact that it’s easy to kick off one or more AIs that no longer run on a local machine. You can skip the setup entirely.

‘Claude Managed Agents’ is a big focus at Anthropic. While there, I met Katelyn Lesse, head of engineering for Claude Platform, who explained that Claude Managed Agents is a large, complex project which her team built over a six-month period. It’s a hosted service to execute long-running agents on various cloud providers.

Cloud agents are the “big deal”, not the Slack integration

Also last week, I had the opportunity to attend a private AI builders event, where Peter Steinberger discussed his workflow.

He talked about how he has gotten really tired of having several OpenClaw agents running on his local machine, which heat up the CPU and slow down his whole system. So, he built Crabbox as a way to run OpenClaw agents in the cloud:

Suddenly, the same solution of cloud agents has emerged in separate places – at Anthropic and with Peter’s OpenClaw – in response to issues caused by locally-running agents. I also learned that cloud agents are becoming a big deal at OpenAI and Cursor, too.

OpenAI bets big on Cloud Agents

OpenAI acquired Ona, (formerly Gitpod), the leader in cloud development environments (CDEs). Back in 2021, CDEs were built for developers to develop software faster, and they also happen to be the perfect primitive for agents to run in a sandboxed cloud environment. From the acquisition announcement by OpenAI (emphasis mine):

“As Codex becomes more capable, its most valuable work is unfolding over hours or days, rather than minutes. We believe people should be able to delegate more ambitious work without remaining tied to the machine where it began. The work should continue beyond the initial session, with Codex making it possible to stay connected and check progress, provide direction, make decisions, and review results from anywhere.

Ona will help us do that. Its technology provides secure, persistent environments where agents can access the tools, systems, and context they need to make progress over time.

By bringing Ona to OpenAI, we will expand Codex beyond work tied to a single device or active session and help more organizations deploy agents securely in production.“

At OpenAI’s offices, I asked engineers there if their focus is shifting to cloud-based agents. Their answer: it very much is. This is a fairly recent development and they’re hiring engineers for the Cloud Agents team. Here’s one job ad that’s currently live:

“We are looking for an experienced software engineer to help build and scale our cloud agent platform. You will design and operate systems for orchestrating agents at scale. You will work closely with product engineers on ChatGPT, API, and Codex to define the right abstractions and enable them to ship products quickly. Strong backend or infrastructure experience is important; experience with Python, Rust, distributed systems, cloud infrastructure, or product platforms is especially helpful.”

Cursor: running agents in the cloud is the future

At Cursor, I spent an hour with cofounder Sualeh Asif (formerly the CTO, now Chief Product Officer). Cursor released Cloud Agents at the end of last year, and is starting to focus a lot more on this area. Sualeh revealed some interesting details about working with cloud agents:

• Agents in the cloud don’t have a way to “complain.” With running an agent locally, when it gets warnings or errors, it surfaces them to a human in its response, who instructs it to do X or Y. However, there’s no such loop for a long-running task on the cloud! Cursor came up with the idea for the model “confess” in regular interviews, and the “confessions” are shared with the infra team to improve the agents’ environment.

• Long-running agents have their own challenges. What happens when a node terminates, midway through; how do you move agent execution from one node to the other? There are new, nontrivial engineering challenges the team needs to solve.

Only yesterday, (Monday, 29 June), Cursor launched its iOS app that enables the building of software from anywhere.

This product is built on top of cloud agents to allow for long-running tasks, the company said:

“Cloud agents run in isolated virtual machines with full development environments to test, verify, and demo work. Since they operate asynchronously with their own tools and resources, cloud agents can run for longer and iterate toward merge-ready PRs without intervention.

To take advantage of these capabilities, send a local plan to a cloud agent or move active agents to the cloud to keep running. You can move the cloud session back to your computer to test changes locally before merging”.

Why are cloud agents suddenly a thing?

It figures that running AI agents in the cloud is practical: there’s less setup involved, several can run in parallel, and the cloud is a better, more convenient place for long-running agents than a personal laptop is; i.e., having to keep the lid open even when walking around the office.

But why is this happening now? My hypothesis is that a mix of factors are at play:

• Coding models got ‘good enough’. Before Opus 4.5 / GPT-5.4, AI models could not really code autonomously, so running them for long tasks was pointless!

• Infra for AI coding agents has matured. Ways of giving more context to agents have improved: things like MCP and skills became mainstream and better understood.

• The context window is bigger. Today’s models have context windows of up to 1 million tokens, meaning that more complex instructions, code, and context can be passed in. It’s hard to have agents run for a longer time without access to a large context window.

• Cloud providers have much more GPU capacity. Every cloud provider has been building GPU clusters in the last few years, and now there’s enough that these AI agents can make use of this infra.

2. Mass adoption of coding harnesses by non-developers?

At OpenAI, I also met Andrew Ambrosino, who was the first engineer on the Codex team. Our time together got off to an ideal start, with Andrew saying he needed to show me something incredible:

Read more

Listen and watch now on YouTube, Spotify, and Apple. See the episode transcript at the top of this page, and timestamps for the episode at the bottom.

Brought to You by

• Antithesis – If you’re using agents to code, the problem isn’t writing the code but making sure it didn’t break anything. Antithesis runs your whole system in a hostile environment and identifies hard-to-find bugs before users hit them in production. With Antithesis, teams like Jane Street, Fly.io, and the etcd community use agents safely and ship better code, faster. Learn more.

• Sentry – application monitoring software built by developers, for developers. Sentry’s Seer AI agent is growing on me, as a way to debug errors faster. Same with Sentry’s MCP server. Check out Sentry.

• Google Cloud Run – a fully managed platform that runs your code directly on top of Google’s scalable infrastructure. Run frontend and backend services, batch jobs, host LLMs, and agents without managing infrastructure. Oh, it has automatic zonal failover out of the box – something that Coinbase could have used a few weeks back! Get started.

In this episode

Navdeep Singh – oftentimes better known as NeetCode – is the creator of NeetCode.io, one of the most popular coding interview preparation platforms and YouTube channels for software engineers. Before building NeetCode full-time, he worked as a software engineer at Amazon and Google.

In this episode of The Pragmatic Engineer, I sit down with Navi to discuss his path from Amazon and Google to building his own startup, why he left Amazon after just two months, what he learned at Google, and the decision to leave a stable engineering career to bet on himself. We also discuss what coding interview preparation teaches beyond passing interviews, the value of going deep on difficult problems, and why systems thinking and domain expertise remain essential engineering skills in the age of AI.

Throughout the conversation, NeetCode makes the case that learning hard things is one of the single best investments an engineer can make, helping build the judgment and expertise that remain valuable no matter how the tools change.

Key observations from Navi

Here are 10 interesting takeaways from our chat:

1. Companies have no real method for evaluating engineers – and likely never did. Navi believes the leetcode-style interview process has persisted because it scales well at large tech companies that need to train hundreds or thousands of interviewers, not because it predicts job performance well.

2. The CAP theorem’s “two-out-of-three” framing is widely taught, but technically shaky. Navi believes this theory of distributed data systems is incomplete, and says he felt validated when researcher and author Martin Kleppmann criticized it. It’s a reminder to think independently and not accept theories without understanding them.

3. Amazon’s intense culture left Navi reluctant to ask questions – which paradoxically, helped at Google. In Navi’s first job, he got used to working alone and not seeking help when needed, and continued this working style at Google. His manager there interpreted that behavior as independence, and as a result, he won rapid promotion from L3 to L4 (mid-level engineering role).

4. The NeetCode YouTube channel took off after he said he’d have to post less. Before viewers knew Navi had got a software engineering job at Google, his audience was small. But it turned out that announcing he’d have to post less for this reason boosted his channel! Suddenly, lots of people wanted to know how he’d landed the role.

5. Cheating tools are helping to resurrect in-person, whiteboard interviews at Google. Navi notes Google has restarted onsite coding interviews because it’s the only way interviewers can be sure that candidates aren’t using AI-powered cheating tools which make data structure and algorithms (DSA) interviews easy to pass.

6. Navi finds AI most valuable as a tech debt and refactoring assistant. He’s using AI to clean up years’ worth of low-quality code on NeetCode’s backend, which also validates the decision to take shortcuts in the knowledge they can be corrected later.

7. ‘Effort’ is becoming the differentiator as AI makes everything else cheap. Navi says how you can prompt almost anything, but the capacity to be engaged with and care about your work, and to defend decisions you make, cannot be prompted by an AI tool. These depend on personal qualities like effort and dedication.

8. Announcements of the death of coding are exaggerated. Despite dramatic improvements in the performance of AI models, Navi does not foresee the majority of engineers being laid off. In fact, he sees the opposite: devs are busier than ever.

9. Humans are likely to remain better at weighing up tradeoffs than LLMs are. It’s a fact that LLMs have become a lot better at coding, but Navi doubts they will be much help in decisions involving judgments about tradeoffs.

10. When hiring for NeetCode, personality traits and motivation matter more than coding skill. Navi’s best recent hire is still an undergrad with little coding experience, but does exceptionally well thanks to possessing high agency. Navi says: “even if they have no idea how to start it, by a week later, they’ll have learned everything about it.”

The Pragmatic Engineer deepdives relevant for this episode

• Learnings from conducting ~1,000 interviews at Amazon

• How experienced engineers get unstuck in coding interviews

• The Reality of Tech Interviews in 2025

• Tech hiring: is this an inflection point?

• AI fakers exposed in tech dev recruitment: postmortem

Timestamps

00:00 Intro

02:57 Navi’s take on coding interviews

06:41 Getting into tech

08:56 Why Navi isn’t a fan of the CAP theorem

13:12 Quitting Amazon after two months

18:22 Google vs Amazon

22:26 The origins of NeetCode

25:27 Leaving Google to go all in on NeetCode

32:02 Why Navi doesn’t fix every bug

39:26 The value of coding interview prep

42:57 Systems thinking and domain expertise

47:28 Hiring at Big Tech

52:15 Tech stack at Neetcode

57:57 The NeetCode redesign contest

1:01:46 The future of software engineers

1:09:04 Hot takes: AGI, AI skill erosion, personality traits

1:22:49 “Maybe some people should just give up”

1:24:39 How to be a standout engineer

1:27:55 Book recommendation

References

Where to find Navdeep Singh (NeetCode):

• X: https://x.com/neetcode1

• LinkedIn: https://www.linkedin.com/in/navdeep-singh-3aaa14161

• YouTube: https://www.youtube.com/c/neetcode

• Website: https://neetcode.io

Mentions during the episode:

• A critique of the CAP theorem: https://martin.kleppmann.com/2015/09/17/critique-of-the-cap-theorem.html

• Designing Data-intensive Applications with Martin Kleppmann: https://newsletter.pragmaticengineer.com/p/designing-data-intensive-applications

• PACELC design principle: https://en.wikipedia.org/wiki/PACELC_design_principle

• Amazon Chime: https://aws.amazon.com/chime/getting-started

• Musk’s 5 Step Design Process: https://modelthinkers.com/mental-model/musks-5-step-design-process

• AI Engineering with Chip Huyen: https://newsletter.pragmaticengineer.com/p/ai-engineering-with-chip-huyen

• Angular: https://angular.dev

• Firebase: https://firebase.google.com

• TypeScript: https://www.typescriptlang.org

• An update on recent Claude Code quality reports: https://www.anthropic.com/engineering/april-23-postmortem

• Building Claude Code with Boris Cherny: https://newsletter.pragmaticengineer.com/p/building-claude-code-with-boris-cherny

• Sora: https://en.wikipedia.org/wiki/Sora_(text-to-video_model)

• Attention is all you need: https://arxiv.org/abs/1706.03762

• The End of Programming as We Know It: https://www.oreilly.com/radar/the-end-of-programming-as-we-know-it

• Satya Nadella on X: https://x.com/satyanadella

• Replit: https://replit.com

• Lovable: https://lovable.dev

• 37signals: https://37signals.com

• DHH’s new way of writing code: https://newsletter.pragmaticengineer.com/p/dhhs-new-way-of-writing-code

• MongoDB: https://www.mongodb.com

• Maybe some people should just give up:

—

Production and marketing by Pen Name.

Three weeks ago, at Craft Conference, in Budapest, Hungary, I opened the event with a keynote titled ‘Slow Down to Speed Up’.

As with most of my talks, it came together in stages, including with some input from full subscribers to the Pragmatic Engineer, with whom I shared my thinking in advance, in ‘Ideas: slow down to speed up when working with AI agents’. Thank you for all the comments!

As fate would have it, just two days beforehand, social media giant Meta appositely provided a real-world case study for my talk, with its most embarrassing outage of all time: users could simply ask the Meta AI to change the email of any account, and the bot happily complied – even if the account belonged to someone else entirely – including a former US president. It was a timely example to kick off the talk with. Check out the full keynote that’s available to view on YouTube:

Watch the keynote video

In this article, I summarize the key parts of my Craft Conference keynote in detail, and some responses received at the event. Full subscribers also have access to the slides, here, and at the foot of this article.

We cover:

1. Meta: “AI psychosis” in effect? Meta has been destroying its engineering org, and an obsessive focus on AI seems to be one reason for it. For more on this question, check out this deep dive.

2. Everything’s changed in six months. From around November last year, things changed with a more capable generation of AI agents like Opus 4.5 and GPT-5.4.

3. How are tech companies changing how they work? Anthropic, OpenAI, Google, Uber, startups, and traditional companies.

4. Trends. Individual productivity is up, but team productivity’s flat, tokenmaxxing and tooling adoption, vanishing middle management, CEOs and CTOs back to coding, and more.

5. Trends across software. Falling software quality, GitHub’s constant reliability woes, AI slop overwhelming devs who care about quality, and more.

6. Advice for software engineers and engineering leaders. Suggestions to help future-proof a career.

7. Feedback. “It’s happening here too!” is a common theme, and relief for some that it’s not unique to their own workplace.

1. Meta: “AI psychosis” in effect?

I thought it was a made-up story when I read that Meta had enabled account takeovers via a “zero auth” policy; i.e., simply asking the Meta AI bot was sufficient to change any account’s email address. After all, shipping such a regression would fly in the face of security measures, code reviews, automated testing, and metrics. Plus, the company has dedicated Integrity teams whose mission statement is to ensure something like this never happens… And yet, this bug shipped.

It went undetected by anyone at Meta, and high-profile accounts like that of former US president, Barack Obama, were taken over as a result. Instagram’s dedicated Integrity team seems to have discovered the embarrassing issue via the news.

As mentioned, it was two days before the Craft keynote, so there was enough time to ask around at Instagram and Meta. Engineers at the company there told me this disaster was caused by AI-generated, AI-reviewed code, along with layoffs, and by forced reassignments from Integrity teams and elsewhere onto AI labeling and related duties.

The problem at Meta seems to be that leadership is aggressively pushing AI, while withdrawing resources and headcount from areas responsible for security, quality, and reliability. Since last week’s deepdive into what’s been happening behind the scenes was published, I’ve learned further details:

• Integrity teams at WhatsApp have been hit hard by layoffs and enforced data-labeling reassignments

• Instagram’s design team suffered a 44% cut in headcount during layoffs

• The Developer Documentation and Support team had a full 95% headcount reduction during layoffs

• Data labeling at the ADO group goes beyond “just” labeling; there are many AI training tasks to do. But these are repetitive, unless you get really creative.

Based on everything I heard from talking with Meta folks, AI-induced behavior was indeed at the heart of this outage. AI-generated, AI-reviewed code, and security teams being gutted, were also factors in the beyond-embarrassing incident. As reported in last week’s deepdive:

• Instagram’s Trust and Safety Team lost around 50% of its staff to data labeling and layoffs. Some of the most senior folks were drafted onto AI training tasks.

• AI-generated changes with zero human input, with just an additional AI code review, have been very common in recent months across the codebase. The change that caused this outage looked like one of these

• Normally, the Trust and Safety team would be on top of monitoring and alerting of security breaches, but it is currently in full disarray due to rapid, internal disorganization”.

If major changes like data labeling assignments and staff tracking are undone, then perhaps things at Meta could return to normal. But so far, the most being done is that leadership has boosted budgets for snacks, travel, and events. Hardly the change needed to restore morale and the former culture!

The comparison to the Lumon corporation in the hit show, Severance, was duly made:

Meta’s worst-ever outage can be interpreted as a warning about what happens when there’s so much focus on AI that the basic health of a company’s main – money-spinning – products is neglected. Instagram, WhatsApp, and Facebook generate the bulk of revenue for Meta, but the company is reallocating more engineers to training the coding model, and aggressively cutting the headcounts of vital orgs to do so – up to the point of not having oncall coverage for key services, and security teams being too stretched to do their jobs.

Am I missing some insight about why it’s more important to build a state-of-the-art, likely-closed AI model that’s good at coding, than it is to keep operating revenue-generating businesses with stable infra?

2. Everything’s changed in six months

Independent, experienced software engineers with zero affiliation to AI labs have been saying for a few months that how we do software engineering has been transformed.

David Heinemeier Hansson (DHH), creator of Ruby on Rails in January:

”Just [in] summer 2025, I spoke with Lex Fridman about not letting AI write any code directly, but it turns out part of this resistance was simply based on the models not being good enough at the time! I spent more time rewriting what it wrote, than if I’d done it from scratch. That has now flipped.”

Simon Willison, creator of Django, in May pinpointed the start of the change to late last year:

“The models released in November 2025 elevated agents to being genuinely useful. We’ve had six months to get used to that idea now; it’s no wonder companies are beginning to spend real money on this technology.”

Teams using agents now ship 5x as many pull requests as two years ago. Here’s data from Linear:

Devs using AI harnesses are producing 2.5x as much code versus 18 months ago. Data from Cursor shows that their users, on average, went from adding 3,500 lines of code in January 2025 to 8,600 today:

The size of pull requests is up 3x versus 18 months ago. Also from Cursor:

More AI changes are accepted without human review. Data from Cursor shows a big jump in changes being accepted without human review from around February this year, when Opus 4.7 and GPT 5.5 launched:

We’re seeing a lot more code generated, and less of it than ever being reviewed by devs. In the relatively short time since AI agents became really good last November, there are more pull requests generated by devs, those pull requests are getting better, and code reviews are harder to keep up with. And so, reviews are less stringent and more changes are shipped to production sans human review! As per my discussions with Meta engineers, these kinds of AI-generated, AI-reviewed pull requests [at Meta, they’re called diffs] are what caused the most recent, embarrassing outage at Instagram.

3. How are tech companies changing how they work?

Details from a few larger tech companies:

Anthropic: all-in on AI agents. In March, Boris Cherny, creator of Claude Code, was on the Pragmatic Engineer podcast and shared some details:

• He personally runs ~5x agents parallel, and ships 20–30 PRs/day

• Product requirement documents (PRDs) are dead & prototypes have replaced them inside Anthropic

• ~100% of Claude Code was generated by Claude in March

• ~70-90% of code inside Anthropic was generated by Claude

• Claude Cowork – another billion-dollar product in terms of revenue potential – was built in just 10 days

Since then, Boris has shared that his workflow has changed to setting up loops to run agents.

OpenAI: moving much faster with AI agents. OpenAI’s Codex team was on the main stage at The Pragmatic Summit in February. Tibo Sottiaux (head of engineering, Codex, OpenAI) shared interesting details on how software development is done in the Codex team:

• There’s a “fix this” button integrated into the internal OpenAI mobile app. It makes one-shot fixes to bug reports, which devs review and can merge

• AI code review for all code changes. With a tiered approach, some changes can be merged with just AI review, and more important ones need an extra human review

• Most devs run several agents in parallel, often walking around with their laptop lids open, so the machine doesn’t enter sleep mode and suspend agents

• Code isn’t really written by hand anymore on the Codex team, and is also less common on other teams too

• “Taste” is becoming a core skill for working at the company

• Codex improves itself: it runs its own test suite, runs improvement tasks overnight, and during team meetings it takes actions on topics discussed

Google: AI widespread. Gemini is not as capable at coding as Claude or Codex, as acknowledged by Google’s CEO, but it’s widely used companywide. The less capable coding model could be hurting AI adoption compared to other companies.

Uber: in-house AI infra. We covered in-depth how Uber uses AI for development, touching on internal systems like:

Uber’s MCP Gateway:

Uber Agent Builder:

The AIFX command line interface:

Minion: background agents

Code Inbox:

Smart Assignments as a neat feature of Code Inbox:

Risk Profiles: another smart feature inside Code Inbox:

uReview, Uber’s AI code review tool:

Autocover and Shepherd for large-scale migrations:

Uber is a good case for learning how much of internal developer infra needs to be rebuilt in order to work well with AI agents. Uber built all the tools above because they needed new, better ways to integrate AI agents into the developer workflow, but couldn’t find anything that worked up to requirements. I’d also point out how much time and effort Uber invested in making code review more efficient. Devs are, indeed, getting overloaded with AI code reviews and Uber’s Code Inbox tries to separate the important pieces of code to review from unimportant ones.

Startups are jumping into using AI agents, although their integrations are more basic. In preparation for the keynote, I talked with several startups about their AI usage. Harnesses like Claude Code, Codex, Cursor, OpenCode and others are popular, and I also noticed most startups are heavily integrating AI agents into Slack, so devs can kick off bugfixes or small feature requests straight from the chat tool.

I observed startups being the most likely to experiment with new AI dev tools; from code review, all the way to AI incident management tools.

“Traditional” companies are also heavily investing in AI dev tools. At the recent Pragmatic Summit in San Francisco, Laura Tacho shared interesting details:

• In February, 18,000 Cisco developers used Codex for complex migrations, code review, and refactoring. This was very early – Codex was just starting to gain industry-wide adoption!

• JP Morgan Chase built a multi-agent framework for annotation, using multiple specialized agents to label customer interaction data, and judge agents to aggregate and rank results. These are pretty advanced use cases!

In general, “traditional” companies do not seem to be lagging behind in using, paying for, and adopting AI agents and AI developer tools.

4. Industry trends

There are trends I’ve observed around the adoption of AI dev tools:

Read more

Today, we cover:

1. Meta follow-up. A follow-up on Tuesday’s deepdive into Meta’s ongoing demolition of its own engineering org. Non-engineering teams lost more than 10% of their staff, while Integrity teams were already stretched before the cuts and reallocation.

2. Big implications of the US banning Anthropic’s new model, Fable. The US government wants only US citizens to be able to use Fable. That could make China more influential as other countries and non-US companies look to capable open models – most of which come from China.

3. SpaceX, Cursor, Continue. Elon Musk became a trillionaire with SpaceX’s IPO, while SpaceX acquired Cursor, and Cursor acquired Continue. SpaceX looks like it wants to go head-on against Anthropic and OpenAI.

4. Industry Pulse. Gemini lead quits for OpenAI, Epic Games’ new open source version control system, Microsoft considers dropping OpenAI for DeepSeek, research confirms LLMs amplify existing expertise, and more.

1. Meta follow-up

Tuesday’s article about Meta destroying its engineering culture made quite the splash in terms of comments and debate online. From inside Meta, there was positive feedback on the accuracy of the deepdive into the situation, and there were also a few helpful clarifications and new information which I’m happy to publish here:

Read more

Listen and watch now on YouTube, Spotify, and Apple. See the episode transcript at the top of this page, and timestamps for the episode at the bottom.

Brought to You by

• Antithesis – if you're using agents to code, the problem isn't writing the code but making sure it didn't break anything. Antithesis goes beyond code review and runs your whole system in faster than real-time and identifies hard to find bugs before your users hit them in production. Antithesis enables teams like Jane Street, Fly.io, and the etcd community to use agents safely and ship better code, faster. Learn more

• WorkOS – make your app and agents Enterprise Ready, with SSO, SCIM, RBAC, and more. Get started.

• turbopuffer – a vector and full-text search engine built on object storage. It’s fast, cheap, and extremely scalable. The teams building the smartest AI products out there — Cursor, Notion, Cognition, Anthropic — they all run on turbopuffer.

In this episode

Robert Erez is a principal engineer at Octopus Deploy, and a longtime expert in CI/CD, deployment systems, and software delivery. Rob and I were also once colleagues on the Skype web team, working on large-scale deployments and release processes.

In this episode of The Pragmatic Engineer, I sit down with Rob to discuss how teams deploy software safely and efficiently at scale. We cover Kubernetes, GitOps, platform engineering, progressive delivery, feature flags, cloud development environments, and the growing role of AI in CI/CD workflows. We also get into the tradeoffs in different deployment approaches, why self-hosted software still matters for some organizations, and the recent evolution of software delivery practices.

Key observations on deployments and CI/CD from the conversation with Rob

Here are 10 interesting takeaways from our chat:

1. Roll forward, never backwards. When a system has state – which typically means it uses databases – then doing a rollback can leave the code talking to a schema that’s no longer in sync. Rob’s advice is to not treat a failure in v2 as a trip back to v1, but rather as a push to v3 with the fix in it.

2. GitOps isn’t actually about Git. None of the four pillars of GitOps – 1) declarative, 2) versioned and immutable, 3) pulled, not pushed, 4) continuously reconciled – require Git, although Git can work under these constraints. Yet, the term ‘GitOps’ has made the industry dogmatic about cramming everything into a repo – even things like secrets that absolutely shouldn’t be there!

3. Continuous deployment can be overkill; continuous delivery is more practical. Shipping every single change to prod (continuous deployment) is not as necessary as many people think, Rob says, and there’s often more value in continuous delivery, where changes flow through testing and the deployment process itself is validated. With continuous delivery, you can decide whether to push to production automatically, or click a button once a week.

4. Feature toggles are a better safety net than rollbacks. When something breaks in production, reaching for a toggle to switch a feature off enables you to “stop the bleeding” and then calmly diagnose an issue. Rolling back a feature flag is less nerve-jangling than scrambling to force a redeployment in the middle of the night!

5. One problem with feature flags is that they’re addictive. On the other hand, the ease with which feature flags are added can create a hygiene crisis if they’re continuously added, but not removed. Treat feature-toggle cleanups like a form of gardening and “weed” rolled-out toggles from the codebase.

6. A Git repo can be a bottleneck at scale. Rob mentions that some companies run thousands of independent Kubernetes clusters that pull state from a Git repository. But such clusters can get throttled by the repo, forcing them into workarounds. Pull-based GitOps doesn’t scale infinitely for free.

7. A sizable number of major institutions remain on-prem – and this won’t change. Banks, other financial bodies, and governments, demand full control over their hardware, upgrades, and downtime. That’s why Rob expects this segment won’t move to cloud-based SaaS.

8. Platform teams work at larger companies. These teams earn their keep in big organizations with multiple teams and projects because they offer ways of bringing sanity and focus.

9. There’s a trend of ephemeral environments replacing test/staging environments. Companies used to have a few testers fighting over a handful of static test environments, but today, it’s trivial to spin up a full environment, per-feature branch, pre-merge. This is an “ephemeral” environment for evaluating that things work, which is then torn down once something is merged. It helps speed up the feedback process.

10. AI shifts the CI/CD calculus from speed to risk. Today, shaving ten minutes off the CI build-time matters because a long-running build blocks human devs. But this time saving will be insignificant when an AI agent writes most of the code and “babysits” a slow pipeline without context switching. Then, the new priority will be to reduce the risk of an AI agent shipping a bug to production, so it will make much more sense to run extra, more thorough, tests – and also even slower ones.

The Pragmatic Engineer deepdives relevant for this episode

• Kubernetes and retiring at the top with Kelsey Hightower

• The past and future of modern backend practices

• Microsoft is dogfooding AI dev tools’ future

• How Kubernetes is built with Kat Cosgrove

• How Linux is built with Greg KH

Timestamps

00:00 Intro

02:09 Canary deployments at Skype

05:01 Joining at Octopus Deploy

06:15 Continuous deployment

10:26 Why Kubernetes won

15:51 Kubernetes on-prem

18:50 How GitOps works

25:00 The uses and limitations of GitOps

31:04 The rise of platform teams

35:51 How AI is changing CI/CD

39:49 Progressive delivery explained

47:31 Rollbacks and roll-forwards

50:14 Feature flags

54:32 How development environments are evolving

57:40 Cloud development environments (CDEs)

1:03:45 Self-hosting CI/CD

1:09:25 Getting started with progressive delivery

1:11:15 Book recommendations

References

Where to find Robert Erez:

• X: https://x.com/no_erez

• LinkedIn: https://www.linkedin.com/in/roberterez

Mentions during the episode:

• Skype: https://en.wikipedia.org/wiki/Skype

• Canary deployments: https://docs.aws.amazon.com/whitepapers/latest/overview-deployment-options/canary-deployments.html

• Octopus Deploy: https://octopus.com

• Paul Stovell on LinkedIn: linkedin.com/in/paulstovell

• Kubernetes: https://kubernetes.io

• How Kubernetes is Built with Kat Cosgrove: https://newsletter.pragmaticengineer.com/p/how-kubernetes-is-built-with-kat

• Kubernetes and retiring at the top with Kelsey Hightower: https://newsletter.pragmaticengineer.com/p/kubernetes-and-retiring-at-the-top

• Docker: https://www.docker.com

• HashiCorp: https://www.hashicorp.com

• Mitchell Hashimoto’s new way of writing code: https://newsletter.pragmaticengineer.com/p/mitchell-hashimoto

• Terraform: https://developer.hashicorp.com/terraform

• GitOps: https://about.gitlab.com/topics/gitops/

• Cursor: https://cursor.com

• The Phoenix Project: A Novel About IT, DevOps, and Helping Your Business Win: https://www.amazon.com/Phoenix-Project-DevOps-Helping-Business/dp/0988262592

• Radical Candor: Be a Kick-Ass Boss Without Losing Your Humanity: https://www.amazon.com/Radical-Candor-Kick-Ass-Without-Humanity/dp/1250103509

• Diaspora: https://www.amazon.com/Diaspora-Novel-Greg-Egan/dp/1597805424

• Schild’s Ladder: https://www.amazon.com/Schilds-Ladder-Novel-Greg-Egan/dp/1597805440

• The Clockwork Rocket: Orthogonal Book One: https://www.amazon.com/Clockwork-Rocket-Orthogonal-Book-One/dp/0575095148

—

Production and marketing by Pen Name.

Subscribe now

Many subscribers expense this newsletter to their learning and development budget. If you have such a budget, here’s an email you could send to your manager.

For two decades, Meta had a unique, high-performance engineering org; right up until around April of this year. For the first 20 years of the company’s existence, it had a “move-fast-and-break-things” culture, and in the early 2020s this shifted to a “move-fast-with-stable-infra” one. Engineers I know at the company were empowered to do good work, focus on impact, and to balance business interests with solid engineering.

But in the past few weeks, all that has changed, as if the leadership has been following detailed blueprints on how to demolish a proven, successful engineering culture in the most ruthlessly efficient way possible.

For the past few weeks, I’ve been sharing how bad things are inside the social media company for engineers in one of Silicon Valley’s most prestigious workplaces. In this article, we walk through what’s happened, and ask what’s going through the minds of leadership who are reducing software engineering there from the profit center that it was between 2004 until very recently, to the disdained cost center that it has become in just a few weeks.

We cover:

1. Meta’s pre-AI engineering culture

2. Investing in AI and pressing engineers to always use it

3. Core engineering folks feel treated like trash

4. Most embarrassing-ever outage

5. Internal mess

6. Self-inflicted wounds

7. Is “AI psychosis” just a Meta issue?

1. Meta’s pre-AI engineering culture

I’d split Meta’s engineering culture into two eras: “move fast and break things”, and then “move fast with stable infra.”

“Move fast and break things”

In the 2010s, Facebook’s unconventional engineering culture had grown somewhat legendary in the tech industry, as the company went against conventional best practices and succeeded massively.

In 2012, when Facebook hit the billion-users landmark, the company produced a small physical book about its culture which was placed on employees’ desks. Presented with retro propaganda design, it was dubbed the “little red book”, co-opting the name of a famous volume of the thoughts of Chairman Mao, (1964).

At around 70 pages long, Facebook’s version codified its engineering culture: speed, fearlessness, taking ownership, and thinking outside of the box.

Back then, mantras in Facebook’s little red book were also in print across campus, and included:

• Move Fast and Break Things

• Done is Better Than Perfect

• Fail Harder

• What Would You Do If You Weren’t Afraid?

• Every Day Feels Like a Week

• The Wright Brothers Did Not Have Pilot Licenses

• The Foolish Wait

• Fortune Favors the Bold

There was genuine focus on building good products. Also from the book:

“Move fast with stable infra” culture

In 2022, I did what is one of the longest deepdives we’ve published on the topic of Meta’s engineering culture. By then, things had evolved, and much of any former recklessness was gone, replaced by the principle of moving fast, but with stable infra. Here’s how I described Meta’s engineering culture then:

“The culture is incredibly engineering-centric: much more than most of Big Tech. This might come from Mark Zuckerberg being an engineer himself, or because much of the innovation in the early days of Facebook came from engineers.

Focus on individual impact. Impact has been the bread and butter of the focus at Facebook. This is very true since the early days, and the focus on generating impact remains.

One detail in common with most Big Tech firms is that both the engineering culture and general culture focus so much on individual impact. This results in some people focusing on short-term, measurable wins and assuming that teamwork and split wins between groups might be less rewarded.

The lack of rigid processes. Facebook seems to have the least amount of processes or standardization across all of Big Tech. Don’t even try to compare it to Amazon’s engineering culture and the countless formal processes there. But even compared to companies like Google, Microsoft or Uber, Facebook’s processes are much looser. Most of this comes from the engineering-centric nature of the company and engineers disliking processes.

Surprisingly little emphasis on testing, documentation or code comments. You’ll find shockingly little automated testing and documentation at Facebook, compared to the rest of Big Tech. Inline code comments are also very rare.

A founder-engineer driven company. Facebook is one of the few Big Tech firms whose founder is an engineer, and still is the CEO. Netflix is the other one where founder and co-CEO Reed Hastings was also a software engineer before starting the company. Amazon was the other example of this until recently, but it’s not the case at Google or Apple. There are good examples of smaller companies like Cloudflare, but they’re all younger than Facebook.

Bootcamp. A unique onboarding process, unlike what any other Big Tech firms offer. We cover this more in the Bootcamp & onboarding section.”

Also, Facebook, as a product, has one of the most sophisticated auto rollout systems in the industry. Instagram has a battle-tested infrastructure where it was almost trivial to launch a new social network (Threads) with 100 million users served in its first week.

Engineers whom I knew inside the company are capable, motivated, and product-minded, and their work was appreciated. CEO, Mark Zuckerberg, was influential: he personally coded the first version of Facebook, had stayed close to engineering, and valued software engineers very much. Engineers there felt they were working inside a profit center.

2. Investing in AI and pressing engineers to always use it

Meta has been the only company among the big five of Apple, Microsoft, Amazon, Google, and itself not to own a hardware platform or operating system. Apple has the iPhone, iPad and Macs, Google has Android, ChromeOS and Pixel phones, Microsoft has Windows, and Amazon has the Kindle.

Stepping back, it looks as though the Mark Zuckerberg of today has resolved not to miss a platform opportunity, after the company failed to build its own mobile OS or mobile phone during the 2010s.

This is one reason for investing so much in virtual reality (VR) with Oculus, and in augmented reality with the Meta Glasses. Facebook changed its name to Meta in 2021, back when it looked like VR – and the metaverse – could be massive. Billions was spent on ensuring Meta would be the market leader in this space. But once again, VR didn’t go mainstream; since the end of the pandemic, popular interest in the segment has died down considerably.

When it became clear that AI would become a mega-trend in 2022, Zuckerberg didn’t miss it: he assembled the internal FAIR group (Fundamental AI Research team) as well as a GenAI product organization and released a series of open-weight AI models:

• Llama 1: released in Feb 2023, three months after ChatGPT, built by FAIR

• Llama 2: in June 2023, built by the GenAI product organization (as well as all subsequent Llama models)

• Llama 3: in April 2024. This model was Meta’s most competitive LLM of all, and gained momentum in adoption across the industry

• Llama 4: in April 2025. This model was deeply disappointing

In June that year, Meta acquired a 49% stake in Scale AI to reboot its AI efforts for a whopping $14.8B, and brought in Scale AI’s CEO, Alexandr Wang to take over Meta’s AI strategy. The acquisition of Chinese startup Manus AI for $2B is currently in question after China blocked the deal from being completed.

Based on the investment made into Scale AI and Wang, it’s pretty clear that Meta – and Zuckerberg – is determined to build a state-of-the-art LLM that can be competitive with the latest versions of Claude and ChatGPT. But Meta has to start pretty much from scratch, and it’s up to Alexandr Wang to deliver.

Scale AI brings in a very specific kind of expertise to Meta, as one of the best in the industry in:

• Training data and labeling: Scale started, and is still best known, as a provider of high-quality labeled datasets for machine learning and AI training, including code, text, image, video, etc.

• RLHF and fine-tuning: A RLHF (reinforcement learning from human feedback) flow which Scale runs, where people give feedback for foundation models, as a “human in the loop” data engine that many leading AI labs use to create better LLMs.

Wang seems to have a very broad reign to do what he has been an expert in: creating training data, doing data labeling and RLHF. This is being pulled off with the labor of Meta’s engineering workforce, and by surveilling it.

Problem #1: Tracking keystrokes and mouse clicks, with no option to opt out. In late April, Meta told engineers they were being enrolled into a system that tracks every keystroke and click, to produce training data for Meta’s new AI. There’s no way to opt out.

Needless to say, this is invasive and raises privacy questions: If you log into your personal bank account, does the tool track you? What about when you’re writing a personal email, or responding to a personal call? Meta held no consultation and there are no workarounds; just a top-down decision being pushed through.

This month, Reuters reported that people’s concerns there are finally being heard:

“Meta is dialing back elements of its plan to collect employee mouse movements, keystrokes and other actions for use as AI ​training data, it said in an internal memo on Tuesday, following weeks ​of angry pushback from staffers.

New controls will allow employees to pause ⁠the data collection for up to 30 minutes at a time and ​request exemptions from the initiative, according to the memo, authored by Stephane Kasriel, ​a vice president in Meta’s AI model-building Superintelligence Labs unit.”

From talking with current Meta engineers, I understand the logging system has not been rolled out in the UK due to data protection regulation.

Problem #2: 30-50% of engineers on core teams have been forcefully reassigned to data labeling and RLHF, upsetting folks even more. Also starting in late April, product engineering teams received a mandate from above, whereby 30-50% of engineers were to leave the team and join the ADO org (Agent Data Optimisation).

“Forceful” reassignment is very relevant here because of Meta’s traditional engineering culture. Between its founding in 2004 and until last year, Meta gave engineers autonomy to choose where they work and what they work on. This was structural to how the company worked:

• Engineers were not hired for a specific team (save for at the Staff+, levels, in some cases). They were hired to the company

• During a 6-week bootcamp, new hires got familiar with Meta’s engineering culture and chose a team

• Team matching meant talking with multiple teams who had headcount, doing small work with them, and finding a match

• Internal transfers were easy, and often initiated by engineers

Team selection via bootcamp started to die down in around 2024, but any Meta engineer with at least two years’ tenure knows that previously they chose what to work on, and of course, could pick the most impactful thing to work on. And then, out of the blue, they’re assigned to a division where the impact is not clear, the work is menial, and doing it too long will surely hurt their career prospects.

“Data labeling” is more involved work, even though a bit repetitive. There are labeling tasks, where you create a website, then look at it and decide if it looks good or not, and give this feedback: this is the “typical” data labeling. But this is the less frequent type of work. But then there are more involved AI training tasks, which looks like this:

• Come up with a task that the AI should do

• Then write the tests that confirm the result

• Package all of this up into a Docker container, using Harbor framework

• Then read the code that the AI writes - often doing this based on feedback from several models, and give it feedback

This work is not easy to do — and you can see why you need good software engineers to do it! — but it gets repetitive really quickly. Most engineers who I talked to said that they found it hard to do this work motivated day-in, day-out. Then again, I did talk with an engineer inside the org who said that by varying the technologies they use, and challenging themselves, they found this work motivating and interesting! This engineer expects there to be more software engineering work coming up, after the phase of training passes. There’s a lesson here: when life gives you lemons, you can either complain about it, or you can make lemonade, like this dev has done so.

This training work is secretive across the AI industry, and vendors are offering $100+ per hour for devs to do this type of work. There are rumors across the industry that OpenAI and Anthropic spends more on these training environments (to make the models great at coding) than the model training run itself!

Infrastructure and security teams were hit especially hard by reassignments. I talked with several engineers in infra orgs, who had 30-50% of their teams drafted into the ADO org. And in some cases, it was the best engineers who left.

One engineer told me that the whole situation feels like the movie, The Hunger Games, when tributes are randomly selected and then removed from their environment, to something completely different. Except, at Meta, many more folks are being affected, with between three and five from a 10-person team going from building products used by hundreds of millions, to giving human feedback on AI-generated GitHub repos, over and over. So, a wider impact than in the Hunger Games, but with less drastic consequences.

Around 6,500 people are in the ADO org, more than at OpenAI and Anthropic. Roughly four to five thousand of these are software engineers. Meta has around 25,000 engineers, meaning that one in every 5-6 software engineers may now find themselves doing data labeling full time.

As you can imagine, people are actively open to new positions, and nobody is updating their job title on LinkedIn and elsewhere to “data labeling at Meta.”

I’ve spoken with people in this role and they don’t like doing it, and feel upset about the top-down decision making. The silver lining is that they still have a job, have retained their salary, and were not part of layoffs. They still have time to leave Meta for something that pays comparably and is not a data labeling job.

3. Core engineering folks feel treated like trash

Problem #3: a month-long waiting game, stoking fear across the company. On 20 April, Reuters reported that Meta planned to lay off 10% of staff in a month’s time, and Meta confirmed the news, meaning there was a period of four weeks when everyone knew that they could be unemployed very soon.

Forced reassignments to data labeling started to happen. As I covered at the time:

“Understandably, there are mixed feelings about this redeployment [to data labeling], with layoffs coming soon. On Wednesday, 20 May, Meta will announce layoffs. Perhaps those moved to do data labeling could actually be “safer” than colleagues on product teams. Of course, this is speculation, but it would be cruel if Meta cut devs reassigned to data labeling.”

Problem #4: Performance review is hyper-aggressive at Meta, so devs optimize all metrics. The internal performance review process, PSC (Performance Summary Cycle), is very stringent, compared to Google and Apple, I’ve learned. Managers inside Meta “fight” over the pay packets of their employees, which involves “knocking down” the packet of engineers on other teams, so their direct reports are ranked higher. It’s common to weaponize metrics in this process – be that business impact, the number of code reviews, number of lines of code written, pre-AI (see our Coding Machine archetype podcast on this.)

Quotas are handed down to managers for the splits of the workforce to be put in each ‘bucket’, and the internal politics gets heated as managers try to get their reports into higher buckets.

After a few years, engineers at Meta learn that the best way to not get a bad PSC rating is to have all metrics – impact, code committed, and other numbers – higher than their peers’ are. Learn more about the internal politics of performance calibrations.

Problem #5: tokens are measured as part of perf, so devs aggressively optimize for it. When layoffs were confirmed, engineers also learned that managers shall inspect token count during perf reviews. This raised worries that those with low token counts might be marked as underperformers and dismissed.

So, what is the natural reaction to this as an engineer at Meta? They started using AI tools for the sake of generating more tokens. This happened while Meta had an internal token leaderboard, encouraging tokenmaxxing. As I wrote on 16 April:

“As per The Information, Meta employees used a total of 60.2 trillion AI tokens (!!) in 30 days. If this was charged at Anthropic’s API prices, it would cost $900M. Of course, Meta is likely purchasing tokens at a discount, but that could still come in at $100M+ – in large part from senseless “tokenmaxxing”.”

The biggest problem: people stop caring about real work and focus on performative work. Let’s check the four ingredients that Meta’s leadership has decided to introduce to their workplace:

1. Tracking the keyboards and mouse clicks of all engineers, where legally possible

2. Reassign a good chunk of engineers to fulltime data labeling

3. Let staff know that 10% of them will be laid off

4. Have a culture where devs optimize for any and all metrics measured during PSC

5. Measure token usage as part of PSC

Shake this mix up well, and what do you get? Two things:

1. Everyone overuses AI to boost their personal stats. An engineering workforce that pretends to work with as much AI, and as little human input, as possible. It’s a strange incentive where an outage caused by a failure to review code properly is not grounds for dismissal, but writing code by hand – instead of having an AI agent write it – could cost you your job

2. Every longer-tenured engineer is seeking a new job, or at least considering it. Those who have been around at Meta longer term have seen enough. Let me describe this visually:

Fresh data seems to confirm that starting in May, a lot more engineers at Meta are looking for an “out.” Here’s how signups to leading Big Tech interview preparation service interviewing.io from Meta have changed over the last year and a half: we can see a massive jump in May vs the year before, as shared by founder and CEO Aliner Lerner with us:

(We covered more data from interviewing.io and other interesting data sources in the deepdive State of the software engineering job market in 2026)

To its credit, Meta has given out generous retention equity packages to several engineers considered key on the remaining teams. These packages make it harder to get matching compensation elsewhere. Still, I talked to one engineer who got an equity top-up, and said that this approach helped him decide to leave as soon as possible because he feels bitter about the lack of autonomy and having no control over things.

4. Most embarrassing-ever outage

Meta’s core infra and security teams have suddenly found themselves severely understaffed. Most folks are pushing AI-generated code merged with AI-only reviews, without paying much attention to quality. After all, they’re dealing with the possibility of unemployment, while firefighting to operate a team without its best engineers whose headcount has been cut in half, all with the knowledge that AI usage could affect their own job security.

Two weeks ago, on 30 May, the most embarrassing outage in Meta’s history happened. Here’s software engineer Siddharth Sundharam’s summary (emphasis mine):

“Yesterday, a slew of Instagram accounts, including some high profile ones like the Obama White House account, seemingly got hacked.

Look, I’m no spring chicken. I’ve spent almost a decade and a half identifying vulnerabilities and exploits at unicorn scale, but this is hands down the most unserious, “almost too stupid to be true” of them all.

The Takeover Flow:

Step 01: Faking the Location & Initiating Support. All the attacker needs to kick this off is your account username. Then, they hop on a VPN or proxy close to your city so Instagram’s security algorithms don’t suspect a thing. (You can quite easily get this from your public profile or “About” section or a hundred other ways.) Once it looks like the request is coming from the correct region, they tell the Meta support AI that the account is hacked and ask it to send the verification codes to an arbitrary email address they control.

Step 02: That’s It. Really, that’s it.

The first proper zero auth password reset I’ve seen in production. There appears to be no additional check as to whether the email being given is actually something the user has used before. Once the AI sends the security code to the attacker’s email, the attacker passes it right back to complete the verification. The platform hands over a fresh password reset link, granting full ownership to the attacker.”

This is a security breach in which Meta left its extra-secure, reinforced front door unlocked, so that anyone could come in, and there was no alarm to notify anybody when it happened! It seemed Meta only noticed when users started reporting it on social media!

From talking with folks inside Meta, I’ve learned that AI was at the heart of this outage. AI-generated, AI-reviewed code, and security teams being gutted were together the cause of this beyond-embarrassing incident. I poked around, and here’s what I gathered:

• Instagram’s Trust and Safety Team lost around 50% of its staff to data labeling and layoffs. Some of the most senior folks were drafted onto AI training tasks.

• AI-generated changes that saw no human input, just another AI code review, were very common during the last two months, across the codebase. The change that caused this outage looked like one of these

• Normally, the Trust and Safety team would be on top of monitoring and alerting of security breaches, but it is currently in full disarray due to rapid, internal disorganization.

Meta’s Chief Security Officer resigned the very next day. The outage was resolved on Monday, 1 June, and an investigation started as part of the SEV process. On Tuesday, Meta’s Chief Information and Security Officer (CISO), Guy Rosen, announced his departure.

Coincidence? I suspect not: the CISO might have stepped down if they warned against the Security org being gutted but were then ignored, and so no longer trusts leadership. I also imagine the CISO didn’t have the idea to move a good half of Instagram’s security team over to data labeling.

As a note, the previously worst outage was all Meta services going down for seven hours in 2021, due to a DNS / BGP configuration issue. It was a bad outage, but Meta handled the follow up well, in my opinion. After that 2021 outage, Meta shared a postmortem and apology. It has not done so for the latest Instagram account takeover outage.

5. Internal mess

Wired shares more details on just how bad the situation is inside Meta, right now:

“Someone interrupted a livestreamed, employee-only presentation at Meta earlier this week with an expletive-filled outburst about “being the company’s bitch,” according to a recording heard by WIRED. The individual then asked the people leading the call to write to a specific Meta AI executive and “tell him that he’s a piece of shit.”

The incident, which took place on a call open to thousands of employees, reflects growing frustration inside the company’s Applied AI team, which was formed in March to support the work of AI researchers at Meta Superintelligence Labs. Three current employees tell WIRED there is widespread dissatisfaction with how Meta assembled the unit of about 6,500 engineers and product managers and the drudgework they allege they have been assigned to improve AI models.

“It’s literally the gulag,” one of the employees claims. “You have zero purpose in life all of a sudden, you barely interact with anyone, you just have these tasks every week.”

There’s more: Meta’s Chief Product Officer, Chris Cox, reportedly admitted to staff that Meta’s upper leadership (the folks above him, meaning the C-level at Meta) created the mess. Also from Wired:

During a meeting this week open to all employees at Instagram, Meta chief product officer Chris Cox addressed the “difficult” and “brutal” environment created by the “insanity of this company” in the past few months, according to a recording heard by WIRED. Cox applauded Instagram employees for launching features and serving around 2 billion users amid what he compared to “running a marathon in the middle of a hailstorm and then, like, your teammate gets replaced and then we’re recording you.”

“It’s like what the fuck,” he said, drawing laughs, before repeating himself. “It is like what the fuck.”

6. Self-inflicted wounds

So, is there an ultimate source of the “insanity of this company”, as CPO Chris Cox put it? Engineers whom I talked to point the finger at two individuals: Mark Zuckerberg and Alexandr Wang. Zuckerberg has full control over the business, and has made the decisions to reallocate a good part of engineering folks to data labeling, to roll out tracking software, and to lay off 10% of staff when Meta achieved record revenue and profits. As the CEO, the buck clearly stops with him.

But it’s hard to unsee that – outside of layoffs – everything that Meta is doing is taken from the Scale AI playbook, and that surely comes from Wang:

• Mandatory keystroke and mouse tracking to generate training data

• Forced data labeling with 4,500+ engineers is to generate high-quality RLHF, surely for Meta’s under-construction coding LLM

• Taking away the best engineers from the heart of the business is surely signed off by Mark Zuckerberg, believing that it is more important for Meta to train a coding AI than it is to operate its core business like Instagram, Facebook or Messenger reliably. Oh, did I mention that on Saturday (12 June) Facebook and Instagram had another SEV0, that is, a full-on outage?

Before all this happened, Meta was on track to overtake Google as the world’s #1 ads business by the end of the year. But for some reason, Mark Zuckerberg decided that building a coding LLM is more important.

Meta’s leadership is now trying to undo all the damage they have done. Wired reports that Meta CTO, Andrew Bosworth, admitted to staff that the AI reorg was atrocious and committed to better communication in the future.

To me, it looks obvious that Zuckerberg doesn’t care how engineers feel about the massive changes, and that Bosworth likely ignored the chaos, all while engineers know for a fact that the next AI model matters more than they do to the business. Bosworth also said that employees will have access to AI coaching tools. Very considerate, given the situation!

Based on all I’ve learned, Meta’s engineering culture is dead because leadership has made it clear that engineering at the company is a cost center.

Needless to say, I hope my assessment is way off, but I’ve seen nothing yet from Mark Zuckerberg and Alexandr Wang – the two executives creating this current mess – to suggest it is. There may be a short time period where, if major changes like data labeling assignments and staff tracking are undone, then things at Meta could return to normal. The longer the current conditions persist, the more tenured engineers will surely leave.

7. Is “AI psychosis” just a Meta issue?

It’s tragic to see a technical founder at Meta so focused on AI that he neglects the engineers who built the heart of his company. But is Meta a one-off exception?

Mitchell Hashimoto (creator of Ghostty, founder of HashiCorp) says he is seeing similar behavior by other founders (emphasis mine:)

“I strongly believe there are entire companies right now under heavy “AI psychosis” and it’s impossible to have rational conversations about it with them. I can’t name any specific people because they include personal friends I deeply respect, but I worry about how this plays out.

I lived through the great MTBF vs MTTR (mean-time-between-failure vs. mean-time-to-recovery) reckoning of infrastructure during the transition to cloud and cloud automation. All those arguments are rearing their ugly heads again but now it’s... the whole software development industry (maybe the whole world, really).

It’s frightening, because ‘psychosis folks’ operate under an almost absolute “MTTR is all you need” mentality: “it’s fine to ship bugs because the agents will fix them so quickly and at a scale humans can’t do!” We learned in infrastructure that MTTR is great but you can’t yeet resilient systems entirely.

The main issue is I don’t even know how to bring this up to people I know personally, because bringing this topic up leads to immediate dismissals like “no no, it has full test coverage”, or “bug reports are going down” or something, which just don’t paint the whole picture.

We already learned this lesson once in infrastructure: you can automate yourself into a very resilient catastrophe machine. Systems can appear healthy by local metrics while globally becoming incomprehensible. Bug reports can go down while latent risk explodes. Test coverage can rise while semantic understanding falls. Changes happen so fast that nobody notices the underlying architecture decaying.

I worry.”

The takeover outage at Instagram was exactly like this: the engineering team dropped the quality bar for AI-generated and AI-reviewed code, probably expecting that they could recover quickly from failures. And they did indeed recover… after the damage was done, high-profile Instagram accounts were hacked, and the system was compromised, all very publicly.

Mitchell highlights the specific concern of founders over-estimating the capabilities of AI, and consequently casting aside sensible safeguards when shipping code to production.

Takeaways

Most of us probably have something to learn from the disastrous events at Meta caused by hyper-focus on AI to the exclusion of people who are the lifeblood of that company. In some good news, I’m hearing that in the UK, some of the 10% layoffs have suddenly been cancelled: at the end of the mandatory consultation period, several infra and security teams are learning that no engineers on their team will be let go, as originally expected.

Meta has a booming business, and is already a beneficiary of AI via increased ads revenue. Meanwhile, my Facebook feed is filled with fake, AI-generated videos, with hundreds of comments from bots and people who seemingly don’t realize it’s AI. It all seems like just more content for Meta to show ads next to.

And yet, despite business booming, Meta’s leadership has gone on a crusade to inflict the most damage possible on its engineering org. Apparently, they’re now learning that most of it was pointless.

If you’re in a leadership position and feeling the temptation to make drastic org changes for AI-related reasons, take a deep breath and see where it left Meta. Meanwhile, If you’re an engineer at a company whose leadership is over-indexing on AI, consider forwarding this article as additional context.

If you’re hiring standout engineers who are extremely hands-on with AI, then it’s never been easier to get talent from Meta, than right now. Every engineer I know at the company is an extremely early adopter of AI, and knows how to build products and AI infra. These folks have soured on the company and its leadership. Meta’s loss of talent will be the gain of other startups and the rest of Big Tech; it’s one benefit of AI that’s probably a bit unexpected – not least of all by Meta!

It seems like the old mantra of “move fast and break things” has now extended to Meta’s engineering org itself, with the company’s rush to over-invest in AI, so it will avoid missing the latest mega-trend in the tech industry.

Today, we cover:

1. Anthropic alienates customers with Fable’s data retention and nerf policies. Anthropic’s latest mode, Fable, stores customer prompts and data for 30+ days and performs worse if Anthropic thinks devs’ usage could potentially pose a commercial threat. The launch is an urgent reminder to have an off-ramp from Claude if you want to be able to vote with your feet.

2. New trend: smart model routing. Are there any ‘intelligent’ router solutions out there which select the right model for the right task? I looked into it, and there are a few options. More suggestions welcome!

3. Reliability fail: No automated zone failover for Coinbase’s global trading service. Back in 2016, Uber had a cross-region failover for its core business. Ten years later, Coinbase does not, so it’s little wonder the platform suffered an embarrassing 10-hour outage. The big mess is a head scratcher.

4. Industry pulse. Anthropic and OpenAI file for IPOs, open source project quits GitHub after maintainer banned without appeal, Opendoor “reshores” jobs from India to the US with AI-native engineers, and more.

5. Are LLMs eroding software engineering skills? A software engineer admits they feel increasingly useless due to how capable LLMs are, in an article that has resonated with lots of folks. My sense is that we give too much credit to LLMs, while underestimating our own capabilities and understanding.

1. Anthropic’s new model release is a reminder to have an off-ramp plan from Claude

Read more