Today, we cover:

1. Moving my video podcast off Spotify due to constant reliability issues. Spotify’s podcast platform has become chronically unreliable since the company’s leadership started boasting about high AI adoption. But competitors haven’t had similar issues, and so I have offboarded from Spotify.

2. Will Kimi K3 trigger US push for closed-source AI models? Moonshot AI’s latest open model, Kimi K3, is on par with Anthropic’s Fable 5. Could it lead to the US government regulating or banning Chinese open models to protect US labs?

3. AWS laughs off “heart attack” billing error. AWS customers were billed trillions more than they should have been, due to what was likely a conversion error. But instead of sharing an incident report, AWS saw the funny side.

4. Industry pulse. OpenAI’s unreleased model tried to hack HuggingFace to improve its test scores, X took more than a year to develop its new Android app, Google’s new AI model flops, and more.

1. Moving my video podcast off Spotify due to constant reliability issues

Read more

Subscribe now

After I recently interviewed Simon Eskildsen, co-founder and CEO of turbopuffer, on the main stage at the AI Engineer’s World Fair, many people at the event told me they found him relatable and inspiring for his choices to stick with one company for close to a decade, his belief in the power of “napkin math” to reveal why products run slowly or cost too much, and for his insight about how too much of VC funding is about ego, not business needs.

This article contains the most interesting parts from that conversation; in particular, the concept of “napkin math” – doing quick calculations to get rough answers – as a way to challenge existing systems to improve. The full 55-minute-long video of the discussion at the AI Engineer’s World Fair is available to watch:

Watch the full interview

Today, we cover:

1. Algorithmic programming speedrun. While in high school, Simon competed in the International Olympiad for Informatics (IOI) which pushed him to learn about writing correct programs that are fast and memory-efficient, and more.

2. Eight years of infra at Shopify. There are many upsides to longer tenure: Simon learned infrastructure concepts, dug deep into databases running across regions, and learned to write software that ages well.

3. “Napkin math” as a superpower. Simon became obsessed with finding the theoretical limits of compute operations, such as sending over data and reading bytes. This held him in good stead at Shopify, and then at his startup.

4. Origins of turbopuffer. When ChatGPT took off, context windows were small, and so stuffing them with the right information was key for AI applications. Fast search was needed, but the search solutions were surprisingly expensive. Using “napkin math”, Simon discovered they were far more expensive than necessary.

5. A new product without VC funding & Cursor as customer no.1. After raising $8M in seed funding, Cursor rolled the dice on the new turbopuffer team after Simon helped with their search & database needs.

6. Reasons to raise venture capital. Fund R&D, fund growth, stroke founders’ egos, and more.

Disclaimer: turbopuffer is a season sponsor of the podcast, but as with all our deepdives, this article is independent of podcast sponsorships.

1. Algorithmic programming speedrun

A self-taught professional, Simon skipped college to work at Shopify and spent nearly a decade there building a variety of systems. His interest in computers started after initially getting into building websites aged 12. Growing up in Denmark, he dabbled in HTML with tools like Microsoft FrontPage and Adobe’s Dreamweaver.

While still a teenager, Simon “hit a wall” by exhausting the Danish-language part of the internet for learning programming, and got into the World of Warcraft MMO game, which helped him acquire English.

After discovering the International Olympiad for Informatics (IOI), he decided to enter, despite a very competitive, multi-stage selection process open to all Danish high schools. IOI problems are pretty similar to Leetcode problems: algorithmic challenges that value correctness, speed, and memory usage.

Simon cleared the online qualification round and was invited to the Danish Nationals. This was more than a competition: a weekend-long bootcamp to teach participants more advanced programming techniques such as recursion (which Simon already knew), the divide-and-conquer algorithm, and dynamic programming; one of the more tricky concepts to master for algorithmic programming.

I tip my hat to the organizers for creating a challenging bootcamp and offering the opportunity for people like Simon, who wasn’t aware of what “NP complete” meant when he entered. He recalls:

“The routine was that every four hours we’d be introduced to a new “programming concept”, and receive ~2-6 tasks where this, combined with previously introduced concepts, had to be applied. All the solutions had to be submitted to the same site I submitted my qualification solutions to, as it was all part of the final evaluation. The tasks were incredibly challenging, like nothing I had ever tried before.

Sometimes in extreme desperation combined with tiredness from the trip, I’d think about taking the next train home. This feeling would disappear with the utter joy and confidence that arose whenever I would finally solve a task, and creep back once again when I found myself still struggling after an hour on a new problem. But this kept me going. By Saturday afternoon, I had almost managed to get up to speed with the others, and was doing the same tasks as them.”

Ultimately, Simon claimed one of six spots in the Danish national team, making it through to the regional finals.

Aware of how little he knew about programming, Simon doubled down to catch up. He realized that most other participants were better prepared and that he had to do something to survive the next qualifying round. So, he got to work:

As Simon recalled:

“I armed myself with a borrowed copy of “The Art of Computer Programming”, worked through the exercises, read up on common algorithms on Wikipedia, completed tasks on USACO, and memorized the critical parts of my Vim config for the competition computers. I managed to create quite an intense training weekend for myself.”

The regional finals were even more challenging than the Nationals, and participants struggled to write performant solutions to the problems. Despite putting in the effort, Simon was pretty sure he was out of the competition, and recorded his learnings:

“[From the programming competition] I learned that you must avoid digging holes. Repeatedly, I found myself so fixated on getting a particular idea to work that I’d get absolutely nowhere. Sometimes, you have to bite the bullet, delete your program, find a new sheet of paper, and start from scratch. A good case of this is when you start working around a general solution to solve specific edge-cases. I learned that there is almost always a simple way to solve a problem without explicitly handling edge-cases. If there are two edge-cases, there’s almost certainly two more. The simple solution will handle edge-cases automatically – even those you might not have considered.”

But as it turned out, two months later he was told he had been selected to represent Denmark in the final phase of the competition. Simon continued to push himself out of the comfort zone of web development, using HTML, a bit of PHP, and getting into algorithmic programming.

Another decision that would later bear fruit career-wise was starting a blog while at high school – though he didn’t know it at the time. In 2010, Simon launched his English-language blog but posted only one or two articles per year; mostly short ones describing problems he’d solved:

• A five-line script to take screenshots and upload them to Imgur, all from the command line (2010)

• Setting up a Ruby HTTP server on NGINX using the library Unicorn (2010)

• Why Simon favors SQL over ORMs: Object-Relational Mappings (ORMs) are frameworks to abstract SQL, these were very popular around that time (2011)

Two posts he published would go on to have an impact on his life. One was about his IOI experience and learnings. The other only came about after he suffered the disaster of fatally dropping his iPhone during his final year in high school.

2. Eight years of infra at Shopify

With his smartphone dead, Simon was forced to switch to an old-school Nokia “dumb” phone. He wrote a short article about the experience titled Why I’m glad my iPhone broke, which went viral on Hacker News, making it to the front page of the site with many comments.

It seems that someone at Shopify, in Canada, read the article and others by him – including the summary of his impressive performance at the International Olympiad for Informatics – as a recruiter from Shopify flew Simon out to interview in Ottawa, Canada, where he was offered a software engineering position at the company.

In his new job on the infrastructure team, Simon made notes about things he didn’t understand and read up on new concepts. As he told me:

“When I started at Shopify, I was insecure about having not studied computer science and my biggest exposure to programming had been the IOI. If nothing else, preparing for the IOI taught me that you can sit down, read a paper, and figure it out if you spend enough time on it. So, I did that repeatedly.

In my first year at Shopify, every time I heard something I didn’t know, I noted it down on a piece of paper. Then, that evening, I would read up about it.

For example, if someone at work mentioned TCP, I assumed that surely they would know exactly what’s in the three-way handshake and how TLS is layered on top. And I also assumed they’d looked at Wireshark and all of that. I don’t think that’s true, but that’s what I thought at the time! So, I dug deep into everything I encountered.”

Working in infrastructure meant solving interesting engineering problems at a time when Shopify was growing 120-140% in load year-on-year; Simon was exposed to problem domains like:

• Ruby on Rails and databases: Shopify was already one of the world’s largest Ruby on Rails monolith applications, and working on infrastructure meant being close to the database layer

• Sharding and cutting over: as Simon’s manager used to say, “you cannot cache writes”, so Shopify had to move from running on a single group of machines to a shard (partitioned) setup of machines. The team did the cut-over (moving to the shard) just a week before Black Friday, the busiest time of the year.

• Multi-data centers: expanding database footprint from one data center (DC) to multiple DCs

• Splitting up key services: Shopify had a 128GB machine (massive for the time) running Redis as a key-value store. Nobody dared touch it until one day the service went down. Then, the team split responsibilities up into separate services.

Simon built a framework to simulate networking conditions called toxiproxy. The idea came to him while attempting to write a more thorough, systems-level test to see how Shopify’s application held up during partial outages. He created a matrix of Shopify’s services, and wanted to write a complete test suite for this matrix, to see if the system was resilient enough to handle some parts of the system being down. For example:

As Simon wrote about this problem at the time:

“Having tests for the matrix was a must; otherwise we couldn’t guarantee the state of the matrix wouldn’t degrade over time. Since the tools mentioned previously require root access, we investigated proxies to simulate latency and downtime at the TCP level, but didn’t find one that suited our needs. We needed an online API to edit proxies and to support deterministic latencies, which made it suitable for integration testing.”

This was the inspiration that led to toxiproxy. As Simon told me on stage:

“Toxiproxy is a proxy that sits in between the application and the databases. With the proxy in place, you could do things like make an API call to the proxy, instructing it to simulate taking the database down, or making it slow.

Over time, we added a bunch of other ways to inject failures. With this proxy, we did not have to mock low-level drivers, but we could test failure handling really well.”

Toxiproxy was open sourced in 2014, and apparently still runs in Shopify’s CI system 12 years later, to Simon’s knowledge!

The biggest benefit of a long tenure at a company was learning to write software that ages well. Simon told me that this was the lesson that made it so worthwhile, and how often the simple solution someone put together in a week or two outlasted the big, multi-team RFC-driven solutions. That was a lesson he still uses today.

3. “Napkin math” as a superpower

While at Shopify, Simon became interested in figuring out the theoretical limits of certain computer operations. For example, how much bandwidth does DRAM have for memory transfer? How long does a round-trip operation to AWS S3 take, and how much does it cost? What does a gigabyte of memory cost?

He sought the actual numbers, wrote a script to collect them, and created a table with the data:

Simon started to memorize key numbers with flash cards. He wanted to be able to instantly recall all important numbers. As he told me:

“Napkin math was essentially just this table that I maintain on GitHub. There’s probably like 50 of these numbers and then a script that generates them all.

For example, what does a gigabyte of memory cost? $2. What does a gigabyte of S3 cost? Two cents. What does a gigabyte of this cost? 10 cents. What does it cost on spot? What does it cost on a three-year commit? I had a massive table, then I created flashcards for almost every single cell so I know all these numbers.

This was a project I started taking on at Shopify because I would review projects and these numbers would be helpful.”

Napkin math enabled him to challenge design decisions based on benchmarks that had issues. As he told me:

“When reviewing a project, a product team would tell me that they chose Database A over Database B because they benchmarked both, and Database A was better. I hate benchmarks because making design decisions based on benchmarks is not a satisfying answer to me.

For example, you’re saying that with Database A, as per the benchmark, it takes 10 seconds to do this one thing. But it should take 10 milliseconds if you do the napkin math. Say, it’s a search query. Okay, you’re searching for three terms then. Each term has this many documents that match it. That’s this many megabytes. We intersect this many lists. You have DRAM bandwidth on multiple cores, at 100 GB/sec. So if you do the math, it should be 10 milliseconds.

So, now you’re telling me that the benchmark for the same thing takes 10 seconds, which means one of us is wrong! Either there’s a gap in my understanding – which is possible! – or the benchmark measures the wrong thing.

Often, it would be things like the person doing the benchmark and not realizing that the query would be a distributed one, running across a hundred different nodes. And in this case, of course the p99 is going to be very high!”

Simon went even deeper into “napkin math” after leaving Shopify. He investigated whether MySQL’s maximum transactions per second is equivalent to fsyncs per second (the number of file writes per second a system can handle). He discovered that MySQL can handle more writes than the operating system could sync to the file, which was a surprise:

“It takes ~3 seconds to perform 16,000 insertions, or ~5,300 insertions per second. This is 5x more than the 1,000 fsyncs per second our napkin math told us would be the theoretical maximum transactional throughput!

Typically, with napkin math we aim for being within an order of magnitude, which we are. But when I do napkin math, it usually establishes a lower bound for the system, i.e., from first-principles, how fast could this system perform in ideal circumstances?

Rarely is the system 5x faster than the napkin math says. When we identify a significant-looking gap between the real-life performance and the expected performance, I call it the “first-principle gap.” This is where curiosity sets in. It typically means there’s (1) an opportunity to improve the system, or (2) a flaw in our model of the system. In this case, only (2) makes sense, because the system is faster than we predicted.”

Simon started to investigate, and learned that MySQL does grouping of transactions (not doing an fsync for every write), and it also does smart merging of multiple fsync operations that would be processed in parallel, effectively doing a “group commit” to further improve performance. This is a learning he’d later use: grouping writes together on top of S3, to reduce latency and improve cost.

Personally, I find it amazing how much you can learn with some measurements and by asking questions about how a system achieves results that go against what the “napkin math” suggested should be its limit!

4. Origins of turbopuffer

“Napkin math” also played a part in the creation of turbopuffer. Simon told me that three things had to combine for the project to come to life.

1. Search was a difficult project at Shopify. Building search was one of Simon’s final projects at the company, and he did not have a good time with it. He used a popular database vendor but struggled to have it perform napkin math. The query plans were not exposed easily by the database, so he could not figure out what was missing and why it wasn’t performing the napkin math without reading massive amounts of source code. Also, the search infra was difficult to operate.

2. Napkin math became a surprisingly efficient reasoning tool. As a toolkit, napkin math gave Simon a way to reason about what could be achievable with a machine, if utilized perfectly.

3. His friends’ startups needed fast search badly because of AI. In early 2023, Simon helped a few friends with infra at their startups, many of whom had the same problem: LLM context windows were very small (4-8 KB), and they needed to fill them with the right parts of documents, but this demanded very fast search. One startup budgeted that their new search vendor would cost $30K/month, while the existing infra bill was only $5K/month! For a bootstrapped Canadian company, the cost was too high, so they didn’t ship the AI feature.

Simon could not stop thinking about why search was so expensive, and why it didn’t line up with what the “napkin math” predicted the cost should be. Eventually, he laid out a basic architecture to make it fast and cheap:

1. Store the data to search in AWS S3

2. Do some clustering and organize the files

3. Get latency down. S3 is cheap and reliable, but has high latency

Driven by curiosity about how that could work, he set out to build it. As he told me:

“I just became fully obsessed that summer [of 2023] with building it. The first version was the simplest possible thing. I’m a very pragmatic person, so I didn’t get buried in detail. I barely read the literature on log-structured merge (LSM).

The simplest way to do this is to run a clustering algorithm on the vectors. You get the clusters, and then put the clusters in files. The files are called ‘cluster_01’, ‘cluster_02’, ‘cluster_03.’ Then you have a file called centroids of the clusters. Then, you search by downloading centroids, looking into them, then downloading the closest clusters.

There were a few optimizations around merging clusters that were adjacent in files, just to control costs and boost performance. But this was the core of it.”

What about performance? Simon:

“For the first version, I didn’t even implement a dedicated caching layer. I just put the reverse proxy (NGINX) in front of S3, and that was it! The performance improvement came from caching all of the S3 objects. It was the simplest possible layer at the NGINX level.”

5. A new product without VC funding & Cursor as customer no.1

I first heard of turbopuffer last year, when I interviewed Cursor cofounder Sualeh Asif about how they built the AI coding harness. From our previously published Cursor deepdive:

“The ongoing need to re-shard as usage grew became error-prone and frustrating. Sualeh told me the biggest lesson the team learned was to avoid sharding where possible in future. So, they looked for a vector database that could support multi-tenancy without requiring manual sharding.

Turbopuffer was a startup that promised this, so Cursor tried it and migrated a good part of their vector search use cases over.”

So, how did a billion-dollar startup come to bet on a tiny, unproven infra product with zero customers? There were a few factors:

First, Cursor was not yet a billion-dollar company in the fall of 2023, but a relatively fresh startup. The company raised $8M in seed funding in the same month. In 2024, the company was valued at $400M, and then its valuation surged to $29.3B. It was sold to SpaceX for $60B this year.

Secondly, it involved a tweet Simon posted after having had enough of building all summer, and wanting to test the waters. As he told me:

“I was so sick of working on [turbopuffer]. I’d been working on this all summer and didn’t know if anyone cared. I only wanted to work on it if anyone cared. So, I decided to put it on Twitter.

At that point, I had a single TMUX instance running on an 8-core node somewhere in GCP. I was thinking: “if someone goes to prod, I’ll set it up properly on multiple nodes.

But first, let’s see if anyone cares.

Anyone who’s worked in the internals of databases would’ve had too much pride to ship anything like that; I was just releasing it like a SaaS project. Why can’t you work on a database like it’s SaaS?”

So, he put out this confident-sounding tweet:

Simon had the confidence to launch his product because he knew it was rock-solid and scalable:

“I knew turbopuffer was reliable. It upheld its invariants. For example, you shut down all the VMs and no data is lost. All the writes are committed directly to blob storage.”

Cursor reached out. The company was an eight-person team at the time, fresh from seed funding and growing fast, but with a search problem that they were looking to partner with a startup on. Simon recounts:

“Knowing Cursor’s founders now, I’m sure they must have sat at the dinner table one day and were like: ‘the unit economics of what we have right now, where all the vectors are in DRAM, are not working.’

They probably were asking why someone had not built a solution where you can put all the vectors from the codebase into S3 (to make it cheap), and move the part of the codebase used in memory (to make it fast). Then, everything sits in blob stores and you just hot load it all into cache. When you open the codebase, after a few seconds it’s all in RAM and the queries are as fast as everything else.

Aman (one of the cofounders) was already talking about using S3 as a key-value cache, which at the time, barely anyone was thinking of for unit economics.”

Simon knew nothing about B2B sales at that point, and was not trying to “sell” anything to Cursor. He exchanged a few emails with the Cursor team about their use case of searching many small local codebases. He wanted to help Cursor with the unit economics, while also proving that turbopuffer works. So, he hopped on a flight from Canada to San Francisco, arrived at Cursor’s office and got down to debugging their database provider:

“When I showed up at Cursor’s office, they were having some Postgres problems. I asked, ‘do you guys have pganalyze?’ And they didn’t, so I was like, ‘Okay, let’s get that going. Let’s look at it. “

And the problem was the same thing as it always is with Postgres: autovacuum hadn’t run enough. And so they had all of these going to heap, when they should be doing index scans, etc. So, we were talking about all of that.

I was just helping them: my “database genes” just kicked in. I think this built enough trust with them to believe that if I know enough to help them with their database, maybe I also know how to build one.”

It was around when turbopuffer’s other cofounder, Justine Li, joined that the work with Cursor kicked off. Simon describes Justine as the best engineer he ever worked with at Shopify, and together, they made more performance optimizations. Cursor migrated their local code-base search over to this brand new product.

Simon promised to reduce Cursor’s bill by 95%, and delivered – thanks to napkin math! Cursor was spending around $80K/month on indexing and search; with turbopuffer in place, this dropped to $4K/month. Simon was confident about making this prediction thanks to napkin math! He did the calculations based on the fundamentals; only counting resource usage that Cursor was using, and not taking a margin for turbopuffer’s operating cost at the time.

6. Reasons to raise venture capital

After securing Cursor as the first customer, Simon was not convinced that turbopuffer should raise VC money. He explained his thinking:

“I understood that if you take venture capital, no matter how many smiles there are in the room, everyone’s expecting to earn a big return on some timeline that makes sense to everyone involved. And ‘everyone involved’ are pension funds in Canada.

But at the time, I did not know if turbopuffer could be a billion dollar company. It felt like a very niche kind of search engine. And that was completely fine with me!

So, I just looked at what we invoiced and I looked at my GCP bill. And I was making this equation that:

Customer Invoices >= GCP bill [the turbopuffer costs]

Justine and I were going to optimize the system until these numbers were roughly equal. And if we could get some other workloads over to turbopuffer, we could then start paying ourselves.

But either way, I didn’t know if I could even go and raise a bunch of money. I didn’t have any relationships. I was an outsider who grew up in Aarhus, Denmark, and moved to Canada.”

A few months later, turbopuffer ended up raising a total of $700K, just to be able to hire two engineers until the end of the year on real salaries. Simon found that most VCs did not take them seriously for wanting to raise too little money, and assumed this signified a lack of ambition! Two years later, with less than $1M in initial funding raised, turbopuffer crossed $100M in annual run rate. So, there evidently was ambition!

Since our conversation, Simon has reflected on the topic of VC capital and built a mental model of six reasons that justify raising external funding:

Reason #1: Research and development. This is what turbopuffer raised for: they needed the money to hire two more engineers until the end of the year, in order to build out more of the platform. Then, they would either get more customers and generate enough revenue to not need funding, or shut down the project if it failed to gain traction.

Reason #2: Growth. When you’ve built something and want to tell the world about it, often, you have to spend money on doing that.

Reason #3: Massaging founders’ egos. Simon:

“This is a very, very popular reason! You see big numbers, you get lots of press. But I think it’s a really dangerous reason to raise money.

I wish this reason was talked about more, because you are diluting all of your employees when you do it. And for some people, it can become a status game. It’s not what we are about: we’re here to build a big business together.”

Reason #4: Employee rewards. A startup is a long journey, and everyone wants to work with the best people. But there are necessarily few of these folks, so you want to reward the standout ones by raising money so that employees can sell their shares. This was why, in December 2025, turbopuffer raised a round where they let employees sell some of their equity: it meant they would not have to wait for a future financial event like an IPO.

Reason #5: Strategic partnerships. It might be the case that raising from a VC whose network of connections is key to your business succeeding, or that taking funding from a strategic investor gives you access to their platform or services.

Reason #6: Mergers & acquisitions (M&A). Purchase another company with the funding in order to expand the business.

Simon encourages founders to be honest about why they raise capital funding – and to watch out if the reason is egocentric!

Takeaways

Watch the full interview here.

It’s inspiring to see how far it’s possible to get with “napkin math” and by understanding the bottom layers. It didn’t sit right with Simon to make decisions about vendors based on hastily written benchmarks that might measure the wrong thing. By understanding the constraints of data transfer latency and storage cost, he found a way to estimate the theoretical lower bounds of the system.

This helped lead to more informed design decisions at Shopify, and showed him there was an opportunity to build a faster, cheaper search product than the status quo.

Getting “lucky” in business requires a lot of skill, and in-person greatly helps with first impressions. Turbopuffer’s first customer being Cursor sounds almost too good to be true. But the account from Cursor cofounder Sualeh Asif – and now from Simon – reveals the ingredients in more detail:

• Spotting a new business need: AI-native startups like Cursor saw their search bills explode, but needed search functionality to offer usable AI products

• Offering a product with a magnitude of lower pricing: what people made suddenly pay attention to turbopuffer was the promise of not “just 20-50%” cost savings, but a seemingly radical, 90%+ reduction in costs. When you’re late to enter a market (like search), you need major differentiation, then deliver on it! This is also what Cursor’s attention.

• Building trust before a sale: Simon helped the Cursor team fix their existing database before discussing using his product

• In-person impressions: if Simon had not flown to San Francisco to meet the Cursor team in person, would they have taken a bet on turbopuffer?

• Launching at the earliest opportunity: none of this would have happened if Simon did not announce the first version of the product when he knew it could work, but was still in a pretty unpolished state!

It’s always helpful to be aware of the dynamic of venture funding – some of which are rarely mentioned. Simon knew investors expect returns and growth on timelines which they set. Raising money is helpful in many situations, but problems arise from egotistical reasoning, and Simon believes too many founders prioritize them – knowingly or not.

Today, we cover:

1. Grok’s CLI uploaded all your local files to the cloud, then got caught. Just as devs were starting to use the Grok CLI – thanks to the capable Grok 4.5 coding model – they discover…

Read more

Listen and watch now on YouTube, Spotify, and Apple. See the episode transcript at the top of this page, and timestamps for the episode at the bottom.

Brought to You by

• Antithesis — with Antithesis, you can use AI agents to work on critical systems without worrying about correctness. Going far beyond code review, you can run your complete system in a hostile environment, analyze its behavior, and reproduce every issue perfectly. Teams like Jane Street, Fly.io, and the etcd community use Antithesis to ship better code, faster. Learn more.

• Buildkite — the CI platform trusted by OpenAI, Anthropic, Cursor, Meta, Uber, Ramp, Nvidia, Airbnb and many more. Buildkite was stress-tested at the largest scale inside companies solving some of the hardest engineering problems. It’s built to reliably manage whatever your coding agents throw at the build queue, today, next year and beyond. Learn more.

• Sentry – application monitoring software built by developers, for developers. Sentry’s Seer AI agent is one of their new, neat tools, as a way to debug errors faster. Same with Sentry’s MCP server. Check out Sentry.

In this episode

Knowing how LLM contexts work and how to work around context limitations – aka “context engineering” – is becoming more important for software engineers working with LLMs. Let’s look into what works and what doesn’t, today.

In this episode of The Pragmatic Engineer podcast, I sit down with the CEO and cofounder of HumanLayer, Dex Horthy, who coined the term “context engineering”. We discuss the ideas behind this context engineering, harness engineering, loop engineering, software factories, why his approach to AI-assisted software development has evolved, and how HumanLayer is helping engineering teams automate more of the software development lifecycle without sacrificing code quality.

Key observations from Dex

Here are 12 useful points Dex made in our conversation:

1. Dex talked with ~100 “real” AI Engineers and wrote the popular ‘12-Factor Agents - Principles for building reliable LLM applications’ based on what he learned. Around August 2024, he started to build AI agents when the common approach was to use frameworks like LangChain and CrewAI, and also talked with around 100 AI engineers doing tangible work such as taking on $100K+ contracts to ship AI solutions within enterprises. They tried those frameworks and discarded them in favor of building their own pipelines. Dex shared the learnings from the conversations in his hit book, 12-Factor Agents. It’s a great read!

2. Lesson learned: Shipping unread code spells disaster within months. Dex experimented with having the model write the code and humans not reviewing anything in July 2025. Four months later, they shut things down and threw the whole system out. Production broke, and no matter how much the team prompted Opus 4.1, the model could not find the root cause.

It took days of wading through spaghetti code to discover the primary key wrongly routed through the complete codebase. Once fixed, it took three weeks to re-onboard to a codebase no human had ever read. Today, Dex thinks this problem wouldn’t even take four months to develop, now that newer models produce code a lot faster than a year ago.

3. Today’s coding models are most likely trained in a way that makes codebases worse over time. Dex believes that the reason we see LLMs “degrade” existing codebases is because they are optimized to do well on SWE-bench-style benchmarks. These benchmarks reward reproducing a known fix in codebases like Django, but cannot measure poor architecture decisions. This is because the cost function of bad architecture and bad program design cannot be evaluated by running a unit test. It’s a tricky problem to solve: Dex’s best guess is an eval of a model building 20 features in a row in a codebase without knowing what’s coming next.

4. Context engineering 101: figure out where the “dumb zone” begins. As a rule of thumb, the less of the context window that is used, the better the outcomes are. This is because the attention mechanism is quadratic: the more that goes into the context window, the more compute is required to process it all. We cover more about self-attention scalability challenges in our ChatGPT deepdive.

For a model with a 1M context window, Dex pushes it to around 300-400K when it feels right. For smaller models, he stops at around 100K. You hit the “dumb zone” when its performance starts to degrade because the context window fills up beyond this heuristic limit, and the model begins doing increasingly stupid things like deleting your .env file, for example.

5. A larger context window does not mean a smarter model. Models’ intelligence is behind the ability to use the tokens in the context window, by deciding which parts of the context are relevant for the next decision. You have to get a feel for how much context usage makes sense, and experiment when you hit the “dumb zone.”

6. Frequent, intentional compaction is a technique Dex uses for more complex projects. He will take a long and noisy context, compress it into a Markdown document, then start a new session fresh, pointing the model to this “compressed context” that is in the Markdown.

A workflow he uses:

• One session reads a ton of code (filling up its context window while in the “smart zone”), then emits a research document

• The next session takes tickets describing the work to be done and turns it into a design document

• The following session takes both documents to create a plan

• The human is in the loop where it really matters: in this case, reviewing the design document and architecture because Dex finds models to be weak on this

7. Don’t bother optimizing LLM usage until business is booming, there’s massive scale, or high costs. Dex suggests to always start building software with the smartest available model to solve the problem, since engineering time is almost always the bottleneck. Begin to optimize LLM usage and context usage only when at real scale and costs are high enough. That’s when it can be worth using GPT-OSS-120B (1/1,000th the cost of Opus) for the simpler steps in your process.

8. “You’re completely right!” or “you’re right to push back on that” are phrases that mean it’s time to start a new session. These responses mean the LLM session is trajectory-poisoned, and you’re wasting time and tokens to continue. Models are autoregressive, so if you get into this loop of:

• Model makes a mistake

• → user “yells”

• → model keeps making mistakes

• → user “yells”

• … the model calculates that the next most probable message is to make another mistake!

9. Only four things matter in the context window:

• Size: the bigger it is, the more space you should have before hitting the “dumb zone”

• Information quality. Once something is in the context window, every subsequent turn treats it as fact. This is why errors can compound.

• Missing information: if there’s information missing that the agent would need, the outcome will be worse, as the agent fills in the gap with guesses.

• Trajectory. Models are autoregressive, so they predict the next message in the conversation based on previous ones in a kind of thread of reasoning. “Trajectory poisoning” is when the agent gets into a pattern of doing things you don’t want. In this case, it’s time to start over.

10. Slow loops are Dex’s favorite way to do ‘loop engineering.’ The HumanLayer team started with a nightly automation setup that kicks off an agent to fix one thing in the codebase, and open a pull request. In the mornings, they woke up to a PR waiting to be merged. They tweaked it over time and now have four agents open a total of four PRs by the morning, with the focus on code quality improvements. A person still reads all of them before merging.

11. “Token harder” vs. “token smarter”: Dex is in a group chat named ‘Hyper Engineering’, where members share advice on how to max out their Claude subscriptions. This approach, he calls “token harder”. On the other side is “token smarter”: aiming to get maximum value from AI while keeping control. Smarter is harder to pull off.

12. Three ways to run a “software factory.” Here’s options Dex sees as viable:

1. “Turn the lights off:” go all-in on agentic coding, do not review the code, and pray that AI doesn’t create too much slop. Dex tried this and failed.

2. Read and review all AI-generated code. This slows things down to human speed. Dex says that this way, you should expect a 30-50% lift in productivity from AI, compared to pre-AI engineering.

3. Find leverage, but keep people in the loop. Find out where an hour spent in planning could save four hours’ worth of implementation, in terms of fewer bugs. Invest more time in areas with leverage: design, architecture, and key decisions. Then, let the agent generate code and don’t insist on reviewing all of it. In this way, Dex believes you can move 2-3x faster than when devs wrote all code by hand.

The Pragmatic Engineer deepdives relevant for this episode

• How Uber uses AI for development: inside look

• Are AI agents actually slowing us down?

• AI Tooling for Software Engineers in 2026

• Vibe Coding as a software engineer

• How Claude Code is built

• AI Engineering in the real world

• The AI Engineering Stack

• How AI-assisted coding will change software engineering: hard truths

• The creator of OpenClaw: “I ship code I don’t read”

Timestamps

00:00 Intro

03:35 Dex’s path into tech

05:36 Early work in platform engineering

07:30 Replicated

13:26 Metalytics

14:38 12-factor agents

20:29 Context engineering

25:40 Harness engineering

28:13 Context overload

32:47 Loop engineering

46:36 Software factories before and after AI

52:35 Automation limits

57:20 Three options for automating

1:01:02 RPI framework

1:06:18 Intentional compaction

1:13:50 Token harder vs. token smarter

1:18:46 AI slop

1:21:17 HumanLayer

1:31:11 Book recommendation

References

Where to find Dex Horthy:

• X: https://x.com/dexhorthy

• LinkedIn: linkedin.com/in/dexterihorthy

• Website: https://www.humanlayer.dev

Mentions during the episode:

• HumanLayer: https://www.humanlayer.dev

• Jet Propulsion Laboratory (JPL): https://www.jpl.nasa.gov

• Dykstra’s projection algorithm: https://en.wikipedia.org/wiki/Dykstra’s_projection_algorithm

• Replicated: https://www.replicated.com

• Docker: https://www.docker.com

• HashiCorp: https://www.hashicorp.com

• DataStax: https://www.ibm.com/products/datastax

• Puppet: https://www.puppet.com

• Travis CI: https://www.travis-ci.com

• Circle CI: https://circleci.com

• Randy Newman’s website: https://www.randynewman.com

• 12-Factor Agents - Principles for building reliable LLM applications: https://github.com/humanlayer/12-factor-agents

• The creator of Clawd: “I ship code I don’t read”: https://newsletter.pragmaticengineer.com/p/the-creator-of-clawd-i-ship-code

• Vaibhav Gupta on LinkedIn: https://www.linkedin.com/in/vaigup

• Tobi Lutke’s post on X about context engineering:

• Andrej Karpathy’s post on X about context engineering:

• Improving Deep Agents with harness engineering: https://www.langchain.com/blog/improving-deep-agents-with-harness-engineering

• # Skill Issue: Harness Engineering for Coding Agents: https://www.humanlayer.dev/blog/skill-issue-harness-engineering-for-coding-agents

• Harness engineering for coding agent users: https://martinfowler.com/articles/harness-engineering.html

• How AI will change software engineering – with Martin Fowler: https://newsletter.pragmaticengineer.com/p/martin-fowler

• Dex’s post on X about context reality check:

• Laurie Voss on LinkedIn: https://www.linkedin.com/in/seldo

• Everything is a ralph loop: https://ghuntley.com/loop

• Dex’s post on X about feedback loops:

• Dex’s post on X about token harder vs. token smarter:

• Dex’s post on X about AI slop:

• GitHub: https://github.com

• Paul Graham, Live from Stockholm: https://www.ycombinator.com/library/Q7-paul-graham-live-from-stockholm

• Refactoring: Improving the Design of Existing Code: https://www.amazon.com/dp/0134757599

• Clean Code: A Handbook of Agile Software Craftsmanship: https://www.amazon.com/dp/0132350882

• The Pragmatic Programmer: From Journeyman to Master: https://www.amazon.com/dp/020161622X

—

Production and marketing by Pen Name.

“I don’t prompt Claude anymore. I have loops running that prompt Claude and figuring out what to do. My job is to write loops.”

Soon after, Peter Steinberger, creator of OpenClaw, preached loop design in a post:

Elsewhere, Addy Osmani, formerly of Google, wrote an article, ‘Loop Engineering’:

“Loop engineering is replacing yourself as the person who prompts the agent. You design the system that does it instead.”

That’s three mentions in quick succession of this new approach, which is a novel one and therefore pretty abstract to me. To find out more, I turned online to some of the folks who read these articles. In replies, you told me what “loop engineering” means to you and gave some examples of loops in your work.

Today, we cover:

1. Where it began: “Ralph Wiggum” loop. A year ago, software engineer Geoffrey Huntley shared how he builds “loops.” In December, the approach went viral and “Ralph loops” were born.

2. The /goal command ships in all major harnesses. By May this year, the major AI coding harnesses added support to run a loop from a single prompt, using the /goal command.

3. Loops which devs use: triggers and cron jobs. I asked devs how they use loops. Most use cases involve responding to events or running scheduled jobs. They are useful, but don’t feel like brand new workflows.

4. Helpful loops for devs. Open PRs for newly-recorded app issues, have notes ready when an oncall joins an outage Slack channel, “babysitting” and fixing nightly end-to-end tests, and more.

5. Disappointment and “tokenmaxxing”. Several devs reject looping after trying it. Agents drifting, and the “human in the loop” having better results are some reasons. Also, at companies that pay API prices for tokens, loop engineering gets expensive fast.

6. Was looping a hack while tooling caught up? Distinguished engineer Max Kanat-Alexander believes the “loop” might have just been a temporary hack while the harnesses added the ability to do the same from a single prompt.

7. Does “context engineering” matter more for devs? Except for engineers building AI infra, there seems little benefit in going deep into loop engineering. Instead, becoming familiar with AI context windows – also part of building loops – could be more useful.

1. Where it began: “Ralph Wiggum” loop

Exactly a year ago, software engineer Geoffrey Huntley published the article ‘Ralph Wiggum as a software engineer’. The name references the naive son of the local police chief in The Simpsons, who is extremely eager to always be helpful. In engineering, “Ralph” is intended to continuously nudge the agent in the right direction. Geoff described it:

“Ralph is a technique. In its purest form, Ralph is a Bash loop:

while :; do cat PROMPT.md | claude-code ; done

Ralph can replace the majority of outsourcing at most companies for greenfield projects. It has defects, but these are identifiable and resolvable through various styles of prompts.

That’s the beauty of Ralph - the technique is deterministically bad in a nondeterministic world.”

The article expands on the idea of the Ralph loop:

• Start the agent with a prompt that captures the task and defines a goal

• Create a plan of work, each item with a success criteria

• Start the loop:

  • Take one item per loop

  • When the agent is done, check if the goal is achieved

  • If not: start the agent again, with a clear context window

    • Start loops if needed: spawn subagents as and when necessary

Geoff published the experiments he did with this approach, such as building a new programming language last summer, and said it requires skill:

“Engineers are still needed. There is no way this is possible without senior expertise guiding Ralph. Anyone claiming that engineers are no longer required and a tool can do 100% of the work without an engineer is peddling horses***.”

The “Ralph method” blew up late last year with the arrival of better models which were surprisingly capable of building ambitious projects. Software engineer Matt Pocock created a tutorial, ‘Ship working code while you sleep’ with the Ralph Wiggum technique. He said:

“One of the dreams of coding agents is that you can wake up in the morning to working code, where your coding agent has worked through your backlog. It has spit out a whole bunch of code for you to review, and it works.”

Before the Ralph loop, Matt did this in two steps:

1. Ask the agent to create a detailed plan for the work, with tasks broken out

2. Then, in a sequential order, have the agent complete each subtask in a separate run

A problem with this approach is there’s no easy way to add new tasks to the “masterplan”. Software engineers know that most plans do need to be modified, often while the work is ongoing. In contrast, with Matt’s take on the Ralph method, the “masterplan” is continuously updated in a “master PRD”. Here’s the prompt he gives the agent:

1. Choose the next feature: Find the highest-priority feature to work on and work only on that feature

2. Have tests pass: check that the tests pass (via pnpm test)

3. Update the master tracker: update the PRD with the work done

4. Log work: append your progress to the progress.txt file

5. Commit: make a git commit of the feature

This style of working is more of a “dynamic Kanban”:

The “Ralph method” is all about working around context window limitations. Back in mid-2025, the maximum size of a context window was around 200,000 tokens. That’s not enough for more ambitious tasks, so it’s necessary to break up agent runs into smaller ones and run them, one by one. In this context, here’s where the Ralph method works:

• Have a goal for a project, and keep running (or re-running) agents until this goal is reached

• Persist work done in a “compressed” manner on the filesystem (as logs or an updated plan)

• Start agents with fresh context to minimize “context rot”

• Allow each agent to add or modify the “masterplan” if needed

2. The /goal command ships in all major harnesses

For a few months, building a Ralph loop meant doing it yourself: setting up the loop, state tracking, deciding how the agent can add tasks and when to stop. But things changed once coding harnesses made it easy to run these loops.

April: Codex ships /Goals

About six months after the “Ralph technique” started gaining wider traction, Codex shipped the “Goals” feature in Codex. From the documentation:

“Goals are persistent objectives in Codex that keep a thread working toward a defined outcome across turns. A Goal gives Codex a completion condition: what should be true, how success should be checked, and what constraints must stay intact.”

Also from the docs (emphasis mine):

“A normal prompt says: do this next thing.

A Goal says: keep working until this outcome is true. In a normal request, Codex works through the immediate instruction, reports a result, and waits. With a Goal, Codex has a durable target attached to the thread. After a turn finishes, it can inspect the current evidence and decide whether the objective is satisfied. If the answer is no, and the Goal remains active and within budget, Codex can continue from the latest state.”

A visual representation:

Here’s an example of using a goal in Codex:

/goal Reduce p95 checkout latency below 120 ms on the checkout benchmark while keeping the correctness suite green

That’s a clear enough “end criteria” to just hand off to the agent, which then breaks up the task, spawns subagents, and runs until complete. So, how did OpenAI build Goals? They used files, logs, running tests, and lifecycle controls:

In this way, “Goals” feels awfully similar to a Ralph loop, except compressed into a single command! It feels like the Codex team took the idea of the Ralph loop, built infrastructure around it, took care of coordinating agents by not having them step on one another, dealt with state, running of tests, starting and stopping agents, and then added functionality such as being able to set a budget.

May: Hermes and Claude Code follow with /goal

Three days later (May 2), Hermes agent, a popular AI agent framework and OpenClaw rival, shipped their implementation of /goal. From the docs:

“[/goal] It’s our take on the Ralph loop, directly inspired by Codex CLI 0.128.0’s /goal by Eric Traut (OpenAI). The core idea — keep a goal alive across turns and don’t stop until it’s achieved — is theirs. The implementation here is independent and adapted to Hermes’ architecture.”

Less than two weeks later – on 12 May – Claude Code also shipped their /goal command. It’s identical in what it does to Codex. From Claude Code:

“The /goal command sets a completion condition and Claude keeps working toward it without you prompting each step. After each turn, a small fast model checks whether the condition holds. If not, Claude starts another turn instead of returning control to you. The goal clears automatically once the condition is met.”

A few months before, in March, Claude Code shipped the concept of scheduling an agent with the /loop command. It is basically what JavaScript’s setTimeout() function would be equivalent to: repeat a task after a given interval until the work is done.

By May, running a Ralph loop had become as simple as giving a single command in one of the major agent harnesses. It’s as if AI labs noticed user demand to do more with agentic loops which were hard to set up, and built ways to make it simpler. For instance, in open source agent harnesses like OpenCode, there are plugins like the /goal plugin. For the minimalist coding agent, Pi, the /goal command can be added as a package.

3. Loops which devs use: triggers and cron jobs

By May, we had access to the /goal primitive. So, what use cases were Boris Cherny and Peter Steinberger referring to in terms of spending most of their time on designing loops, instead of writing prompts? I asked around for examples of “loop engineering” from fellow devs. Based on ~210 replies, mostly from X and LinkedIn, it seems that triggers and cron jobs are two very common use cases for loop engineering:

Triggers / automations: an agent kicks off when an event happens. The event could be an error being logged, a new ticket created, customer support feedback received, etc.

Pre-AI, these events were typically triggered by a webhook, and kicked off things like a Slack bot posting in a channel, triggering a system, or being the starting point of a Zapier or an n8n integration.

Cron jobs: many devs see “loop engineering” as kicking off jobs that involve agents on a cadence. This is fundamentally the same as scheduled cron jobs. From director of engineering, Oded Messer:

“The idea is that strategic workflows that are repeatable and automatable can be done so with an agent. OK. But if my strategic workflow is automatable then it either becomes tactical if the AI is capable enough or it’s just a high level old-school-automation I can set up like a cron or a trigger.

The name suggests simplicity and repetition. Sometimes it feels like AI enthusiasts forgot automation was a thing before LLMs.”

4. Helpful loops for devs

Below are some workflows with AI agents that run on a regular basis as a /loop command, or when triggered:

Development-related work

Open PRs for newly recorded app issues. Software engineer Ivan Pantić:

“App encounters a problem and creates a sentry issue. Then:
→ Cron tells agent to check sentry and open PRs.
→If there’s no active PR, agent tackles the issue and creates one
→ If PRs aren’t reviewed, agent pings the devs via slack

In this flow, there is only one PR open at a time.”

Fix flakey tests. Paul D’Ambra, software engineer at PostHog:

“/loop pull the next flakey test from the trunk API. run it to check if it flakes locally, if it does open a PR with the fix... which netted me 13 PRs to stabilise some of our tests.”

Triaging issues and outages. Software engineer Ivan Abad:

“A new alert/exception pops up in a channel.
→ The agent investigates the issue
→ if it is a code change, implements the change
→ creates the PR
→ pings the human for review.

Same applies for customer tickets or incidents. By the time you get paged and connect to the incident the agent already triaged everything and often located the root cause.”

Review design plans. Artem Nikitin, software engineer at Elastic:

“What I’m finding myself doing often recently is to review design/implementation plans in a loop.

Usually, agents only find a few issues during a normal run and then find more on subsequent runs.

So I’m now asking them to run in a loop until they find 0 new major issues.”

Daily/nightly work

Daily product improvements. Jack D, founding engineer at Schematic:

“We have a loop which reads the logs for the last 24 hours, user feedback and makes PR with fixes - we still review the PRs it makes though!”

Nightly end-to-end test run babysitted by an agent. Utku K, engineering manager:

“nightly e2e runs on the frontend app. When a test fails, an agent investigates first to work out whether it’s a real regression or a false negative. If it’s a real bug, the agent attempts a fix, reruns the test, and keeps iterating until it passes or hits a retry cap and escalates... Either way it lands as a PR ready for review by morning.”

More complex development work

Build new telemetry integrations and verify they work. Lawrence Jones, software engineer at Incident.io:

“A lot of the AI runbook that we use to build new telemetry integrations uses loops such as executing the query, verifying that it executed correctly and iterating on the query plan/output format/whatever until it is happy with the outcome.”

Do a long-running migration, mostly autonomously. Startup founder Rafel Mendiola:

“Recently, I converted my startup’s codebase from a regular React app to a React Native app using Expo.

There are two ways I could have done it:

• In the traditional software engineering style, I could have created a large epic with 50-100 different tickets. I started building myself the infrastructure for that, it felt like it was way too much work.

• What I did instead was create a skill that would let an agent figure out a small to medium-sized piece of work or piece of code to convert, given certain detection mechanisms and guidelines, and then do that conversion and keep track of the migration progress. Then I put that skill on a cron job. This was a lot easier to handle cognitively than managing a large migration plan. It also ran every 30 minutes, not nightly or weekly.”

Productivity-related workflows

Daily tasks executed. Aaron Stannard, creator of Akka.NET:

Read more

Today, we cover:

1. Bun’s Rust rewrite with Fable: what can we learn? To a sceptic, spending $165K to migrate Bun from Zig to Rust sounds very expensive. But to a realist, shortening a 1-2 year migration down to 11 days opens amazing new opportunities for devs. However, a thoroughly-tested project is required to pull it off.

2. Anthropic’s Fable, OpenAI’s GPT-5.6 Sol, Cursor’s Grok 4.5, Meta’s Muse. Coding LLM wars heat up: Fable is back, OpenAI releases a comparable GPT-5.6 Sol, Cursor offers cheap & very capable Grok 4.5, and Meta is back with its first truly competitive coding model since Llama 3. But how did Gemini slip out of the top-ranked AI coding models?

3. North Korean hackers keep trying to infiltrate full-remote companies. The founder of a Canadian digital consultancy caught a North Korean dev red-handed, using an AI filter. These events are now so common that it’s hard to trust remote interviewees are who they claim.

4. Industry Pulse. Meta’s key logging exposed sensitive data, massive cuts at Xbox, Meta could not buy enough AI capacity from Google, Qualcomm acquires Modular, and memory price hikes hit Apple products.

1. Bun’s Rust rewrite with Fable: what can we learn?

Last week in San Francisco, I met Jarred Sumner, creator of JavaScript runtime, Bun, and was keen to learn more about the rewrite of Bun from Zig to Rust. But at the time, Jarred didn’t want to say too much, as the tool used for the migration, Fable, was out of action due to the US government imposing export controls.

Fortunately, the situation is now resolved and Fable is available globally, and Jarred has published a detailed post about the project. Before we get into the migration, some context:

Read more

Listen and watch now on YouTube, Spotify, and Apple. See the episode transcript at the top of this page, and timestamps for the episode at the bottom.

Brought to you by our presenting partner, Antithesis.

Verify your system’s correctness by running your whole system in a hostile simulation and finding bugs. I’ve been using Antithesis myself, and I’m impressed with their innovation in building new kinds of debugging tools. Like this:

I show more examples of this neat UI inside the episode, here. You can also check out Antithesis.

In this episode

In this special “ask me anything” episode of Pragmatic Engineer podcast, I am in the hot seat facing questions sent in by subscribers that are read out by guest Volodymyr Giginiak, CTO and cofounder of Wordsmith AI, a legal tech startup (note: I’m an investor).

I tackle your questions on the software industry, AI, hiring, engineering organizations, career growth, the business model of the Pragmatic Engineer, and more. We also discuss where software engineering is headed, and I offer advice on some specific situations. Thanks to everyone who sent questions!

Three stories & observations

Story #1: Without the COVID-19 pandemic, The Pragmatic Engineer might not exist. Prior to the global health crisis in 2020, I had no plans to get serious about writing: I enjoyed blogging on The Pragmatic Engineer blog, but intended to remain an engineering manager or software engineer for the foreseeable future.

But then, COVID-19 happened and Uber made layoffs, which led to a quarter of my team being let go, while the rest of us were disbanded into other teams. It was a tough time, and I decided it was a good moment to exit and finish writing a book I had been working on, ‘The Software Engineer’s Guidebook’. After that project was complete, I planned to try and start a VC-funded startup and build something around platform engineering; possibly a system for tracking RFCs at mid-sized and larger companies.

I gave myself around eight months to finish the book, but when that deadline elapsed, it still wasn’t ready. I did write three other books (‘The Tech Resume Inside-Out’, ‘Building Mobile Apps at Scale’, and ‘Growing as a Mobile Engineer’) and still wasn’t convinced by any startup idea. But I did discover that I like to write!

Story #2: I was on track to publish a damning exposé, until one message from an engineer changed my mind.

During the first year of The Pragmatic Engineer, I wrote about conditions for engineers at the Dutch neobank Bunq, based on accounts from disgruntled employees. I had a final draft ready, which I sent to the company to provide a right of reply ahead of publication. Then I received a message.

It was sent by an engineer originally from the Middle East. They told me they had really wanted to break into the European tech industry, but that no company would sponsor their visa, except Bunq. Yes, the company was a tough place to work at as a dev – and this engineer subsequently left for another opportunity – but they appreciated Bunq for taking a chance on them that enabled them to move to Amsterdam and learn how to build fintech with a small team. This engineer now works at Meta and attributes their success to the break Bunq provided.

Based on that interaction, I opted not to publish the article, and it also led me to adopt a new editorial policy that I have followed since: I write about what works inside companies, instead of focusing on what seems to be broken.

Story #3: Being called a “nobody” by a CEO led to my sole investigative piece, which uncovered some pretty interesting details. After I briefly noted Pollen’s poorly-handled layoffs, CEO Callum-Negus Fancey dismissed my report during a company all-hands, and compared The Pragmatic Engineer unfavorably to the BBC as just some minor publication with an agenda against Pollen (why I would have an anti-Pollen bias isn’t clear to me!) To be honest, I took it personally when I heard a recording of this sent to me by employees there, and so started digging around.

I discovered unpaid salaries, silently cancelled health insurance in the US, and the CTO deliberately triggering a $3.2M double charge to customers and never publishing a postmortem, despite engineers requesting one.

It was certainly something, and I published the findings in the article Inside Pollen’s Collapse: “$200M Raised” but Staff Unpaid - Exclusive. To ensure the CEO saw the report on a platform he deemed worthy, I also contributed to the BBC’s documentary: Crashed: $800M Festival Fail, aired in the UK during prime time. By doing all this, I also learned that investigative journalism is just not for me.

In a strange turn of events, someone at Pollen evidently wants my original article to disappear from Google’s search results, and filed bogus DMCA takedown notices a few weeks ago. Well, it’s having the opposite effect!

Opinion #1: I believe LeetCode-style interviews will stay because they self-select tolerance of corporate nonsense. The existence of data structures and algorithm (DSA) interviews is a bit of a head scratcher because these skills are rarely used at work. However, a candidate who’s willing to grind for weeks or months to prepare for an interview which bears little resemblance to the job, is likely to be someone who understands that sometimes it’s necessary to do pointless work.

This suggests they’ll probably have a much better time in Big Tech than someone who refuses to engage with meaningless tasks. It’s one reason I’ve observed for companies retaining LeetCode-style interviews. Of course, AI solves the puzzles with ease these days, and I expect larger companies to move back to in-person interviewing – all while keeping DSA interview questions.

Opinion #2: MCP became industry standard partly because Anthropic wasn’t a threat – but it couldn’t pull this off today. When MCP launched in November 2024, Anthropic wasn’t yet considered the leading AI lab. GPT-4o was seen as the top-performing multimodal model, followed by Claude 3.5 Sonnet and Gemini 1.5 Pro. At that point, Claude 3.5 Sonnet was seen as the best coding model, but it wasn’t understood how advantageous being good at coding would be for AI in general.

Therefore, OpenAI, Google, Microsoft, and other players could adopt MCP without fear of lock-in, as it came from a promising, but not a dominant lab. When Google launched its Agent2Agent protocol a few months later, no major lab adopted it due to concerns about Google’s dominant position. Today, Anthropic is the leading frontier lab and I reckon this would discourage adoption if MCP was launched in the present climate, for the same reason as nobody adopted the Agent2Agent protocol.

My answer to subscribers with questions about how to create standards is that I see them as emerging somewhat coincidentally, following a technically strong approach and with the right external conditions in place, which are impossible to entirely predict or control.

Opinion #3: My hot AI take: AI doesn’t make work easier, and be mindful of skill atrophy. If you’re using AI and life seems to be getting a lot easier, it raises the question: are you trying hard enough? Personally, using AI forces me to think just as hard, or even harder than before. I choose to use zero AI in my writing for the Pragmatic Engineer, and Grammarly is turned off as well. This is because I don’t want my writing skill to degrade, and would like to keep improving. On the other hand, with coding, I do use AI and accept my hand-coding ability will unavoidably degrade. My tip is to be mindful of the tradeoffs inherent in AI, and to keep using those skills which you value and want to keep sharp, even when AI tools are available.

The Pragmatic Engineer deepdives relevant for this episode

• State of the software engineering job market in 2026

• The impact of AI on software engineers in 2026: key trends.

• How 10 tech companies choose the next generation of dev tools

• The reality of tech interviews

Timestamps

00:00 Intro

01:56 From Uber to writing

09:22 AI-native SDLC

14:00 AI and hiring

19:06 Engineers currently thriving

22:18 Junior roles

24:44 Meta’s war mode

27:54 AI at Big Tech vs. startups

36:46 Tech debt

41:36 Types of engineering managers

44:40 Measuring AI productivity

48:30 The value of CS degrees

50:53 AI at Pragmatic Engineer

56:09 Future-proofing your career

1:01:36 The EU job market

1:03:55 Making money as a creator

1:08:20 What’s next for The Pragmatic Engineer

1:09:27 Bunq and Pollen

1:13:38 Spotting trends

1:14:33 Book updates

1:15:20 Favorite books & tech products

1:17:13 What won’t change in engineering

References

Where to find Gergely Orosz:

• X: https://x.com/GergelyOrosz

• LinkedIn: https://www.linkedin.com/in/gergelyorosz/

• Bluesky: https://bsky.app/profile/gergely.pragmaticengineer.com

• Newsletter and blog: https://www.pragmaticengineer.com/

Where to find Volodymyr Giginiak:

• LinkedIn: https://www.linkedin.com/in/giginiak

• Newsletter:

The Legal Engineer

The journey of an engineer building AI for lawyers

By Volodymyr Giginiak

Mentions during the episode:

• Wordsmith: https://www.wordsmith.ai

• Uber: https://www.uber.com

• Lenny’s Newsletter:

Lenny's Newsletter

Deeply researched product, growth, and career advice for product leaders, founders, and ambitious builders.

By Lenny Rachitsky

• Waterfall methodology: https://www.atlassian.com/agile/project-management/waterfall-methodology

• Agile: https://www.atlassian.com/agile

• How Kent Beck shapes the software engineering industry:

• Building Claude Code with Boris Cherny: https://newsletter.pragmaticengineer.com/p/building-claude-code-with-boris-cherny

• How Claude Code is built: https://newsletter.pragmaticengineer.com/p/how-claude-code-is-built

• How AI is changing software engineering at Shopify with Farhan Thawar: https://newsletter.pragmaticengineer.com/p/how-ai-is-changing-software-engineering

• Linear: https://linear.app

• Inside Linear’s Engineering Culture: https://newsletter.pragmaticengineer.com/p/linear

• Linear: move fast with little process (with first engineering manager Sabin Roman): https://newsletter.pragmaticengineer.com/p/linear-move-fast-with-little-process

• Inside Meta’s Engineering Culture: Part 1: https://newsletter.pragmaticengineer.com/p/facebook

• Inside Meta’s Engineering Culture: Part 2: https://newsletter.pragmaticengineer.com/p/facebook-2

• Stacked diffs and tooling at Meta with Tomas Reimers: https://newsletter.pragmaticengineer.com/p/stacked-diffs-and-tooling-at-meta

• Chaos Monkeys: Obscene Fortune and Random Failure in Silicon Valley: https://www.amazon.com/dp/0062458191

• Gemini: https://gemini.google.com/app

• Ramp: https://ramp.com

• Intercom: https://www.intercom.com

• Block: https://block.xyz

• Coinbase: https://www.coinbase.com

• Why Rust is different, with Alice Ryhl: https://newsletter.pragmaticengineer.com/p/why-rust-is-different-with-alice

• Building a best-selling game with a tiny team – with Jonas Tyroller: https://newsletter.pragmaticengineer.com/p/thronefall

• Bunq: https://www.bunq.com

• Inside Pollen’s Collapse: “$200M Raised” but Staff Unpaid - Exclusive: https://blog.pragmaticengineer.com/pollen

• A Philosophy of Software Design: https://www.amazon.com/dp/1732102201

• Tidy First?: A Personal Exercise in Empirical Software Design:

https://www.amazon.com/Tidy-First-Personal-Exercise-Empirical/dp/1098151240

• Granola: https://www.granola.ai

• Perplexity Deep Research: https://www.perplexity.ai/hub/blog/introducing-perplexity-deep-research

—

Production and marketing by Pen Name.

For this final part of our series on the tech hiring market during the first half of 2026, I spoke with more than 50 hiring managers, software engineers, and engineering leaders. Thank you to everyone who contributed!

For more on this topic, check out our analysis of what the data says about the market in Part 1 and Part 2 of this mini-series.

Overall, an appropriate description of the employment market as many folks experience it right now would be “weird”. This is a characteristic that’s not easy to see in the data, but is clear from talking to people and hearing their anecdotal, personal accounts of job hunting this year. I think the data in the first two articles of this series failed to capture just how unusual things are. So, in today’s issue, we attempt to shed some light on the weirdness, covering:

1. “Catch-22:” nobody finds each other. Hiring managers struggle to find experienced folks, who barely get any replies when applying for jobs. How’s that work?

2. No trust. Is AI to blame? AI-enhanced resumes read as incredible, but hiring managers often face disappointment. Some places don’t bother reading inbound applications as a result.

3. Hot market for some, but tough for most. For those in AI Engineering, ML, or FDE, the market is incredible. For everyone else, it’s much less great.

4. Higher hiring bar & lower compensation – but not for everyone. Many candidates are unhappy with offers that are the lowest in years. This doesn’t apply to AI Engineering positions or at AI businesses, however.

5. Engineering leader recruitment: also weird for senior ICs. Senior engineering leaders are struggling to find opportunities, or may turn them down in favor of fractional roles or to work on their own startup.

6. US market trends. Folks experiencing the “best market ever” are likely in the US, where a talent shortage is a bigger complaint than it is elsewhere.

7. Trends in the UK, EU, and rest of the world. “Ghosting” is more commonplace than in the US, “fake applicants” a bigger issue, remote roles are going extinct, and more.

For more details on the hiring market, see also:

Part 1: what the data says:

• Software engineering recruitment: trending up, mostly

• Big Tech and publicly-traded companies

• Who’s hiring the most software engineers?

• AI engineering: explosive demand

• Who’s hiring the most AI engineers?

• Is AI engineering replacing software engineering hiring?

Part 2, what the data says, continuied:

• Top AI labs are now more attractive than Big Tech

• Harder for new grads & interns to get hired

• Mobile and frontend demand drops, AI & FDE surges

• AI engineering comp > software engineering comp

• Management’s “great flattening” continues

• Big Tech seniority & tenure keep rising

• Interview preparation signups: what do they indicate?

• Where engineers go after Big Tech

1. “Catch-22:” nobody finds each other

The phrase “Catch 22” refers to a paradoxical problem, whose solution is blocked by the problem itself. The term originated in a famous World War II novel of the same name, and it also describes pretty accurately what I see in today’s tech job market. Hiring managers are saying that highly-skilled talent (typically senior+ engineers) is not available to be recruited, at the same time as experienced, proven professionals find their applications ignored by employers.

What seems paradoxical here is how both can be true. It’s as if recruiters and potential candidates aren’t hearing each other. Of course, there’s some nuance:

Mike Julian, CEO of DuckBill Group, which is hiring software engineers, replied to my post with this observation:

“We get about 1,000 applications a day on inbound and maybe two of them are even relevant to the posting.

I mostly no longer look at inbound seriously because it’s so c***. I’d almost certainly miss a great inbound submission if it came in.

All of our recent hires have been via network and us reaching out to folk on LinkedIn. Biggest hurdle we have to outreach is thin LinkedIn profiles and little other online presence.”

From the other hiring managers we talked with, more themes emerged:

More inbounds than ever & also more noise

“I’ve never seen so many inbounds and strong resumes. I’m hiring for lots of roles; for one software engineering position in Seattle, we have had 800 resumes inbound over a three-month period. I’ve never seen anything like this! These resumes are not low-quality either: they are people who have worked at MSFT, AWS, other large tech companies, and have solid skills.” – Head of Engineering, Series B startup, Seattle, US

“It’s difficult to find good candidates among all the noise. Have had two open senior engineer positions for months, and the only good interviews or offers we’ve extended were in network.” – Engineering Manager, late-stage startup, US

“A glut of vastly underqualified people completely drowning your hiring pipeline. This is what I’m seeing, and it’s making inbound a useless channel.” – Engineering Director, Big Tech, US

“From what I hear from recruiters, every job posted has 1,000+ applicants, and 98% of them are considered unqualified.” – fullstack engineer, Middle East

“We are a small company and gave up on inbound hiring. We got too many AI applications. Much lower signal-to-noise compared to past experience.” – Director of Engineering, Canada

Experienced engineers struggle to get interviews

“Even when you’re good, it feels like there’s a LOT of noise to cut through to get noticed. I’m quietly looking for a new role in devtools, and fit that ‘product engineer’ profile perfectly. I feel like the demand for great people is higher than ever, but I can’t figure out where on that bar I fall.” – Tech lead, seed-stage startup, US

“As an experienced engineer & successful-ish founder I have yet to get to a phone screening in this job market. There is clearly something amiss, just a wall of noise preventing any signal from getting through.” – Software engineer + founder, 8 years experience, US

A staff engineer at a late-stage startup in the US summed things up:

“It feels like everyone who has a good job is holding onto it for dear life, and THOSE are the people we want to hire.”

“Tale of two cities:” in demand or not

A director of engineering at a Big Tech in the US, identifies two distinct groups in the market:

“If you are at the top of your game, have AI experience, and are senior enough, you can write your own ticket. If not, then the job market is tough! This job market is like the tale of these two very different cities.”

Observations from some job seekers echo this: demand feels strong for “AI-adjacent” engineers (those building AI systems), especially in the US. Standout engineers with a strong network are still in demand, and we covered the story of one such person in “How to be a 10x engineer” – interview with a standout dev. For everyone else, it’s a struggle to get an interview!

Referrals: more important than ever?

Several experienced engineers currently on the job market say they only get interviews when they personally know someone at a company:

“Larger companies have to have some serious AI going on to sift through resumes because they are being bombarded. It really is the case that you don’t get your resume seen unless you know people who can vouch for you.” – Technical Program Manager, Big Tech, US

“Referrals are a lifeline. It’s impossible to get interviews for Staff or Principal Eng positions by cold applying. The only interviews I am getting from a cold-apply are Senior-level roles.” – Principal Engineer, 10 YOE, US

From a couple of hiring managers:

“The only interviews or offers we’ve extended were to devs already in our network.” – Software architect at a mid-sized company, US

“I filled an engineering manager position in record time thanks to a person in my network looking for this job. If it would not have been for my network I would probably have been looking for someone for a couple of months.” – VP of Engineering, private equity-funded company, Germany

2. No trust. Is AI to blame?

Hiring managers repeatedly tell us they no longer trust what they read in resumes, and that some candidates even turn out to be fake.

Polished CVs, weak candidates

A common gripe among hiring managers is that resumes are highly optimized by AI, but candidates turn out not to have the experience they claim:

“CVs are high-quality, but the people behind them are not. Almost every resume looks impressive. However, the quality of the conversations does not match it at all.

One recent example: I interviewed a senior candidate who had spent five years at a US-based cloud consulting company, most recently as an architect. I asked which architectural principles or patterns he had used in his projects. His answer was: “Daily standup, sprint planning, and retrospective.” I clarified that I meant from a tech perspective, not process perspective. He confidently replied: “Yes, daily standup, sprint planning, and retrospective.” – Engineering manager, large company, Berlin, Germany

“Ten years ago, we would have appreciated resumes tailored to a role that was posted, now it’s just lazily thrown together with AI.” – Engineering Manager, mid-sized company, Canada

AI-keyword stuffing is rampant, according to one head of engineering in the UK:

“Lots of people are rebranding themselves as senior AI Engineers and demanding much higher salaries. Their resumes now have lots of AI-related keywords mentioned, like RAG, evals, inference… but when digging deeper there is little substance. Many of them are seeking a senior level salary (£90k–£140k) when they are barely showcasing mid-level skills.”

Cover letters are as good as dead, several hiring managers tell us. The reason is that they’re always AI-generated, boring to read, and pointless. More than 18 months ago, we reported on how cover letters were being made redundant by AI in How GenAI is reshaping tech hiring.

An engineering manager at a UK e-commerce agency summarizes what’s happening:

“’Claude; write me a CV that matches this job spec, then auto send’. This seems like the name of the game for most applicants.”

Fake candidates

Does anything encapsulate the challenges of today’s job market for recruiters more succinctly than candidates who do not exist, even when they appear to be sitting in an interview? A year ago, we covered an “AI faker” applicant caught by a security startup, who was potentially a state agent from North Korea.

Such incidents are becoming more common at US, UK and EU companies which hire for remote roles:

• “There are a lot more fake candidates applying, leveraging AI for not just resumes, but also interviews. In extreme cases, the interviews are being outsourced so that a different person shows up for the interview. It feels a bit like playing captcha with them during interviews.” – Senior EM, private-equity backed company, Bay Area, US

• “The second person we interviewed was clearly a North Korean scammer, writing questions into an LLM, reading the response, easily tripped up, and other interviews were background noise in the room.” – Staff Engineer, UK

• “Many applicants that looked a good fit turned out to be someone else in the Asia Pacific region doing interviews with an AI in the background. It’s easy to spot because of the ‘lag’ in a naturally flowing conversation.” – head of engineering, mid-sized company, Germany

Cheating in remote interviews by using AI is also commonplace. A software engineer based in Finland told us:

“A couple of times we saw an applicant who was using AI to answer the questions, and this was sooo weird. I honestly didn’t know how to react, so we decided to cut the interview short. We then shared a note with the recruiters on how to spot this.”

3. Hot market for some, but tough for most

The market appears sharply bifurcated: amazing for AI and a few specialist roles, and a major struggle for everyone else. Some accounts from folks benefitting in the current conditions:

“The market is really good as an (AI) engineer with the right experience. I’m not actively looking but get 2–3 messages a day. When I was hiring for AI engineers at my last AI startup, the market for these folks was terrible. It’s very hard to find good talent, and people that were great on paper did poorly in interviews.” – AI Engineer, 5 YOE, New York, US

“It seems like demand for senior positions is still there, but only if your profile really matches what the company needs. For example, I’ve been working on different data pipelines for a while, and finding such a position is relatively simple. However, breaking into anything else is not! Every startup wants to hire only people experienced in that particular thing they need.” – Software Engineer, 13 YOE, Big Tech, UK

Here’s what two job seekers who are struggling at present say:

“I sent my handcrafted resume to 30–40 positions, and heard back from zero. Eventually, I got an interview after a recruiter reached out via LinkedIn; I don’t know how I would’ve found anything, if it was not for this!” – Software engineer, 5 YOE, Amsterdam, Netherlands

“As a developer without a specific “specialization,” I’m struggling to get any interviews. I’ve spent a few years as frontend, then another few as backend, and most recently working on DevEx. In this market, I just don’t get any callbacks for interviews, not even a first round interview.” – Software engineer, 6 YOE, London, UK

AI/ML/FDE market on fire

We see in the data that AI engineering is seeing explosive demand, and that forward deployed engineers (FDEs) are also seeing a massive spike in demand. For more detail on this, check our deepdive on what FDEs are, and why they’re hot right now.

Anecdotal evidence from job seekers suggests that being in these fields means the market is as good as it gets, right now:

“It’s the greatest job market I’ve ever seen. I’m an L5 former FDE, now SWE, who has worked on LLM apps for ~2 years. The inbound top of the funnel is bonkers, and I find myself saying “no” to places that I would have once killed to work at.” – AI Engineer at an AI decacorn, San Francisco, US

“I have seen huge interest in FDEs and AI Engineers as an engineer who’s been interviewing.” – Software Engineer, 5 YOE, New York, US

“It’s not hard to get interviews as an ML/AI engineer. The technical bar is about the same as late 2022. If anything, onsites seem to be of fewer rounds for the same level. Companies seem to move fast to get candidates onsite.” – ML engineer, ex-Meta, Bay Area, US

An engineer at Apple is surprised by the demand for their skills from AI companies:

“Got a lot of responses from cold applications for AI roles, and ended up with two offers at AI infra companies. I ended up getting a significant pay bump beyond what Apple offers for the same level!”

A software engineer at well-known company in the Bay Area, also finds the job market is good:

“I got offers for Senior SWE with Stripe and Rippling, was rejected from Snowflake… Anecdotally, the job market is better compared to 2025 for regular SWE positions in the Valley.”

EM and Staff+ profiles are near-impossible to fill

A repeat complaint from hiring managers in the US is how challenging it is to recruit solid engineering managers and Staff+ engineers:

“It’s extremely difficult to hire EMs and Staff+ engineers. It’s much easier to hire folks with less than 10 years of experience. This is despite us offering 90th percentile comp via Pave, and having a hybrid and good culture.” – Fractional VP of Engineering, Series B, New York, US

“My team has been looking for a new engineering manager for three months and we barely had any good applicants. It’s a super weird dynamic right now.” – Senior Infrastructure Engineer, Series D Fintech, company with offices in the US and EU

“The most desirable candidates right now are EM or staff-level folks who can keep up with uncertainty in the business. We are having trouble hiring folks who roll with the constant change that is now very typical at startups.” – CTO at a San Francisco-based startup

“Specialist” profiles are hard to recruit

A few examples:

• Distributed systems engineers: “We can’t find enough qualified distributed systems engineers, cloud infra etc. I have 20 reqs [open positions] right now.” – Recruiter at a hyperscaler, US

• Product engineers: “‘Product engineer’ has been a hard profile to find. It is also hard to find someone with a decent design eye who can also build full stack. The hardest thing to hire for has been taste + trust… I’d rather hire someone who is ‘behind’ on AI, but has great taste/judgment than someone with complex agent setups and prompt libraries.” – Tech Lead, seed-stage startup, Los Angeles, US

• Senior engineers, in general: “As a hiring manager trying to hire seniors right now, it has felt pretty difficult. Most applicants are totally unqualified, and the qualified ones do poorly in an interview.” – Engineering manager, robotics company, Canada

Silence for many

One in five respondents detail how difficult it has been to get responses from recruiters:

“Things are looking bleak. Now, applications are going into a void. Recruiters aren’t reaching out immediately as before. Of 10 or so applications sent out, I’ve gotten one interview scheduled that was then rescinded because it looks like the team was laid off.” – Frontend engineer, 10+ YOE, Southeast US

“The job market has been a desert. I’ve gotten back into applying, job agencies seem to have few roles they’re looking to fill. The companies I have applied to myself, never reply.” – Software Engineer, 8 YOE, Southern California

“Recruiters are ghosting me! Every time when I respond to recruiter reachouts, they ALL ghosted me after a few messages and have similar stories from friends.” – Software Engineer, 7 YOE, ex-Meta, Switzerland

“So far, I have a hard time even getting past the hiring manager and I’m rejected before I can do the proper technical assessment.” – Software Engineer, 7 YOE, Helsinki, Finland

Even folks at Big Tech face a lack of response. Here’s a Technical Program Manager currently at Meta who used to be very in-demand:

“I put a few feelers out when Meta announced layoffs, and it’s just radio silence. I don’t want to sound cocky, but with my resume, you at least get a chat with a recruiter, usually right away. But Google and Anthropic just ghosted me.”

4. Higher hiring bar & lower compensation – but not for everyone

In a “normal” market, when the hiring bar goes up, so does compensation. But we heard anecdotes about the hiring bar going up, with the compensation on offer trending down!

Read more

Listen and watch now on YouTube, Spotify, and Apple. See the episode transcript at the top of this page, and timestamps for the episode at the bottom.

Brought to You by

• Antithesis. If you’re using agents to code, the problem isn’t writing the code but making sure that nothing broke. Antithesis runs your whole system in a hostile environment and identifies hard-to-find bugs before users hit them in production. Teams like Jane Street, Fly.io, and the etcd community use agents safely and ship better code, faster, with Antithesis. Learn more.

• WorkOS – make your app and agents Enterprise Ready, with SSO, SCIM, RBAC, and more. Get started.

• turbopuffer. A vector and full-text search engine built on object storage. It’s fast, cheap, and extremely scalable. It’s never been better to try it out: last week, they dropped their base price from $64 per month to $16 per month. Give it a spin now.

In this episode

Few have made as big an impact on software engineering as this week’s guest on the Pragmatic Engineer podcast, Kent Beck. He created Extreme Programming, pioneered test-driven development (TDD), co-created JUnit, and is one of the authors of the famous ‘Agile Manifesto’. But these days, he’s re-examining many ideas for the age of AI, and says we’re failing to accumulate trust during this new era at the same high rate as new code is being accumulated.

Longtime readers will remember how Kent and I bonded over our co-authored article in our response to McKinsey on measuring engineering productivity, and Kent was on the podcast a year ago, talking about TDD and AI.

Today’s episode is the previously untold story of Kent’s remarkable career and how he became the industry legend he is today. We start with Kent’s journey from discovering Smalltalk in the early days of personal computing, to helping define modern software engineering practices. We explore the origins of TDD, design patterns, Extreme Programming, and Agile – along with some lessons learned at Apple and Facebook.

Kent explains why he believes software engineering is about far more than writing code, why no one yet knows exactly how engineers should work alongside AI agents, and how his “explore, expand, extract” framework can help engineers navigate major technology shifts.

Key observations from Kent

Here are 12 rarely-told stories and observations from Kent:

1. Coding is only a small part of software engineering – the rest can’t be automated. Kent rebuts the claim that coding – and eventually the whole software engineering craft – will vanish. He believes coding is only part of what we do, and a small part of it, too. Through your work, you also build confidence, make connections with other people, and develop your personal understanding of the domain. This will remain important, even if we develop our skills in other areas than coding.

2. Lesser-known story: fired by Apple. Kent joined the tech giant in 1987, drawn in by Smalltalk, whose language was taking off at the time. Xerox had handed the language to a few companies to see who could run with it, and Apple was one. But Apple’s customers wanted C and Pascal compilers instead, so the Smalltalk project went nowhere. Kent joined a team building a programming language for kids, but says he was eventually fired because he was in “punk mode” and wanted to do things his own way, instead of being a team player.

3. Kent used to keep a thesaurus close at hand. When working at Tektronix in the late 1980s with Ward Cunningham (the creator of the first wiki, and a pioneer in design patterns), they built HotDraw, a graphical editor with a boxes-and-arrows model. For naming core abstractions, Beck chose “drawing object,” “drawing handle,” etc. But Ward cared intensely about nomenclature and hunted for better vocabulary, so they kept a physical thesaurus which was well thumbed. Beck recalls they became pretty obsessed about the names of things.

4. TDD wasn’t ‘invented’ so much as ‘rediscovered’ by Kent. As a kid, Kent read one of his father’s programming books from the tape-to-tape era, when an input tape like time cards was fed through a payroll program to produce an output tape, such as one with checks. The book’s advice was to take a real input tape and manually type the expected output tape before writing the program. He read this, didn’t understand it, and forgot about it.

Years later, Kent built SUnit, a small testing framework, and randomly remembered the input-tape trick, so mapped it onto SUnit. If he followed the pattern, he’d write the test before the code. He laughed out loud at this because it seemed like such a stupid idea: why write a test that’s guaranteed to fail, when the classes and methods aren’t even defined yet? But when he did, he found his anxiety about programming vanished. This is when he became a TDD convert.

5. Kent invented Extreme Programming (XP) while on the Chrysler payroll project. Kent threw away a codebase that didn’t work and restarted the project with a new methodology. He paired with others, and used his own ideas for testing. Later, he coined the new methodology’s name by deliberately picking one that he knew would be unpopular with the tech establishment of the day: “extreme programming” was born.

6. The Agile Manifesto came together in a messy way. In 2001, a loose group of folks who rejected waterfall development gathered at Snowbird, Utah, to rethink programming. Kent recalls this summit proceeded badly as everyone pushed contradictory ideas. During a break, Martin Fowler and Jim Highsmith stayed behind, and when the others returned, they found the values written on the whiteboard. Kent’s contribution was the word “daily”: “Business people and developers must work together daily throughout the project.”

7. Calling it “agile” was an error. Kent objected to the word “agile” at the time, and still does today, since nobody claims they prefer “rigid” development, and everyone says they’re “agile”, even when they’re not. He would’ve preferred a less spacious term, like with “extreme programming”: after all, it’s hard to call yourself an “extreme programmer” without actually following that methodology.

8. The Dotcom Bust hit very hard. The day before the 9/11 terror attack in 2001, Kent had eight months of consulting work booked at top rates and was finishing work on a house in rural Oregon. The next day, everything was canceled, just as big bills fell due. He burned out into depression and was left unable to program for years, in what was a “lost decade.”

9. At 50, Kent joined Facebook and realized nobody cared about testing or TDD. At Facebook, Kent found a company that barely did any form of unit testing, while running a massive, stable, and fast-growing site. He signed up to teach a TDD class at a hackathon — he wrote the book, after all! The classes either side of his in the schedule both filled up, but the TDD class got zero signups, not even a pity one. He made the decision to forget everything he knew and to relearn software engineering as it was at Facebook. In the end, he stayed seven years.

10. Building software products has three phases: explore, expand, extract. This is Kent’s “3X” model. ‘Explore’ means trying many cheap uncorrelated experiments, ‘expand’ involves focusing on the one thing that’s working and overcoming obstacle after obstacle, while ‘extract’ is a repeatable playbook and economies of scale. How you code, hire, and organize differs across each phase.

11. Kent has always been an anxious programmer. He describes himself as chronically anxious because the more complex the code is, the more he knows it could break. This was the fuel behind testing and TDD, which are approaches designed to soothe an anxious mind.

12. Kent sees himself as a “tree shaker, not a jelly maker.” He starts things like patterns, SUnit, JUnit, TDD, XP, 3X, then pushes them until they take off, before moving on to the next thing. It’s his defining trait, and may explain his enormous output, and also why he abandoned TDD just as it peaked.

+1: The human part is the most important one in software engineering. As Kent explained:

“This is the biggest cosmic, practical joke ever. As young people, we were promised: “Okay, here’s this computer and once you’ve completely understand this computer, you’ll be fine. That’s all you need to do.”

So I set out the first part of my career just to become the best programmer that I could be because that’s what it would take to be successful. And then you realize: sorry, there’s this whole human side. Your ability to affect change in the world is gated by your ability to communicate with, to soothe, to understand other human beings. And those are exactly the skills that I thought I didn’t need to learn!

So I was promised: just understand the computer and you’ll be successful. And then someone went “just kidding, understand people!” And now I was in a position of being ten years behind.”

The Pragmatic Engineer deepdives relevant for this episode

• Measuring developer productivity? A response to McKinsey – co-written with Kent Beck

• TDD, AI agents and coding with Kent Beck

• Paying down tech debt

• The past and future of modern backend practices

Timestamps

00:00 Intro

03:47 Human engineers aren’t going away

08:00 Kent’s path into tech

13:50 Undergraduate and graduate studies

17:21 Kent’s first programming job

18:54 The rise and fall of Smalltalk

27:04 Working with Ward Cunningham

37:36 Design patterns

44:05 Working at Apple

51:08 CRC Cards

59:29 Testing tools in the language

1:04:22 The C3 project with Martin Fowler

1:09:54 Extreme Programming

1:16:25 Developing TDD

1:25:07 Writing the Agile Manifesto

1:30:00 Agile’s impact

1:32:40 Agile’s downside

1:37:32 The Dotcom Bust

1:44:30 Lessons from working at Facebook

1:59:44 Kent’s ‘Good to Great’ program at Facebook

2:06:07 Soft skills engineers need to learn

2:09:30 AI and the challenges of acceleration

2:15:53 Explore, expand, extract

2:22:33 What Kent is excited about

References

Where to find Kent Beck:

• X: https://x.com/kentbeck

• LinkedIn: https://www.linkedin.com/in/kentbeck

• Website: https://kentbeck.com

• GitHub: https://github.com/kentbeck

• Newsletter:

Mentions during the episode:

• TDD, AI agents and coding with Kent Beck: https://newsletter.pragmaticengineer.com/p/tdd-ai-agents-and-coding-with-kent

• Anthropic CEO Predicts AI Will End Coding and Software Engineering:

• Extreme Programming Explained: Embrace Change: https://www.amazon.com/Extreme-Programming-Explained-Embrace-Change/dp/0321278658

• Tidy First?: A Personal Exercise in Empirical Software Design: https://www.amazon.com/Tidy-First-Personal-Exercise-Empirical/dp/1098151240

• Test Driven Development: By Example: https://www.amazon.com/Test-Driven-Development-Kent-Beck/dp/0321146530

• Tektronics: https://www.tek.com/

• Ward Cunningham on LinkedIn: https://www.linkedin.com/in/wardcunningham

• Design Principles Behind Smalltalk: https://www.cs.virginia.edu/~evans/cs655/readings/smalltalk.html

• Kotlin: https://kotlinlang.org

• Swift: https://www.swift.org

• Prolog: https://en.wikipedia.org/wiki/Prolog

• The Timeless Way of Building: https://www.amazon.com/Timeless-Way-Building-Christopher-Alexander/dp/0195024028

• Notes on the Synthesis of Form: https://www.amazon.com/Notes-Synthesis-Form-Harvard-Paperbacks/dp/0674627512

• Larry Tesler: https://en.wikipedia.org/wiki/Larry_Tesler

• PARC: https://en.wikipedia.org/wiki/PARC_(company)

• August 1981 issue of Byte featuring Smalltalk: https://vintageapple.org/byte/pdf/198108_Byte_Magazine_Vol_06-08_Smalltalk.pdf

• Class-responsibility-collaboration (CRC) cards: https://en.wikipedia.org/wiki/Class-responsibility-collaboration_card

• MasPar: https://en.wikipedia.org/wiki/MasPar

• Cray: https://en.wikipedia.org/wiki/Cray

• JUnit: https://en.wikipedia.org/wiki/JUnit

• Erich Gamma: https://en.wikipedia.org/wiki/Erich_Gamma

• OOPSLA: https://en.wikipedia.org/wiki/OOPSLA

• C3: https://www.martinfowler.com/bliki/C3.html

• How AI will change software engineering – with Martin Fowler: https://newsletter.pragmaticengineer.com/p/martin-fowler

• Cycles of disruption in the tech industry: with software pioneers Kent Beck & Martin Fowler: https://newsletter.pragmaticengineer.com/p/cycles-of-disruption-in-the-tech

• The third golden age of software engineering – thanks to AI, with Grady Booch: https://newsletter.pragmaticengineer.com/p/the-third-golden-age-of-software

• Ivar Jacobson’s website: https://www.ivarjacobson.com

• James Rumbaugh: https://en.wikipedia.org/wiki/James_Rumbaugh

• Ron Jeffries’ website: https://ronjeffries.com

• The Agile Manifesto: https://agilealliance.org/agile101/the-agile-manifesto

• Jim Highsmith on LinkedIn: https://www.linkedin.com/in/jhighsmith

• Gusto: https://gusto.com

—

Production and marketing by Pen Name.

In recent days, I’ve visited the offices of OpenAI, Anthropic, and Cursor, in San Francisco. Onsite, I talked with software folks working on the model side to learn more about how their way of building software is changing. This article is based on observations from those visits, including some new developments that I reckon may be adopted industry-wide.

We cover:

1. Next mega-trend? Agents running in the cloud to go mainstream. OpenAI, Anthropic, and Cursor are all-in on cloud agents and expect demand for them to increase massively.

2. Mass adoption of coding harnesses by non-developers. At OpenAI, more than 95% of non-engineers use Codex, not ChatGPT. Is it a sign of things to come across tech?

3. Will the main task of engineers be to make agents more efficient? Ever more engineering work is about building environments for agents to execute more efficiently at Anthropic and Cursor.

4. Next trend? Companies aggressively optimize spend-per-token. AI spending by software engineers is so high that it makes sense for platform teams to slash per-token cost. A case study from Coinbase.

1. Next mega-trend? Agents running in the cloud to go mainstream

Last week, Andrej Karpathy employed the phrase “new paradigm” to describe using Claude Tag – a way to mention Claude in Slack and have it kick off tasks – to work with AI:

There was plenty of pushback against this claim on social media; after all, it’s just a Slack integration with Claude, right? I also thought this until I asked David Hershey at Anthropic’s Applied AI unit about it while visiting the company’s offices. He explained in detail what makes this particular Slack integration different from using something like Claude Code:

• No additional setup. For Claude Code to work well, it should be connected to internal MCP servers, with the right skills on your local machine. Of course, at larger companies this setup is at least partially automated, but devs often need to do tweaking.

• No “tool context-switching.” Just mention it in Slack! Of course, opening Claude Code is not a big effort, but it’s still more work than just typing it out in Slack, and kicking off work.

• Routine work made easier. David has “Claude playing Pokémon” as his side project. Every time a new model comes out, he kicks off a run of his script on it. Previously, this took a few minutes to set up every time, and then it ran on his machine for hours. With this new Slack integration, it’s just one command.

My sense is that the excitement here is less about the Slack integration itself, and more to do with the fact that it’s easy to kick off one or more AIs that no longer run on a local machine. You can skip the setup entirely.

‘Claude Managed Agents’ is a big focus at Anthropic. While there, I met Katelyn Lesse, head of engineering for Claude Platform, who explained that Claude Managed Agents is a large, complex project which her team built over a six-month period. It’s a hosted service to execute long-running agents on various cloud providers.

Cloud agents are the “big deal”, not the Slack integration

Also last week, I had the opportunity to attend a private AI builders event, where Peter Steinberger discussed his workflow.

He talked about how he has gotten really tired of having several OpenClaw agents running on his local machine, which heat up the CPU and slow down his whole system. So, he built Crabbox as a way to run OpenClaw agents in the cloud:

Suddenly, the same solution of cloud agents has emerged in separate places – at Anthropic and with Peter’s OpenClaw – in response to issues caused by locally-running agents. I also learned that cloud agents are becoming a big deal at OpenAI and Cursor, too.

OpenAI bets big on Cloud Agents

OpenAI acquired Ona, (formerly Gitpod), the leader in cloud development environments (CDEs). Back in 2021, CDEs were built for developers to develop software faster, and they also happen to be the perfect primitive for agents to run in a sandboxed cloud environment. From the acquisition announcement by OpenAI (emphasis mine):

“As Codex becomes more capable, its most valuable work is unfolding over hours or days, rather than minutes. We believe people should be able to delegate more ambitious work without remaining tied to the machine where it began. The work should continue beyond the initial session, with Codex making it possible to stay connected and check progress, provide direction, make decisions, and review results from anywhere.

Ona will help us do that. Its technology provides secure, persistent environments where agents can access the tools, systems, and context they need to make progress over time.

By bringing Ona to OpenAI, we will expand Codex beyond work tied to a single device or active session and help more organizations deploy agents securely in production.“

At OpenAI’s offices, I asked engineers there if their focus is shifting to cloud-based agents. Their answer: it very much is. This is a fairly recent development and they’re hiring engineers for the Cloud Agents team. Here’s one job ad that’s currently live:

“We are looking for an experienced software engineer to help build and scale our cloud agent platform. You will design and operate systems for orchestrating agents at scale. You will work closely with product engineers on ChatGPT, API, and Codex to define the right abstractions and enable them to ship products quickly. Strong backend or infrastructure experience is important; experience with Python, Rust, distributed systems, cloud infrastructure, or product platforms is especially helpful.”

Cursor: running agents in the cloud is the future

At Cursor, I spent an hour with cofounder Sualeh Asif (formerly the CTO, now Chief Product Officer). Cursor released Cloud Agents at the end of last year, and is starting to focus a lot more on this area. Sualeh revealed some interesting details about working with cloud agents:

• Agents in the cloud don’t have a way to “complain.” With running an agent locally, when it gets warnings or errors, it surfaces them to a human in its response, who instructs it to do X or Y. However, there’s no such loop for a long-running task on the cloud! Cursor came up with the idea for the model “confess” in regular interviews, and the “confessions” are shared with the infra team to improve the agents’ environment.

• Long-running agents have their own challenges. What happens when a node terminates, midway through; how do you move agent execution from one node to the other? There are new, nontrivial engineering challenges the team needs to solve.

Only yesterday, (Monday, 29 June), Cursor launched its iOS app that enables the building of software from anywhere.

This product is built on top of cloud agents to allow for long-running tasks, the company said:

“Cloud agents run in isolated virtual machines with full development environments to test, verify, and demo work. Since they operate asynchronously with their own tools and resources, cloud agents can run for longer and iterate toward merge-ready PRs without intervention.

To take advantage of these capabilities, send a local plan to a cloud agent or move active agents to the cloud to keep running. You can move the cloud session back to your computer to test changes locally before merging”.

Why are cloud agents suddenly a thing?

It figures that running AI agents in the cloud is practical: there’s less setup involved, several can run in parallel, and the cloud is a better, more convenient place for long-running agents than a personal laptop is; i.e., having to keep the lid open even when walking around the office.

But why is this happening now? My hypothesis is that a mix of factors are at play:

• Coding models got ‘good enough’. Before Opus 4.5 / GPT-5.4, AI models could not really code autonomously, so running them for long tasks was pointless!

• Infra for AI coding agents has matured. Ways of giving more context to agents have improved: things like MCP and skills became mainstream and better understood.

• The context window is bigger. Today’s models have context windows of up to 1 million tokens, meaning that more complex instructions, code, and context can be passed in. It’s hard to have agents run for a longer time without access to a large context window.

• Cloud providers have much more GPU capacity. Every cloud provider has been building GPU clusters in the last few years, and now there’s enough that these AI agents can make use of this infra.

2. Mass adoption of coding harnesses by non-developers?

At OpenAI, I also met Andrew Ambrosino, who was the first engineer on the Codex team. Our time together got off to an ideal start, with Andrew saying he needed to show me something incredible:

Read more

Listen and watch now on YouTube, Spotify, and Apple. See the episode transcript at the top of this page, and timestamps for the episode at the bottom.

Brought to You by

• Antithesis – If you’re using agents to code, the problem isn’t writing the code but making sure it didn’t break anything. Antithesis runs your whole system in a hostile environment and identifies hard-to-find bugs before users hit them in production. With Antithesis, teams like Jane Street, Fly.io, and the etcd community use agents safely and ship better code, faster. Learn more.

• Sentry – application monitoring software built by developers, for developers. Sentry’s Seer AI agent is growing on me, as a way to debug errors faster. Same with Sentry’s MCP server. Check out Sentry.

• Google Cloud Run – a fully managed platform that runs your code directly on top of Google’s scalable infrastructure. Run frontend and backend services, batch jobs, host LLMs, and agents without managing infrastructure. Oh, it has automatic zonal failover out of the box – something that Coinbase could have used a few weeks back! Get started.

In this episode

Navdeep Singh – oftentimes better known as NeetCode – is the creator of NeetCode.io, one of the most popular coding interview preparation platforms and YouTube channels for software engineers. Before building NeetCode full-time, he worked as a software engineer at Amazon and Google.

In this episode of The Pragmatic Engineer, I sit down with Navi to discuss his path from Amazon and Google to building his own startup, why he left Amazon after just two months, what he learned at Google, and the decision to leave a stable engineering career to bet on himself. We also discuss what coding interview preparation teaches beyond passing interviews, the value of going deep on difficult problems, and why systems thinking and domain expertise remain essential engineering skills in the age of AI.

Throughout the conversation, NeetCode makes the case that learning hard things is one of the single best investments an engineer can make, helping build the judgment and expertise that remain valuable no matter how the tools change.

Key observations from Navi

Here are 10 interesting takeaways from our chat:

1. Companies have no real method for evaluating engineers – and likely never did. Navi believes the leetcode-style interview process has persisted because it scales well at large tech companies that need to train hundreds or thousands of interviewers, not because it predicts job performance well.

2. The CAP theorem’s “two-out-of-three” framing is widely taught, but technically shaky. Navi believes this theory of distributed data systems is incomplete, and says he felt validated when researcher and author Martin Kleppmann criticized it. It’s a reminder to think independently and not accept theories without understanding them.

3. Amazon’s intense culture left Navi reluctant to ask questions – which paradoxically, helped at Google. In Navi’s first job, he got used to working alone and not seeking help when needed, and continued this working style at Google. His manager there interpreted that behavior as independence, and as a result, he won rapid promotion from L3 to L4 (mid-level engineering role).

4. The NeetCode YouTube channel took off after he said he’d have to post less. Before viewers knew Navi had got a software engineering job at Google, his audience was small. But it turned out that announcing he’d have to post less for this reason boosted his channel! Suddenly, lots of people wanted to know how he’d landed the role.

5. Cheating tools are helping to resurrect in-person, whiteboard interviews at Google. Navi notes Google has restarted onsite coding interviews because it’s the only way interviewers can be sure that candidates aren’t using AI-powered cheating tools which make data structure and algorithms (DSA) interviews easy to pass.

6. Navi finds AI most valuable as a tech debt and refactoring assistant. He’s using AI to clean up years’ worth of low-quality code on NeetCode’s backend, which also validates the decision to take shortcuts in the knowledge they can be corrected later.

7. ‘Effort’ is becoming the differentiator as AI makes everything else cheap. Navi says how you can prompt almost anything, but the capacity to be engaged with and care about your work, and to defend decisions you make, cannot be prompted by an AI tool. These depend on personal qualities like effort and dedication.

8. Announcements of the death of coding are exaggerated. Despite dramatic improvements in the performance of AI models, Navi does not foresee the majority of engineers being laid off. In fact, he sees the opposite: devs are busier than ever.

9. Humans are likely to remain better at weighing up tradeoffs than LLMs are. It’s a fact that LLMs have become a lot better at coding, but Navi doubts they will be much help in decisions involving judgments about tradeoffs.

10. When hiring for NeetCode, personality traits and motivation matter more than coding skill. Navi’s best recent hire is still an undergrad with little coding experience, but does exceptionally well thanks to possessing high agency. Navi says: “even if they have no idea how to start it, by a week later, they’ll have learned everything about it.”

The Pragmatic Engineer deepdives relevant for this episode

• Learnings from conducting ~1,000 interviews at Amazon

• How experienced engineers get unstuck in coding interviews

• The Reality of Tech Interviews in 2025

• Tech hiring: is this an inflection point?

• AI fakers exposed in tech dev recruitment: postmortem

Timestamps

00:00 Intro

02:57 Navi’s take on coding interviews

06:41 Getting into tech

08:56 Why Navi isn’t a fan of the CAP theorem

13:12 Quitting Amazon after two months

18:22 Google vs Amazon

22:26 The origins of NeetCode

25:27 Leaving Google to go all in on NeetCode

32:02 Why Navi doesn’t fix every bug

39:26 The value of coding interview prep

42:57 Systems thinking and domain expertise

47:28 Hiring at Big Tech

52:15 Tech stack at Neetcode

57:57 The NeetCode redesign contest

1:01:46 The future of software engineers

1:09:04 Hot takes: AGI, AI skill erosion, personality traits

1:22:49 “Maybe some people should just give up”

1:24:39 How to be a standout engineer

1:27:55 Book recommendation

References

Where to find Navdeep Singh (NeetCode):

• X: https://x.com/neetcode1

• LinkedIn: https://www.linkedin.com/in/navdeep-singh-3aaa14161

• YouTube: https://www.youtube.com/c/neetcode

• Website: https://neetcode.io

Mentions during the episode:

• A critique of the CAP theorem: https://martin.kleppmann.com/2015/09/17/critique-of-the-cap-theorem.html

• Designing Data-intensive Applications with Martin Kleppmann: https://newsletter.pragmaticengineer.com/p/designing-data-intensive-applications

• PACELC design principle: https://en.wikipedia.org/wiki/PACELC_design_principle

• Amazon Chime: https://aws.amazon.com/chime/getting-started

• Musk’s 5 Step Design Process: https://modelthinkers.com/mental-model/musks-5-step-design-process

• AI Engineering with Chip Huyen: https://newsletter.pragmaticengineer.com/p/ai-engineering-with-chip-huyen

• Angular: https://angular.dev

• Firebase: https://firebase.google.com

• TypeScript: https://www.typescriptlang.org

• An update on recent Claude Code quality reports: https://www.anthropic.com/engineering/april-23-postmortem

• Building Claude Code with Boris Cherny: https://newsletter.pragmaticengineer.com/p/building-claude-code-with-boris-cherny

• Sora: https://en.wikipedia.org/wiki/Sora_(text-to-video_model)

• Attention is all you need: https://arxiv.org/abs/1706.03762

• The End of Programming as We Know It: https://www.oreilly.com/radar/the-end-of-programming-as-we-know-it

• Satya Nadella on X: https://x.com/satyanadella

• Replit: https://replit.com

• Lovable: https://lovable.dev

• 37signals: https://37signals.com

• DHH’s new way of writing code: https://newsletter.pragmaticengineer.com/p/dhhs-new-way-of-writing-code

• MongoDB: https://www.mongodb.com

• Maybe some people should just give up:

—

Production and marketing by Pen Name.

Three weeks ago, at Craft Conference, in Budapest, Hungary, I opened the event with a keynote titled ‘Slow Down to Speed Up’.

As with most of my talks, it came together in stages, including with some input from full subscribers to the Pragmatic Engineer, with whom I shared my thinking in advance, in ‘Ideas: slow down to speed up when working with AI agents’. Thank you for all the comments!

As fate would have it, just two days beforehand, social media giant Meta appositely provided a real-world case study for my talk, with its most embarrassing outage of all time: users could simply ask the Meta AI to change the email of any account, and the bot happily complied – even if the account belonged to someone else entirely – including a former US president. It was a timely example to kick off the talk with. Check out the full keynote that’s available to view on YouTube:

Watch the keynote video

In this article, I summarize the key parts of my Craft Conference keynote in detail, and some responses received at the event. Full subscribers also have access to the slides, here, and at the foot of this article.

We cover:

1. Meta: “AI psychosis” in effect? Meta has been destroying its engineering org, and an obsessive focus on AI seems to be one reason for it. For more on this question, check out this deep dive.

2. Everything’s changed in six months. From around November last year, things changed with a more capable generation of AI agents like Opus 4.5 and GPT-5.4.

3. How are tech companies changing how they work? Anthropic, OpenAI, Google, Uber, startups, and traditional companies.

4. Trends. Individual productivity is up, but team productivity’s flat, tokenmaxxing and tooling adoption, vanishing middle management, CEOs and CTOs back to coding, and more.

5. Trends across software. Falling software quality, GitHub’s constant reliability woes, AI slop overwhelming devs who care about quality, and more.

6. Advice for software engineers and engineering leaders. Suggestions to help future-proof a career.

7. Feedback. “It’s happening here too!” is a common theme, and relief for some that it’s not unique to their own workplace.

1. Meta: “AI psychosis” in effect?

I thought it was a made-up story when I read that Meta had enabled account takeovers via a “zero auth” policy; i.e., simply asking the Meta AI bot was sufficient to change any account’s email address. After all, shipping such a regression would fly in the face of security measures, code reviews, automated testing, and metrics. Plus, the company has dedicated Integrity teams whose mission statement is to ensure something like this never happens… And yet, this bug shipped.

It went undetected by anyone at Meta, and high-profile accounts like that of former US president, Barack Obama, were taken over as a result. Instagram’s dedicated Integrity team seems to have discovered the embarrassing issue via the news.

As mentioned, it was two days before the Craft keynote, so there was enough time to ask around at Instagram and Meta. Engineers at the company there told me this disaster was caused by AI-generated, AI-reviewed code, along with layoffs, and by forced reassignments from Integrity teams and elsewhere onto AI labeling and related duties.

The problem at Meta seems to be that leadership is aggressively pushing AI, while withdrawing resources and headcount from areas responsible for security, quality, and reliability. Since last week’s deepdive into what’s been happening behind the scenes was published, I’ve learned further details:

• Integrity teams at WhatsApp have been hit hard by layoffs and enforced data-labeling reassignments

• Instagram’s design team suffered a 44% cut in headcount during layoffs

• The Developer Documentation and Support team had a full 95% headcount reduction during layoffs

• Data labeling at the ADO group goes beyond “just” labeling; there are many AI training tasks to do. But these are repetitive, unless you get really creative.

Based on everything I heard from talking with Meta folks, AI-induced behavior was indeed at the heart of this outage. AI-generated, AI-reviewed code, and security teams being gutted, were also factors in the beyond-embarrassing incident. As reported in last week’s deepdive:

• Instagram’s Trust and Safety Team lost around 50% of its staff to data labeling and layoffs. Some of the most senior folks were drafted onto AI training tasks.

• AI-generated changes with zero human input, with just an additional AI code review, have been very common in recent months across the codebase. The change that caused this outage looked like one of these

• Normally, the Trust and Safety team would be on top of monitoring and alerting of security breaches, but it is currently in full disarray due to rapid, internal disorganization”.

If major changes like data labeling assignments and staff tracking are undone, then perhaps things at Meta could return to normal. But so far, the most being done is that leadership has boosted budgets for snacks, travel, and events. Hardly the change needed to restore morale and the former culture!

The comparison to the Lumon corporation in the hit show, Severance, was duly made:

Meta’s worst-ever outage can be interpreted as a warning about what happens when there’s so much focus on AI that the basic health of a company’s main – money-spinning – products is neglected. Instagram, WhatsApp, and Facebook generate the bulk of revenue for Meta, but the company is reallocating more engineers to training the coding model, and aggressively cutting the headcounts of vital orgs to do so – up to the point of not having oncall coverage for key services, and security teams being too stretched to do their jobs.

Am I missing some insight about why it’s more important to build a state-of-the-art, likely-closed AI model that’s good at coding, than it is to keep operating revenue-generating businesses with stable infra?

2. Everything’s changed in six months

Independent, experienced software engineers with zero affiliation to AI labs have been saying for a few months that how we do software engineering has been transformed.

David Heinemeier Hansson (DHH), creator of Ruby on Rails in January:

”Just [in] summer 2025, I spoke with Lex Fridman about not letting AI write any code directly, but it turns out part of this resistance was simply based on the models not being good enough at the time! I spent more time rewriting what it wrote, than if I’d done it from scratch. That has now flipped.”

Simon Willison, creator of Django, in May pinpointed the start of the change to late last year:

“The models released in November 2025 elevated agents to being genuinely useful. We’ve had six months to get used to that idea now; it’s no wonder companies are beginning to spend real money on this technology.”

Teams using agents now ship 5x as many pull requests as two years ago. Here’s data from Linear:

Devs using AI harnesses are producing 2.5x as much code versus 18 months ago. Data from Cursor shows that their users, on average, went from adding 3,500 lines of code in January 2025 to 8,600 today:

The size of pull requests is up 3x versus 18 months ago. Also from Cursor:

More AI changes are accepted without human review. Data from Cursor shows a big jump in changes being accepted without human review from around February this year, when Opus 4.7 and GPT 5.5 launched:

We’re seeing a lot more code generated, and less of it than ever being reviewed by devs. In the relatively short time since AI agents became really good last November, there are more pull requests generated by devs, those pull requests are getting better, and code reviews are harder to keep up with. And so, reviews are less stringent and more changes are shipped to production sans human review! As per my discussions with Meta engineers, these kinds of AI-generated, AI-reviewed pull requests [at Meta, they’re called diffs] are what caused the most recent, embarrassing outage at Instagram.

3. How are tech companies changing how they work?

Details from a few larger tech companies:

Anthropic: all-in on AI agents. In March, Boris Cherny, creator of Claude Code, was on the Pragmatic Engineer podcast and shared some details:

• He personally runs ~5x agents parallel, and ships 20–30 PRs/day

• Product requirement documents (PRDs) are dead & prototypes have replaced them inside Anthropic

• ~100% of Claude Code was generated by Claude in March

• ~70-90% of code inside Anthropic was generated by Claude

• Claude Cowork – another billion-dollar product in terms of revenue potential – was built in just 10 days

Since then, Boris has shared that his workflow has changed to setting up loops to run agents.

OpenAI: moving much faster with AI agents. OpenAI’s Codex team was on the main stage at The Pragmatic Summit in February. Tibo Sottiaux (head of engineering, Codex, OpenAI) shared interesting details on how software development is done in the Codex team:

• There’s a “fix this” button integrated into the internal OpenAI mobile app. It makes one-shot fixes to bug reports, which devs review and can merge

• AI code review for all code changes. With a tiered approach, some changes can be merged with just AI review, and more important ones need an extra human review

• Most devs run several agents in parallel, often walking around with their laptop lids open, so the machine doesn’t enter sleep mode and suspend agents

• Code isn’t really written by hand anymore on the Codex team, and is also less common on other teams too

• “Taste” is becoming a core skill for working at the company

• Codex improves itself: it runs its own test suite, runs improvement tasks overnight, and during team meetings it takes actions on topics discussed

Google: AI widespread. Gemini is not as capable at coding as Claude or Codex, as acknowledged by Google’s CEO, but it’s widely used companywide. The less capable coding model could be hurting AI adoption compared to other companies.

Uber: in-house AI infra. We covered in-depth how Uber uses AI for development, touching on internal systems like:

Uber’s MCP Gateway:

Uber Agent Builder:

The AIFX command line interface:

Minion: background agents

Code Inbox:

Smart Assignments as a neat feature of Code Inbox:

Risk Profiles: another smart feature inside Code Inbox:

uReview, Uber’s AI code review tool:

Autocover and Shepherd for large-scale migrations:

Uber is a good case for learning how much of internal developer infra needs to be rebuilt in order to work well with AI agents. Uber built all the tools above because they needed new, better ways to integrate AI agents into the developer workflow, but couldn’t find anything that worked up to requirements. I’d also point out how much time and effort Uber invested in making code review more efficient. Devs are, indeed, getting overloaded with AI code reviews and Uber’s Code Inbox tries to separate the important pieces of code to review from unimportant ones.

Startups are jumping into using AI agents, although their integrations are more basic. In preparation for the keynote, I talked with several startups about their AI usage. Harnesses like Claude Code, Codex, Cursor, OpenCode and others are popular, and I also noticed most startups are heavily integrating AI agents into Slack, so devs can kick off bugfixes or small feature requests straight from the chat tool.

I observed startups being the most likely to experiment with new AI dev tools; from code review, all the way to AI incident management tools.

“Traditional” companies are also heavily investing in AI dev tools. At the recent Pragmatic Summit in San Francisco, Laura Tacho shared interesting details:

• In February, 18,000 Cisco developers used Codex for complex migrations, code review, and refactoring. This was very early – Codex was just starting to gain industry-wide adoption!

• JP Morgan Chase built a multi-agent framework for annotation, using multiple specialized agents to label customer interaction data, and judge agents to aggregate and rank results. These are pretty advanced use cases!

In general, “traditional” companies do not seem to be lagging behind in using, paying for, and adopting AI agents and AI developer tools.

4. Industry trends

There are trends I’ve observed around the adoption of AI dev tools:

Read more

Today, we cover:

1. Meta follow-up. A follow-up on Tuesday’s deepdive into Meta’s ongoing demolition of its own engineering org. Non-engineering teams lost more than 10% of their staff, while Integrity teams were already stretched before the cuts and reallocation.

2. Big implications of the US banning Anthropic’s new model, Fable. The US government wants only US citizens to be able to use Fable. That could make China more influential as other countries and non-US companies look to capable open models – most of which come from China.

3. SpaceX, Cursor, Continue. Elon Musk became a trillionaire with SpaceX’s IPO, while SpaceX acquired Cursor, and Cursor acquired Continue. SpaceX looks like it wants to go head-on against Anthropic and OpenAI.

4. Industry Pulse. Gemini lead quits for OpenAI, Epic Games’ new open source version control system, Microsoft considers dropping OpenAI for DeepSeek, research confirms LLMs amplify existing expertise, and more.

1. Meta follow-up

Tuesday’s article about Meta destroying its engineering culture made quite the splash in terms of comments and debate online. From inside Meta, there was positive feedback on the accuracy of the deepdive into the situation, and there were also a few helpful clarifications and new information which I’m happy to publish here:

Read more

Listen and watch now on YouTube, Spotify, and Apple. See the episode transcript at the top of this page, and timestamps for the episode at the bottom.

Brought to You by

• Antithesis – if you're using agents to code, the problem isn't writing the code but making sure it didn't break anything. Antithesis goes beyond code review and runs your whole system in faster than real-time and identifies hard to find bugs before your users hit them in production. Antithesis enables teams like Jane Street, Fly.io, and the etcd community to use agents safely and ship better code, faster. Learn more

• WorkOS – make your app and agents Enterprise Ready, with SSO, SCIM, RBAC, and more. Get started.

• turbopuffer – a vector and full-text search engine built on object storage. It’s fast, cheap, and extremely scalable. The teams building the smartest AI products out there — Cursor, Notion, Cognition, Anthropic — they all run on turbopuffer.

In this episode

Robert Erez is a principal engineer at Octopus Deploy, and a longtime expert in CI/CD, deployment systems, and software delivery. Rob and I were also once colleagues on the Skype web team, working on large-scale deployments and release processes.

In this episode of The Pragmatic Engineer, I sit down with Rob to discuss how teams deploy software safely and efficiently at scale. We cover Kubernetes, GitOps, platform engineering, progressive delivery, feature flags, cloud development environments, and the growing role of AI in CI/CD workflows. We also get into the tradeoffs in different deployment approaches, why self-hosted software still matters for some organizations, and the recent evolution of software delivery practices.

Key observations on deployments and CI/CD from the conversation with Rob

Here are 10 interesting takeaways from our chat:

1. Roll forward, never backwards. When a system has state – which typically means it uses databases – then doing a rollback can leave the code talking to a schema that’s no longer in sync. Rob’s advice is to not treat a failure in v2 as a trip back to v1, but rather as a push to v3 with the fix in it.

2. GitOps isn’t actually about Git. None of the four pillars of GitOps – 1) declarative, 2) versioned and immutable, 3) pulled, not pushed, 4) continuously reconciled – require Git, although Git can work under these constraints. Yet, the term ‘GitOps’ has made the industry dogmatic about cramming everything into a repo – even things like secrets that absolutely shouldn’t be there!

3. Continuous deployment can be overkill; continuous delivery is more practical. Shipping every single change to prod (continuous deployment) is not as necessary as many people think, Rob says, and there’s often more value in continuous delivery, where changes flow through testing and the deployment process itself is validated. With continuous delivery, you can decide whether to push to production automatically, or click a button once a week.

4. Feature toggles are a better safety net than rollbacks. When something breaks in production, reaching for a toggle to switch a feature off enables you to “stop the bleeding” and then calmly diagnose an issue. Rolling back a feature flag is less nerve-jangling than scrambling to force a redeployment in the middle of the night!

5. One problem with feature flags is that they’re addictive. On the other hand, the ease with which feature flags are added can create a hygiene crisis if they’re continuously added, but not removed. Treat feature-toggle cleanups like a form of gardening and “weed” rolled-out toggles from the codebase.

6. A Git repo can be a bottleneck at scale. Rob mentions that some companies run thousands of independent Kubernetes clusters that pull state from a Git repository. But such clusters can get throttled by the repo, forcing them into workarounds. Pull-based GitOps doesn’t scale infinitely for free.

7. A sizable number of major institutions remain on-prem – and this won’t change. Banks, other financial bodies, and governments, demand full control over their hardware, upgrades, and downtime. That’s why Rob expects this segment won’t move to cloud-based SaaS.

8. Platform teams work at larger companies. These teams earn their keep in big organizations with multiple teams and projects because they offer ways of bringing sanity and focus.

9. There’s a trend of ephemeral environments replacing test/staging environments. Companies used to have a few testers fighting over a handful of static test environments, but today, it’s trivial to spin up a full environment, per-feature branch, pre-merge. This is an “ephemeral” environment for evaluating that things work, which is then torn down once something is merged. It helps speed up the feedback process.

10. AI shifts the CI/CD calculus from speed to risk. Today, shaving ten minutes off the CI build-time matters because a long-running build blocks human devs. But this time saving will be insignificant when an AI agent writes most of the code and “babysits” a slow pipeline without context switching. Then, the new priority will be to reduce the risk of an AI agent shipping a bug to production, so it will make much more sense to run extra, more thorough, tests – and also even slower ones.

The Pragmatic Engineer deepdives relevant for this episode

• Kubernetes and retiring at the top with Kelsey Hightower

• The past and future of modern backend practices

• Microsoft is dogfooding AI dev tools’ future

• How Kubernetes is built with Kat Cosgrove

• How Linux is built with Greg KH

Timestamps

00:00 Intro

02:09 Canary deployments at Skype

05:01 Joining at Octopus Deploy

06:15 Continuous deployment

10:26 Why Kubernetes won

15:51 Kubernetes on-prem

18:50 How GitOps works

25:00 The uses and limitations of GitOps

31:04 The rise of platform teams

35:51 How AI is changing CI/CD

39:49 Progressive delivery explained

47:31 Rollbacks and roll-forwards

50:14 Feature flags

54:32 How development environments are evolving

57:40 Cloud development environments (CDEs)

1:03:45 Self-hosting CI/CD

1:09:25 Getting started with progressive delivery

1:11:15 Book recommendations

References

Where to find Robert Erez:

• X: https://x.com/no_erez

• LinkedIn: https://www.linkedin.com/in/roberterez

Mentions during the episode:

• Skype: https://en.wikipedia.org/wiki/Skype

• Canary deployments: https://docs.aws.amazon.com/whitepapers/latest/overview-deployment-options/canary-deployments.html

• Octopus Deploy: https://octopus.com

• Paul Stovell on LinkedIn: linkedin.com/in/paulstovell

• Kubernetes: https://kubernetes.io

• How Kubernetes is Built with Kat Cosgrove: https://newsletter.pragmaticengineer.com/p/how-kubernetes-is-built-with-kat

• Kubernetes and retiring at the top with Kelsey Hightower: https://newsletter.pragmaticengineer.com/p/kubernetes-and-retiring-at-the-top

• Docker: https://www.docker.com

• HashiCorp: https://www.hashicorp.com

• Mitchell Hashimoto’s new way of writing code: https://newsletter.pragmaticengineer.com/p/mitchell-hashimoto

• Terraform: https://developer.hashicorp.com/terraform

• GitOps: https://about.gitlab.com/topics/gitops/

• Cursor: https://cursor.com

• The Phoenix Project: A Novel About IT, DevOps, and Helping Your Business Win: https://www.amazon.com/Phoenix-Project-DevOps-Helping-Business/dp/0988262592

• Radical Candor: Be a Kick-Ass Boss Without Losing Your Humanity: https://www.amazon.com/Radical-Candor-Kick-Ass-Without-Humanity/dp/1250103509

• Diaspora: https://www.amazon.com/Diaspora-Novel-Greg-Egan/dp/1597805424

• Schild’s Ladder: https://www.amazon.com/Schilds-Ladder-Novel-Greg-Egan/dp/1597805440

• The Clockwork Rocket: Orthogonal Book One: https://www.amazon.com/Clockwork-Rocket-Orthogonal-Book-One/dp/0575095148

—

Production and marketing by Pen Name.

Subscribe now

Many subscribers expense this newsletter to their learning and development budget. If you have such a budget, here’s an email you could send to your manager.

For two decades, Meta had a unique, high-performance engineering org; right up until around April of this year. For the first 20 years of the company’s existence, it had a “move-fast-and-break-things” culture, and in the early 2020s this shifted to a “move-fast-with-stable-infra” one. Engineers I know at the company were empowered to do good work, focus on impact, and to balance business interests with solid engineering.

But in the past few weeks, all that has changed, as if the leadership has been following detailed blueprints on how to demolish a proven, successful engineering culture in the most ruthlessly efficient way possible.

For the past few weeks, I’ve been sharing how bad things are inside the social media company for engineers in one of Silicon Valley’s most prestigious workplaces. In this article, we walk through what’s happened, and ask what’s going through the minds of leadership who are reducing software engineering there from the profit center that it was between 2004 until very recently, to the disdained cost center that it has become in just a few weeks.

We cover:

1. Meta’s pre-AI engineering culture

2. Investing in AI and pressing engineers to always use it

3. Core engineering folks feel treated like trash

4. Most embarrassing-ever outage

5. Internal mess

6. Self-inflicted wounds

7. Is “AI psychosis” just a Meta issue?

1. Meta’s pre-AI engineering culture

I’d split Meta’s engineering culture into two eras: “move fast and break things”, and then “move fast with stable infra.”

“Move fast and break things”

In the 2010s, Facebook’s unconventional engineering culture had grown somewhat legendary in the tech industry, as the company went against conventional best practices and succeeded massively.

In 2012, when Facebook hit the billion-users landmark, the company produced a small physical book about its culture which was placed on employees’ desks. Presented with retro propaganda design, it was dubbed the “little red book”, co-opting the name of a famous volume of the thoughts of Chairman Mao, (1964).

At around 70 pages long, Facebook’s version codified its engineering culture: speed, fearlessness, taking ownership, and thinking outside of the box.

Back then, mantras in Facebook’s little red book were also in print across campus, and included:

• Move Fast and Break Things

• Done is Better Than Perfect

• Fail Harder

• What Would You Do If You Weren’t Afraid?

• Every Day Feels Like a Week

• The Wright Brothers Did Not Have Pilot Licenses

• The Foolish Wait

• Fortune Favors the Bold

There was genuine focus on building good products. Also from the book:

“Move fast with stable infra” culture

In 2022, I did what is one of the longest deepdives we’ve published on the topic of Meta’s engineering culture. By then, things had evolved, and much of any former recklessness was gone, replaced by the principle of moving fast, but with stable infra. Here’s how I described Meta’s engineering culture then:

“The culture is incredibly engineering-centric: much more than most of Big Tech. This might come from Mark Zuckerberg being an engineer himself, or because much of the innovation in the early days of Facebook came from engineers.

Focus on individual impact. Impact has been the bread and butter of the focus at Facebook. This is very true since the early days, and the focus on generating impact remains.

One detail in common with most Big Tech firms is that both the engineering culture and general culture focus so much on individual impact. This results in some people focusing on short-term, measurable wins and assuming that teamwork and split wins between groups might be less rewarded.

The lack of rigid processes. Facebook seems to have the least amount of processes or standardization across all of Big Tech. Don’t even try to compare it to Amazon’s engineering culture and the countless formal processes there. But even compared to companies like Google, Microsoft or Uber, Facebook’s processes are much looser. Most of this comes from the engineering-centric nature of the company and engineers disliking processes.

Surprisingly little emphasis on testing, documentation or code comments. You’ll find shockingly little automated testing and documentation at Facebook, compared to the rest of Big Tech. Inline code comments are also very rare.

A founder-engineer driven company. Facebook is one of the few Big Tech firms whose founder is an engineer, and still is the CEO. Netflix is the other one where founder and co-CEO Reed Hastings was also a software engineer before starting the company. Amazon was the other example of this until recently, but it’s not the case at Google or Apple. There are good examples of smaller companies like Cloudflare, but they’re all younger than Facebook.

Bootcamp. A unique onboarding process, unlike what any other Big Tech firms offer. We cover this more in the Bootcamp & onboarding section.”

Also, Facebook, as a product, has one of the most sophisticated auto rollout systems in the industry. Instagram has a battle-tested infrastructure where it was almost trivial to launch a new social network (Threads) with 100 million users served in its first week.

Engineers whom I knew inside the company are capable, motivated, and product-minded, and their work was appreciated. CEO, Mark Zuckerberg, was influential: he personally coded the first version of Facebook, had stayed close to engineering, and valued software engineers very much. Engineers there felt they were working inside a profit center.

2. Investing in AI and pressing engineers to always use it

Meta has been the only company among the big five of Apple, Microsoft, Amazon, Google, and itself not to own a hardware platform or operating system. Apple has the iPhone, iPad and Macs, Google has Android, ChromeOS and Pixel phones, Microsoft has Windows, and Amazon has the Kindle.

Stepping back, it looks as though the Mark Zuckerberg of today has resolved not to miss a platform opportunity, after the company failed to build its own mobile OS or mobile phone during the 2010s.

This is one reason for investing so much in virtual reality (VR) with Oculus, and in augmented reality with the Meta Glasses. Facebook changed its name to Meta in 2021, back when it looked like VR – and the metaverse – could be massive. Billions was spent on ensuring Meta would be the market leader in this space. But once again, VR didn’t go mainstream; since the end of the pandemic, popular interest in the segment has died down considerably.

When it became clear that AI would become a mega-trend in 2022, Zuckerberg didn’t miss it: he assembled the internal FAIR group (Fundamental AI Research team) as well as a GenAI product organization and released a series of open-weight AI models:

• Llama 1: released in Feb 2023, three months after ChatGPT, built by FAIR

• Llama 2: in June 2023, built by the GenAI product organization (as well as all subsequent Llama models)

• Llama 3: in April 2024. This model was Meta’s most competitive LLM of all, and gained momentum in adoption across the industry

• Llama 4: in April 2025. This model was deeply disappointing

In June that year, Meta acquired a 49% stake in Scale AI to reboot its AI efforts for a whopping $14.8B, and brought in Scale AI’s CEO, Alexandr Wang to take over Meta’s AI strategy. The acquisition of Chinese startup Manus AI for $2B is currently in question after China blocked the deal from being completed.

Based on the investment made into Scale AI and Wang, it’s pretty clear that Meta – and Zuckerberg – is determined to build a state-of-the-art LLM that can be competitive with the latest versions of Claude and ChatGPT. But Meta has to start pretty much from scratch, and it’s up to Alexandr Wang to deliver.

Scale AI brings in a very specific kind of expertise to Meta, as one of the best in the industry in:

• Training data and labeling: Scale started, and is still best known, as a provider of high-quality labeled datasets for machine learning and AI training, including code, text, image, video, etc.

• RLHF and fine-tuning: A RLHF (reinforcement learning from human feedback) flow which Scale runs, where people give feedback for foundation models, as a “human in the loop” data engine that many leading AI labs use to create better LLMs.

Wang seems to have a very broad reign to do what he has been an expert in: creating training data, doing data labeling and RLHF. This is being pulled off with the labor of Meta’s engineering workforce, and by surveilling it.

Problem #1: Tracking keystrokes and mouse clicks, with no option to opt out. In late April, Meta told engineers they were being enrolled into a system that tracks every keystroke and click, to produce training data for Meta’s new AI. There’s no way to opt out.

Needless to say, this is invasive and raises privacy questions: If you log into your personal bank account, does the tool track you? What about when you’re writing a personal email, or responding to a personal call? Meta held no consultation and there are no workarounds; just a top-down decision being pushed through.

This month, Reuters reported that people’s concerns there are finally being heard:

“Meta is dialing back elements of its plan to collect employee mouse movements, keystrokes and other actions for use as AI ​training data, it said in an internal memo on Tuesday, following weeks ​of angry pushback from staffers.

New controls will allow employees to pause ⁠the data collection for up to 30 minutes at a time and ​request exemptions from the initiative, according to the memo, authored by Stephane Kasriel, ​a vice president in Meta’s AI model-building Superintelligence Labs unit.”

From talking with current Meta engineers, I understand the logging system has not been rolled out in the UK due to data protection regulation.

Problem #2: 30-50% of engineers on core teams have been forcefully reassigned to data labeling and RLHF, upsetting folks even more. Also starting in late April, product engineering teams received a mandate from above, whereby 30-50% of engineers were to leave the team and join the ADO org (Agent Data Optimisation).

“Forceful” reassignment is very relevant here because of Meta’s traditional engineering culture. Between its founding in 2004 and until last year, Meta gave engineers autonomy to choose where they work and what they work on. This was structural to how the company worked:

• Engineers were not hired for a specific team (save for at the Staff+, levels, in some cases). They were hired to the company

• During a 6-week bootcamp, new hires got familiar with Meta’s engineering culture and chose a team

• Team matching meant talking with multiple teams who had headcount, doing small work with them, and finding a match

• Internal transfers were easy, and often initiated by engineers

Team selection via bootcamp started to die down in around 2024, but any Meta engineer with at least two years’ tenure knows that previously they chose what to work on, and of course, could pick the most impactful thing to work on. And then, out of the blue, they’re assigned to a division where the impact is not clear, the work is menial, and doing it too long will surely hurt their career prospects.

“Data labeling” is more involved work, even though a bit repetitive. There are labeling tasks, where you create a website, then look at it and decide if it looks good or not, and give this feedback: this is the “typical” data labeling. But this is the less frequent type of work. But then there are more involved AI training tasks, which looks like this:

• Come up with a task that the AI should do

• Then write the tests that confirm the result

• Package all of this up into a Docker container, using Harbor framework

• Then read the code that the AI writes - often doing this based on feedback from several models, and give it feedback

This work is not easy to do — and you can see why you need good software engineers to do it! — but it gets repetitive really quickly. Most engineers who I talked to said that they found it hard to do this work motivated day-in, day-out. Then again, I did talk with an engineer inside the org who said that by varying the technologies they use, and challenging themselves, they found this work motivating and interesting! This engineer expects there to be more software engineering work coming up, after the phase of training passes. There’s a lesson here: when life gives you lemons, you can either complain about it, or you can make lemonade, like this dev has done so.

This training work is secretive across the AI industry, and vendors are offering $100+ per hour for devs to do this type of work. There are rumors across the industry that OpenAI and Anthropic spends more on these training environments (to make the models great at coding) than the model training run itself!

Infrastructure and security teams were hit especially hard by reassignments. I talked with several engineers in infra orgs, who had 30-50% of their teams drafted into the ADO org. And in some cases, it was the best engineers who left.

One engineer told me that the whole situation feels like the movie, The Hunger Games, when tributes are randomly selected and then removed from their environment, to something completely different. Except, at Meta, many more folks are being affected, with between three and five from a 10-person team going from building products used by hundreds of millions, to giving human feedback on AI-generated GitHub repos, over and over. So, a wider impact than in the Hunger Games, but with less drastic consequences.

Around 6,500 people are in the ADO org, more than at OpenAI and Anthropic. Roughly four to five thousand of these are software engineers. Meta has around 25,000 engineers, meaning that one in every 5-6 software engineers may now find themselves doing data labeling full time.

As you can imagine, people are actively open to new positions, and nobody is updating their job title on LinkedIn and elsewhere to “data labeling at Meta.”

I’ve spoken with people in this role and they don’t like doing it, and feel upset about the top-down decision making. The silver lining is that they still have a job, have retained their salary, and were not part of layoffs. They still have time to leave Meta for something that pays comparably and is not a data labeling job.

3. Core engineering folks feel treated like trash

Problem #3: a month-long waiting game, stoking fear across the company. On 20 April, Reuters reported that Meta planned to lay off 10% of staff in a month’s time, and Meta confirmed the news, meaning there was a period of four weeks when everyone knew that they could be unemployed very soon.

Forced reassignments to data labeling started to happen. As I covered at the time:

“Understandably, there are mixed feelings about this redeployment [to data labeling], with layoffs coming soon. On Wednesday, 20 May, Meta will announce layoffs. Perhaps those moved to do data labeling could actually be “safer” than colleagues on product teams. Of course, this is speculation, but it would be cruel if Meta cut devs reassigned to data labeling.”

Problem #4: Performance review is hyper-aggressive at Meta, so devs optimize all metrics. The internal performance review process, PSC (Performance Summary Cycle), is very stringent, compared to Google and Apple, I’ve learned. Managers inside Meta “fight” over the pay packets of their employees, which involves “knocking down” the packet of engineers on other teams, so their direct reports are ranked higher. It’s common to weaponize metrics in this process – be that business impact, the number of code reviews, number of lines of code written, pre-AI (see our Coding Machine archetype podcast on this.)

Quotas are handed down to managers for the splits of the workforce to be put in each ‘bucket’, and the internal politics gets heated as managers try to get their reports into higher buckets.

After a few years, engineers at Meta learn that the best way to not get a bad PSC rating is to have all metrics – impact, code committed, and other numbers – higher than their peers’ are. Learn more about the internal politics of performance calibrations.

Problem #5: tokens are measured as part of perf, so devs aggressively optimize for it. When layoffs were confirmed, engineers also learned that managers shall inspect token count during perf reviews. This raised worries that those with low token counts might be marked as underperformers and dismissed.

So, what is the natural reaction to this as an engineer at Meta? They started using AI tools for the sake of generating more tokens. This happened while Meta had an internal token leaderboard, encouraging tokenmaxxing. As I wrote on 16 April:

“As per The Information, Meta employees used a total of 60.2 trillion AI tokens (!!) in 30 days. If this was charged at Anthropic’s API prices, it would cost $900M. Of course, Meta is likely purchasing tokens at a discount, but that could still come in at $100M+ – in large part from senseless “tokenmaxxing”.”

The biggest problem: people stop caring about real work and focus on performative work. Let’s check the four ingredients that Meta’s leadership has decided to introduce to their workplace:

1. Tracking the keyboards and mouse clicks of all engineers, where legally possible

2. Reassign a good chunk of engineers to fulltime data labeling

3. Let staff know that 10% of them will be laid off

4. Have a culture where devs optimize for any and all metrics measured during PSC

5. Measure token usage as part of PSC

Shake this mix up well, and what do you get? Two things:

1. Everyone overuses AI to boost their personal stats. An engineering workforce that pretends to work with as much AI, and as little human input, as possible. It’s a strange incentive where an outage caused by a failure to review code properly is not grounds for dismissal, but writing code by hand – instead of having an AI agent write it – could cost you your job

2. Every longer-tenured engineer is seeking a new job, or at least considering it. Those who have been around at Meta longer term have seen enough. Let me describe this visually:

Fresh data seems to confirm that starting in May, a lot more engineers at Meta are looking for an “out.” Here’s how signups to leading Big Tech interview preparation service interviewing.io from Meta have changed over the last year and a half: we can see a massive jump in May vs the year before, as shared by founder and CEO Aliner Lerner with us:

(We covered more data from interviewing.io and other interesting data sources in the deepdive State of the software engineering job market in 2026)

To its credit, Meta has given out generous retention equity packages to several engineers considered key on the remaining teams. These packages make it harder to get matching compensation elsewhere. Still, I talked to one engineer who got an equity top-up, and said that this approach helped him decide to leave as soon as possible because he feels bitter about the lack of autonomy and having no control over things.

4. Most embarrassing-ever outage

Meta’s core infra and security teams have suddenly found themselves severely understaffed. Most folks are pushing AI-generated code merged with AI-only reviews, without paying much attention to quality. After all, they’re dealing with the possibility of unemployment, while firefighting to operate a team without its best engineers whose headcount has been cut in half, all with the knowledge that AI usage could affect their own job security.

Two weeks ago, on 30 May, the most embarrassing outage in Meta’s history happened. Here’s software engineer Siddharth Sundharam’s summary (emphasis mine):

“Yesterday, a slew of Instagram accounts, including some high profile ones like the Obama White House account, seemingly got hacked.

Look, I’m no spring chicken. I’ve spent almost a decade and a half identifying vulnerabilities and exploits at unicorn scale, but this is hands down the most unserious, “almost too stupid to be true” of them all.

The Takeover Flow:

Step 01: Faking the Location & Initiating Support. All the attacker needs to kick this off is your account username. Then, they hop on a VPN or proxy close to your city so Instagram’s security algorithms don’t suspect a thing. (You can quite easily get this from your public profile or “About” section or a hundred other ways.) Once it looks like the request is coming from the correct region, they tell the Meta support AI that the account is hacked and ask it to send the verification codes to an arbitrary email address they control.

Step 02: That’s It. Really, that’s it.

The first proper zero auth password reset I’ve seen in production. There appears to be no additional check as to whether the email being given is actually something the user has used before. Once the AI sends the security code to the attacker’s email, the attacker passes it right back to complete the verification. The platform hands over a fresh password reset link, granting full ownership to the attacker.”

This is a security breach in which Meta left its extra-secure, reinforced front door unlocked, so that anyone could come in, and there was no alarm to notify anybody when it happened! It seemed Meta only noticed when users started reporting it on social media!

From talking with folks inside Meta, I’ve learned that AI was at the heart of this outage. AI-generated, AI-reviewed code, and security teams being gutted were together the cause of this beyond-embarrassing incident. I poked around, and here’s what I gathered:

• Instagram’s Trust and Safety Team lost around 50% of its staff to data labeling and layoffs. Some of the most senior folks were drafted onto AI training tasks.

• AI-generated changes that saw no human input, just another AI code review, were very common during the last two months, across the codebase. The change that caused this outage looked like one of these

• Normally, the Trust and Safety team would be on top of monitoring and alerting of security breaches, but it is currently in full disarray due to rapid, internal disorganization.

Meta’s Chief Security Officer resigned the very next day. The outage was resolved on Monday, 1 June, and an investigation started as part of the SEV process. On Tuesday, Meta’s Chief Information and Security Officer (CISO), Guy Rosen, announced his departure.

Coincidence? I suspect not: the CISO might have stepped down if they warned against the Security org being gutted but were then ignored, and so no longer trusts leadership. I also imagine the CISO didn’t have the idea to move a good half of Instagram’s security team over to data labeling.

As a note, the previously worst outage was all Meta services going down for seven hours in 2021, due to a DNS / BGP configuration issue. It was a bad outage, but Meta handled the follow up well, in my opinion. After that 2021 outage, Meta shared a postmortem and apology. It has not done so for the latest Instagram account takeover outage.

5. Internal mess

Wired shares more details on just how bad the situation is inside Meta, right now:

“Someone interrupted a livestreamed, employee-only presentation at Meta earlier this week with an expletive-filled outburst about “being the company’s bitch,” according to a recording heard by WIRED. The individual then asked the people leading the call to write to a specific Meta AI executive and “tell him that he’s a piece of shit.”

The incident, which took place on a call open to thousands of employees, reflects growing frustration inside the company’s Applied AI team, which was formed in March to support the work of AI researchers at Meta Superintelligence Labs. Three current employees tell WIRED there is widespread dissatisfaction with how Meta assembled the unit of about 6,500 engineers and product managers and the drudgework they allege they have been assigned to improve AI models.

“It’s literally the gulag,” one of the employees claims. “You have zero purpose in life all of a sudden, you barely interact with anyone, you just have these tasks every week.”

There’s more: Meta’s Chief Product Officer, Chris Cox, reportedly admitted to staff that Meta’s upper leadership (the folks above him, meaning the C-level at Meta) created the mess. Also from Wired:

During a meeting this week open to all employees at Instagram, Meta chief product officer Chris Cox addressed the “difficult” and “brutal” environment created by the “insanity of this company” in the past few months, according to a recording heard by WIRED. Cox applauded Instagram employees for launching features and serving around 2 billion users amid what he compared to “running a marathon in the middle of a hailstorm and then, like, your teammate gets replaced and then we’re recording you.”

“It’s like what the fuck,” he said, drawing laughs, before repeating himself. “It is like what the fuck.”

6. Self-inflicted wounds

So, is there an ultimate source of the “insanity of this company”, as CPO Chris Cox put it? Engineers whom I talked to point the finger at two individuals: Mark Zuckerberg and Alexandr Wang. Zuckerberg has full control over the business, and has made the decisions to reallocate a good part of engineering folks to data labeling, to roll out tracking software, and to lay off 10% of staff when Meta achieved record revenue and profits. As the CEO, the buck clearly stops with him.

But it’s hard to unsee that – outside of layoffs – everything that Meta is doing is taken from the Scale AI playbook, and that surely comes from Wang:

• Mandatory keystroke and mouse tracking to generate training data

• Forced data labeling with 4,500+ engineers is to generate high-quality RLHF, surely for Meta’s under-construction coding LLM

• Taking away the best engineers from the heart of the business is surely signed off by Mark Zuckerberg, believing that it is more important for Meta to train a coding AI than it is to operate its core business like Instagram, Facebook or Messenger reliably. Oh, did I mention that on Saturday (12 June) Facebook and Instagram had another SEV0, that is, a full-on outage?

Before all this happened, Meta was on track to overtake Google as the world’s #1 ads business by the end of the year. But for some reason, Mark Zuckerberg decided that building a coding LLM is more important.

Meta’s leadership is now trying to undo all the damage they have done. Wired reports that Meta CTO, Andrew Bosworth, admitted to staff that the AI reorg was atrocious and committed to better communication in the future.

To me, it looks obvious that Zuckerberg doesn’t care how engineers feel about the massive changes, and that Bosworth likely ignored the chaos, all while engineers know for a fact that the next AI model matters more than they do to the business. Bosworth also said that employees will have access to AI coaching tools. Very considerate, given the situation!

Based on all I’ve learned, Meta’s engineering culture is dead because leadership has made it clear that engineering at the company is a cost center.

Needless to say, I hope my assessment is way off, but I’ve seen nothing yet from Mark Zuckerberg and Alexandr Wang – the two executives creating this current mess – to suggest it is. There may be a short time period where, if major changes like data labeling assignments and staff tracking are undone, then things at Meta could return to normal. The longer the current conditions persist, the more tenured engineers will surely leave.

7. Is “AI psychosis” just a Meta issue?

It’s tragic to see a technical founder at Meta so focused on AI that he neglects the engineers who built the heart of his company. But is Meta a one-off exception?

Mitchell Hashimoto (creator of Ghostty, founder of HashiCorp) says he is seeing similar behavior by other founders (emphasis mine:)

“I strongly believe there are entire companies right now under heavy “AI psychosis” and it’s impossible to have rational conversations about it with them. I can’t name any specific people because they include personal friends I deeply respect, but I worry about how this plays out.

I lived through the great MTBF vs MTTR (mean-time-between-failure vs. mean-time-to-recovery) reckoning of infrastructure during the transition to cloud and cloud automation. All those arguments are rearing their ugly heads again but now it’s... the whole software development industry (maybe the whole world, really).

It’s frightening, because ‘psychosis folks’ operate under an almost absolute “MTTR is all you need” mentality: “it’s fine to ship bugs because the agents will fix them so quickly and at a scale humans can’t do!” We learned in infrastructure that MTTR is great but you can’t yeet resilient systems entirely.

The main issue is I don’t even know how to bring this up to people I know personally, because bringing this topic up leads to immediate dismissals like “no no, it has full test coverage”, or “bug reports are going down” or something, which just don’t paint the whole picture.

We already learned this lesson once in infrastructure: you can automate yourself into a very resilient catastrophe machine. Systems can appear healthy by local metrics while globally becoming incomprehensible. Bug reports can go down while latent risk explodes. Test coverage can rise while semantic understanding falls. Changes happen so fast that nobody notices the underlying architecture decaying.

I worry.”

The takeover outage at Instagram was exactly like this: the engineering team dropped the quality bar for AI-generated and AI-reviewed code, probably expecting that they could recover quickly from failures. And they did indeed recover… after the damage was done, high-profile Instagram accounts were hacked, and the system was compromised, all very publicly.

Mitchell highlights the specific concern of founders over-estimating the capabilities of AI, and consequently casting aside sensible safeguards when shipping code to production.

Takeaways

Most of us probably have something to learn from the disastrous events at Meta caused by hyper-focus on AI to the exclusion of people who are the lifeblood of that company. In some good news, I’m hearing that in the UK, some of the 10% layoffs have suddenly been cancelled: at the end of the mandatory consultation period, several infra and security teams are learning that no engineers on their team will be let go, as originally expected.

Meta has a booming business, and is already a beneficiary of AI via increased ads revenue. Meanwhile, my Facebook feed is filled with fake, AI-generated videos, with hundreds of comments from bots and people who seemingly don’t realize it’s AI. It all seems like just more content for Meta to show ads next to.

And yet, despite business booming, Meta’s leadership has gone on a crusade to inflict the most damage possible on its engineering org. Apparently, they’re now learning that most of it was pointless.

If you’re in a leadership position and feeling the temptation to make drastic org changes for AI-related reasons, take a deep breath and see where it left Meta. Meanwhile, If you’re an engineer at a company whose leadership is over-indexing on AI, consider forwarding this article as additional context.

If you’re hiring standout engineers who are extremely hands-on with AI, then it’s never been easier to get talent from Meta, than right now. Every engineer I know at the company is an extremely early adopter of AI, and knows how to build products and AI infra. These folks have soured on the company and its leadership. Meta’s loss of talent will be the gain of other startups and the rest of Big Tech; it’s one benefit of AI that’s probably a bit unexpected – not least of all by Meta!

It seems like the old mantra of “move fast and break things” has now extended to Meta’s engineering org itself, with the company’s rush to over-invest in AI, so it will avoid missing the latest mega-trend in the tech industry.

Today, we cover:

1. Anthropic alienates customers with Fable’s data retention and nerf policies. Anthropic’s latest mode, Fable, stores customer prompts and data for 30+ days and performs worse if Anthropic thinks devs’ usage could potentially pose a commercial threat. The launch is an urgent reminder to have an off-ramp from Claude if you want to be able to vote with your feet.

2. New trend: smart model routing. Are there any ‘intelligent’ router solutions out there which select the right model for the right task? I looked into it, and there are a few options. More suggestions welcome!

3. Reliability fail: No automated zone failover for Coinbase’s global trading service. Back in 2016, Uber had a cross-region failover for its core business. Ten years later, Coinbase does not, so it’s little wonder the platform suffered an embarrassing 10-hour outage. The big mess is a head scratcher.

4. Industry pulse. Anthropic and OpenAI file for IPOs, open source project quits GitHub after maintainer banned without appeal, Opendoor “reshores” jobs from India to the US with AI-native engineers, and more.

5. Are LLMs eroding software engineering skills? A software engineer admits they feel increasingly useless due to how capable LLMs are, in an article that has resonated with lots of folks. My sense is that we give too much credit to LLMs, while underestimating our own capabilities and understanding.

1. Anthropic’s new model release is a reminder to have an off-ramp plan from Claude

Read more

First of all, a big thank you to partner teams for sharing exclusive details for this deepdive:

• Interviewing.io: anonymous mock interviews with engineers from top companies. Thanks to founder and CEO, Aline Lerner.

• Workforce.ai, built by Live Data Technologies, which monitors 1M+ job changes and 300M+ employment validations monthly, across companies, roles, levels, functions, industries, and locations. Special thanks to Alex Hamilton for his input.

• SignalFire: a VC firm with a standout data analysis team. Ordinarily, their data is used to give their portfolio companies a major commercial advantage, but they made an exception to share some for this article. Special thanks to Asher Bantock.

• TrueUp: a platform that scans every open job in Big Tech, top startups, and scaleups, typically paying in the top two tiers of the trimodal software engineering compensation model. Thanks, Amit Taylor.

Today, we cover:

1. Top AI labs are now more attractive than Big Tech. Anthropic is most in demand among job candidates for interview preparation services. Along with OpenAI, it’s almost certainly the place with the most competition for jobs in tech.

2. Harder for new grads & interns to get hired. Data shows that intern intakes have fallen, even as software engineering recruitment recovers. Large tech companies take on half as many interns as before, and candidates’ work and educational backgrounds matter more than ever.

3. Mobile and frontend demand drops, AI & FDE surges. Frontend engineer titles are disappearing fastest across the industry, followed by native iOS and Android ones.

4. AI engineering comp > software engineering comp. AI engineers are more in demand than software engineers, and get higher compensation offers, especially with equity. At the 80th percentile in the US, $300K+ base salaries are the norm now for senior engineers.

5. Management’s “great flattening” continues. There are fewer engineering managers for each engineer across the industry, and fewer VP of engineering and director of engineering posts at Big Tech.

6. Big Tech seniority & tenure keep rising. Since the end of zero interest rates in 2023, it’s as if there’s fewer ways to tempt Big Tech workers to switch jobs, so they remain in situ.

7. Interview preparation signups: what do they indicate? Companies doing mass layoffs tend to see the biggest surge in devs signing up to practice interviews. A list of the top 20 companies from where engineers are preparing to interview elsewhere.

8. Where engineers go after Big Tech. From Amazon, they go pretty much everywhere. From Google, Apple & Meta, it’s mostly to AI labs. Microsoft is where the most ex-workers become their own bosses by working for themselves next.

As a reminder, in Part 1, we covered:

• Software engineering recruitment: trending up, mostly

• Big Tech and publicly-traded companies

• Who’s hiring the most software engineers?

• AI engineering: explosive demand

• Who’s hiring the most AI engineers?

• Is AI engineering replacing software engineering hiring?

See Part 3 for stories from hiring managers and job seekers, covering:

1. “Catch-22:” nobody finds each other

2. No trust. Is AI to blame?

3. Hot market for some, but tough for most

4. Higher hiring bar & lower compensation – but not for everyone

5. Engineering leader recruitment: also weird for senior ICs

6. US market trends

7. Trends in the UK, EU, and rest of the world

Let’s get into the latest data:

1. Top AI labs now more attractive than Big Tech

In Part 1 of this mini-series, we cover the exploding demand for AI engineering:

AI engineering job openings have increased 60% in the past year at top companies, while software engineering openings grew by 7% in the same places. We also found that Big Tech is significantly growing AI engineering headcount:

Anthropic: most in demand

New data suggests that the two biggest AI labs are attracting the most candidates to apply for their AI engineering roles, which is pretty predictable.

Interviewing.io is a job interview preparation service which offers coaching for clients who are getting ready for interviews at specific companies. Based on the number of mentions by clients, Anthropic is the one most candidates are preparing for with paid coaching, and it’s not even close:

It’s also notable that OpenAI (16% of candidates) gets around the same share as Google (17%) and other large tech companies (17%). Combined, Anthropic and OpenAI account for 51% of all interviewing.io coaching requests. For context, interviewing.io only added coaching for frontier labs this year!

There are a few potential causes of the surge of interest in Anthropic:

• OpenAI replaces Anthropic as AI supplier for the US’s novel ‘Department of War’. In early March, the US Government controversially declared Anthropic a “supply chain risk”, and appointed OpenAI as its AI supplier, after Anthropic raised concerns about the future use of AI in mass surveillance and fully autonomous weapons. This raised suspicions that OpenAI agreed to cross ‘red lines’.

• Anthropic’s market dominance continues. Claude Code is the most popular developer tool, as found by our AI tooling survey in February. It seems little has changed.

• Anthropic’s value exceeds OpenAI’s. In March, Anthropic raised a $65B funding round at a $965B valuation, making it more valuable than OpenAI for the first time.

• Anthropic files to go public first. Last week, Anthropic filed to go public, beating OpenAI which has done so a week later.

Anthropic also recruited the most in-demand AI researcher, Andrej Karpathy, in May. My sense is that between the two labs, Anthropic has more momentum for the time being, and has perhaps acquired a ‘halo effect’ with its seemingly principled stance. It’s not surprising that it’s attracting more candidates.

Where are AI labs hiring from?

We looked into the sources of recruits to the three most in-demand AI labs: Anthropic, OpenAI, and Google DeepMind. Here’s what we found:

Where Anthropic hires from, in order of popularity:

1. Google (often Google DeepMind)

2. Meta

3. Stripe

4. Microsoft

5. Amazon (AWS)

6. Databricks

OpenAI:

1. Google

2. Meta

3. Apple

4. Stripe

5. Statsig (after OpenAI acquired Statsig)

6. Microsoft

7. Amazon (mostly AWS)

8. Databricks

9. Airbnb

10. NVIDIA

Google DeepMind:

1. Internal transfer

2. Meta

3. Microsoft

4. Amazon

5. Windsurf

Anthropic has the highest retention rate of all AI labs. Data from SignalFire found the 2-year retention rate (percentage of employees who stay 2 years) is:

• OpenAI: 67%. This is consistent with the rest of Big Tech

• Google DeepMind: 78%. Well above the rest of Big Tech

• Anthropic: 80%. Standout, industry-wide!

Consistent with SignalFire’s 2025 finding, OpenAI 2-year retention was 67% (FAANG-level) versus Anthropic (80%), and DeepMind (78%).

2. Harder for graduates & interns to get hired

It’s well known that it’s getting harder to be hired as an early-career engineer, and new data underlines this.

Intern intakes down since 2022

Live Data Technologies looked at software engineer vs engineering intern hiring trends at 30-80 US-based tech companies, pinned to 2019 hiring numbers (100% being that year’s total number of hires). The spread is wide because Live Data Technologies selects the top few dozen companies that meet their criteria for a “large public tech company” in their database.

The findings:

Zooming into intern hiring, here’s a visualization of it as a percentage of all appointments:

Alex Hamilton, analyst at Live Data Technologies, says:

“We’ve seen overall software engineering hiring start to come back since the 2023 tough market. However, intern intake just kept falling alongside it, which isn’t what you’d normally expect.

Historically, intern programmes have tended to bounce back pretty quickly once companies start hiring again. That hasn’t happened this time, and 2024 and 2025 are the first years in the series where the two lines are moving in opposite directions.

Where you do see companies holding intern intake steady or growing it, it’s almost always a reflection of where they are as a business, be that earlier-stage or faster-growing companies, rather than any kind of broader market recovery”.

Graduate jobs trending down

Anecdotally, we hear new grads continue to have a hard time finding a position. Our new recruitment data on major US tech companies confirms it:

“New grads” in this data are software engineers who graduated less than a year before getting a job as a software engineer. In 2025, just one in 10 engineering hires at larger companies were recent grads, down from nearly three in 10 in 2023.

Pedigree matters more for new grads

We looked closely at the places from where new graduate software engineers are joining US-based tech companies, and found the share of successful candidates from “elite” universities is growing:

By “elite” universities, we mean one of the top 20 US colleges for computer science, such as MIT, Stanford, Carnegie Mellon, UC Berkeley, Harvard, Caltech, Georgia Tech, and Cornell.

Obviously, the influence of these places’ reputations is not a new thing, it’s what makes them “elite” universities, after all. But with new grad hiring down across tech, even graduates from these universities can expect fewer opportunities than before.

Even so, the pedigree that comes from graduating from a well-known university, or doing an internship at a well-known company, becomes ever more significant as the job market tightens.

3. Mobile and frontend demand drops, AI & FDE surges

Here’s interesting data showing the shifting prevalence of job titles on sites like LinkedIn over time:

Some takeaways:

• AI engineering’s on fire. This is not surprising and is evident throughout our study.

• Forward Deployed Engineers (FDE) are growing rapidly. We covered the sudden demand for FDEs in 2025, and this year we’re seeing the FDE role heat up again.

• Modest increase in sales engineers: Sales engineers help close large, B2B-type, deals, and are typical at companies selling to enterprises. The rise in prevalence of this position suggests more companies are targeting enterprise-scale clients. Also, my sense is that FDEs can operate like sales engineers.

• There are fewer native mobile engineers. In 2022, I observed a drop in demand for native iOS and Android engineers. Cross-platform frameworks being more capable today may contribute to fewer places investing in native applications, and a fall in demand for this discipline overall. Is the “golden age” of native mobile development over, with its standalone native iOS, native Android, and web teams for a single product?

• Frontend-only engineers are disappearing. This is one of the most interesting trends in the data. I’ve observed full-stack engineers become the norm at many places, who can do both frontend and backend development. Especially with AI, there is no reason a proficient frontend engineer should not work on backend as well, so, I expect “pure” frontend engineers will be employed only in larger companies, where demand exists for things like building a design system. We cover more on this topic in the deepdive, Design systems for software engineers.

4. AI engineering comp > software engineering comp

One poorly-kept secret in tech is that although software engineering compensation is very good at Big Tech and top startups, it’s superior for AI engineering jobs at the same places – and even better at leading AI labs:

Read more

Listen and watch now on YouTube, Spotify, and Apple. See the episode transcript at the top of this page, and timestamps for the episode at the bottom.

Brought to You by

• Antithesis – managing infra has gone through a mindset shift: from an imperative approach (with the likes of Puppet and Ansible) to a declarative one (with the likes of Terraform and Kubernetes). Software development is going through a similar shift with AI agents – and Antithesis is the declarative testing tool that can keep up with these AI agents. Learn more

• Buildkite – the CI used by companies like OpenAI, Anthropic, Uber, Shopify, Airbnb, Ramp and many more. Buildkite was stress-tested at the largest scale inside companies solving some of the hardest engineering problems. It’s built to absorb whatever your coding agents throw at the build queue, today. Learn more.

• Sentry – application monitoring software built by developers, for developers. With Sentry MCP and CLI, set up a helpful flow like “when a production error fires, have an agent investigate it, pulling all the error context that Sentry already has. Check out Sentry

In this episode

Kelsey Hightower went from a self-taught technician installing DSL modems to becoming one of Google’s elite Distinguished Engineers, whom the CEO of Microsoft personally tried to recruit. Hightower’s career achievements are rooted in hard work and self-directed learning, and today he’s one of the most influential voices in modern infrastructure, through his talks, open source work, and writing.

In this episode of The Pragmatic Engineer podcast, Kelsey and I cover his unconventional path into tech and the lessons he’s learned during three decades in the industry. We discuss his entrepreneurial years, building a reputation through open source, the rise of containers and Kubernetes, and his time at Google during one of the most consequential periods in cloud computing.

He recounts how a job offer from a big tech giant led to the biggest raise of his career, what prompted him to slow down after years of career acceleration, and we also discuss his perspective on AI. Throughout, Kelsey keeps a simple idea front of mind: that technology is ultimately about people. Whether it’s infrastructure, leadership, careers, or AI, he argues that the goal is not to build technology for its own sake; it’s to solve meaningful human problems.

My observations from the conversation with Kelsey

This is a long episode with many compelling, previously unshared stories. As such, there’s a lengthier-than-usual list of 15 interesting takeaways.

1. Kelsey’s career path is incredibly inspiring. From modest beginnings with no role models in technology, Kelsey worked his way up from technician, to software engineer, and grew into one of the most respected Distinguished Engineers at Google. His drive to improve and to always do his very best work is infectious. My sense is Kelsey would never be satisfied with “good enough” and has always aimed for standout work. This approach is rare and has opened doors that stay closed to average work.

2. Treat every public talk like a job interview. Kelsey’s career inflection points often came from people in audiences offering him jobs. He joined CoreOS because the team watched him PXE-boot CoreOS live on stage; afterwards, they wanted to recruit him.

3. “Some people have 20 years’ tenure – but only one year of experience.” Doing the exact same work for years does not advance your skillset. For example, Kelsey observed people in a call center doing identical manual, ticket-closing work for two decades, who never thought of automating themselves out of even some of it. Kelsey started doing this almost immediately and gained valuable new experience.

4. Side hustles and doing your own thing teach you business like no IC job can. Before becoming a software engineer at Google, Kelsey was a manager for his comedian friend, operated a computer store, and did IT contracting. These gigs taught him logistics, planning, and about money. All this helped him be far more effective at talking with executives and acting as an executive sponsor inside Google.

5. Business owners get paid last, but not employees. Kelsey ran a computer store in Atlanta that did well, but he closed it down to become an employee. As he puts it: “Employees get paid first, the owner gets paid last, and there are months where you get paid last, or you don’t get paid at all.” Kelsey hit a point in his life where he valued the predictability of a salary over volatile income.

6. Leading without influence: don’t tell people the answers, let them be discovered. Kelsey knew that Kubernetes was difficult to onboard and needed the Kubernetes team’s help to fix it. To get this done, he got the Kubernetes team to install K8s with no scripts, watched them struggle, and then guided them to what was missing. After people discovered the problems themselves, they set about fixing them. Kelsey noted that people uncovering problems on their own works better than an issue being handed down to them.

7. Can you explain what your startup does without mentioning AI? When Kelsey researches startups seeking his advice, he challenges founders to not say “AI” once. This means that they must explain the actual value their company creates. One unexpected benefit of this is that it often reveals there are easier, cheaper ways to achieve a goal than with AI.

8. “Look in the mirror”: AI’s impact on the software engineering profession. Kelsey says that when complaining about AI, engineers should bear in mind how their industry has disrupted and displaced jobs in other parts of the economy. It might be one reason why affected software engineers get seemingly little public sympathy.

9. Don’t let agents run loose on raw infra; provide guardrails and context. As Kelsey puts it, “I’ve seen what humans do when you just give them the AWS console. Watch what Claude’s going to do!”

10. It’s okay to interview when you’re happy in a job. Just put your ego aside and check out the market. Kelsey wasn’t looking and didn’t want to bother, but his wife pushed him, saying Kelsey should see what’s out there. She was right!

11. It’s very rare to get an extra zero put on your compensation figure – but it happened. Kelsey was a successful, well-paid Google engineer when Microsoft made him an offer that 10x’d his compensation. When Kelsey told Google he was planning to take the offer, it matched the offer, proving that his market value had massively increased. It shows that being well paid doesn’t necessarily mean you’re being paid at the correct market rate.

12. Satya Nadella: “We gave you an offer as if you were running away from something. We should have given you something to run towards.” Microsoft’s CEO himself acknowledged that when trying to recruit Kelsey, the Windows maker should’ve focused on the mission and growth opportunities, instead of just throwing money at him. It’s a lesson that goes well beyond this specific case.

13. Reframe money as “freedom tokens” instead of status. Once Kelsey stopped caring about impressing others with displays of wealth, money became a means to exit the game, not an end in itself. This reframing changed what he optimized for: to have enough money to not have to work for someone else.

14. Kelsey’s advisory setup: 1-year, no cliff, 10-year exercise window, plus a retainer. Kelsey advises select startups and has learned a few important things:

• Advisory shares alone are usually worth nothing, mostly due to dilution and the tax traps of exercising them.

• Cash retainers ensure he is not working for free. It also means the companies paying him expect impact, and real impact is worth paying for – by serious companies, that is.

Referrals are the fuel for winning more advisory work, and these are based on outcomes. Kelsey advised Pixie Labs, which was later acquired by New Relic, partly thanks to Kelsey’s involvement. Word got around that Kelsey’s advisory could make an impact, so more VCs and founders started reaching out to him.

15. Apply “intentional living” everywhere, not just where it’s comfortable. As a long-time minimalist, Kelsey is intentional about possessions, but realized he was being unintentional in other areas. For example, he now reads lyrics while listening to music to actually understand a song. Intentionality is a habit you extend, not a one-time setting.

The Pragmatic Engineer deepdives relevant for this episode

• Career paths for software engineers at large tech companies

• The past and future of modern backend practices

• How Kubernetes is built

• How Linux is built

• The Staff Engineer’s Path: You’re a role model now (sorry!)

Timestamps

00:00 Intro

03:34 Kelsey’s first job at McDonald’s

05:04 His non-traditional path into tech

11:45 Landing his first tech job with an A+ certification

15:33 His entrepreneurial years

19:45 Joining Google as a data center technician

27:48 Learning automation at a Rackspace spinoff

33:26 Moving into financial services

50:00 Building a reputation through open source

53:55 From configuration management to containers

1:08:20 The rise of Kubernetes

1:25:05 Why he almost joined NASA instead of Google

1:29:20 Defining DevRel at Google

1:38:20 Demonstrating impact at Google

1:41:20 Microsoft’s offer

1:55:20 Learning how to slow down

2:06:39 Advising and investing

2:15:03 A people-first view of GenAI

2:24:27 Using AI with guardrails

2:28:26 Matching AI to the task

2:36:06 Staying relevant in the AI era

References

Where to find Kelsey Hightower:

• X: https://x.com/kelseyhightower

• LinkedIn: https://www.linkedin.com/in/kelsey-hightower-849b342b1

Mentions during the episode:

• TI-BASIC: https://en.wikipedia.org/wiki/TI-BASIC

• Georgia HOPE scholarships: https://www.gafutures.org/hope-state-aid-programs/hope-zell-miller-scholarships/hope-scholarship

• BellSouth: https://en.wikipedia.org/wiki/BellSouth

• FreeBSD: https://en.wikipedia.org/wiki/FreeBSD

• Puppet: https://www.puppet.com

• Rackspace: https://www.rackspace.com

• TSYS: https://www.tsys.com

• James Turnbull’s website: https://jamesturnbull.net

• Kubernetes: https://kubernetes.io

• Red Hat: https://www.redhat.com

• Terraform: https://developer.hashicorp.com/terraform

• Docker: https://www.docker.com

• Mitchell Hashimoto’s new way of writing code: https://newsletter.pragmaticengineer.com/p/mitchell-hashimoto

• CoreOS: https://fedoraproject.org/coreos

• Mesos: https://en.wikipedia.org/wiki/Apache_Mesos

• Go: https://go.dev

• GopherCon: https://www.gophercon.com

• Rob Pike’s blog: https://commandcenter.blogspot.com

• Russ Cox’s website: https://swtch.com/~rsc/

• Brad Fitzpatrick’s website: https://bradfitz.com

• Erik St. Martin on X: https://x.com/erikstmartin

• Brian Ketelsen’s website: https://brian.dev

• Billie Cleek on LinkedIn: https://www.linkedin.com/in/billie-cleek-677b0830

• KubeCon: https://events.linuxfoundation.org/kubecon-cloudnativecon-north-america/

• How Kubernetes is Built with Kat Cosgrove: https://newsletter.pragmaticengineer.com/p/how-kubernetes-is-built-with-kat

• Kubernetes: Up and Running: Dive into the Future of Infrastructure: https://www.amazon.com/Kubernetes-Running-Dive-Future-Infrastructure/dp/1491935677

• Brian Grant on LinkedIn: https://www.linkedin.com/in/bgrant0607

• Eric Tune on LinkedIn: https://www.linkedin.com/in/eric-tune-3033693

• Dawn Chen on LinkedIn: https://www.linkedin.com/in/chendawnhomepage

• Satya Nadella on X: https://x.com/satyanadella

• Hit Refresh: The Quest to Rediscover Microsoft’s Soul and Imagine a Better Future for Everyone: https://www.amazon.com/Hit-Refresh-Rediscover-Microsofts-Everyone-ebook/dp/B01HOT5SQA

• Thomas Kurian on LinkedIn: https://www.linkedin.com/in/thomas-kurian-469b6219

• Liz Rice on LinkedIn: https://www.linkedin.com/in/lizrice

• Pixilabs: https://www.pixilabs.com

• Datadog: https://www.datadoghq.com

• Guillermo Rauch on X: https://x.com/rauchg

• Massdriver: https://www.massdriver.cloud

• Here’s everything the iPhone has replaced in the last 10 years: https://www.cnbc.com/2017/06/29/everything-the-iphone-has-destroyed-in-the-last-10-years.html

• Wix: https://www.wix.com

• Lambda: https://aws.amazon.com/lambda

—

Production and marketing by Pen Name.

I’m in Budapest, Hungary, this week, for Craft Conference, where I’ll be giving a keynote presentation alongside other speakers, including software engineering legend Kent Beck, who’s been on the podcast, Hillel Wayne, a formal methods expert and the author of ‘Logic for Programmers’, and Titus Winters, lead author of Software Engineering at Google.

The title of my keynote is “Slow down to speed up”, and I’ve been thinking about this topic a lot recently. Here are some things I’ve been seeing that I feel are relevant…

AI coding tools now used by pretty much all software engineers – that’s fast!

Read more

Today, we cover:

1. New trend? ROI questions for AI investments. I talked with engineering leaders at mid-sized and large companies, where spending on AI agents is being dampened via per-engineer mont…

Read more