Today, we cover:

1. Anthropic alienates customers with Fable’s data retention and nerf policies. Anthropic’s latest mode, Fable, stores customer prompts and data for 30+ days and performs worse if Anthropic thinks devs’ usage could potentially pose a commercial threat. The launch is an urgent reminder to have an off-ramp from Claude if you want to be able to vote with your feet.

2. New trend: smart model routing. Are there any ‘intelligent’ router solutions out there which select the right model for the right task? I looked into it, and there are a few options. More suggestions welcome!

3. Reliability fail: No automated zone failover for Coinbase’s global trading service. Back in 2016, Uber had a cross-region failover for its core business. Ten years later, Coinbase does not, so it’s little wonder the platform suffered an embarrassing 10-hour outage. The big mess is a head scratcher.

4. Industry pulse. Anthropic and OpenAI file for IPOs, open source project quits GitHub after maintainer banned without appeal, Opendoor “reshores” jobs from India to the US with AI-native engineers, and more.

5. Are LLMs eroding software engineering skills? A software engineer admits they feel increasingly useless due to how capable LLMs are, in an article that has resonated with lots of folks. My sense is that we give too much credit to LLMs, while underestimating our own capabilities and understanding.

1. Anthropic’s new model release is a reminder to have an off-ramp plan from Claude

Read more

First of all, a big thank you to partner teams for sharing exclusive details for this deepdive:

• Interviewing.io: anonymous mock interviews with engineers from top companies. Thanks to founder and CEO, Aline Lerner.

• Workforce.ai, built by Live Data Technologies, which monitors 1M+ job changes and 300M+ employment validations monthly, across companies, roles, levels, functions, industries, and locations. Special thanks to Alex Hamilton for his input.

• SignalFire: a VC firm with a standout data analysis team. Ordinarily, their data is used to give their portfolio companies a major commercial advantage, but they made an exception to share some for this article. Special thanks to Asher Bantock.

• TrueUp: a platform that scans every open job in Big Tech, top startups, and scaleups, typically paying in the top two tiers of the trimodal software engineering compensation model. Thanks, Amit Taylor.

Today, we cover:

1. Top AI labs are now more attractive than Big Tech. Anthropic is most in demand among job candidates for interview preparation services. Along with OpenAI, it’s almost certainly the place with the most competition for jobs in tech.

2. Harder for new grads & interns to get hired. Data shows that intern intakes have fallen, even as software engineering recruitment recovers. Large tech companies take on half as many interns as before, and candidates’ work and educational backgrounds matter more than ever.

3. Mobile and frontend demand drops, AI & FDE surges. Frontend engineer titles are disappearing fastest across the industry, followed by native iOS and Android ones.

4. AI engineering comp > software engineering comp. AI engineers are more in demand than software engineers, and get higher compensation offers, especially with equity. At the 80th percentile in the US, $300K+ base salaries are the norm now for senior engineers.

5. Management’s “great flattening” continues. There are fewer engineering managers for each engineer across the industry, and fewer VP of engineering and director of engineering posts at Big Tech.

6. Big Tech seniority & tenure keep rising. Since the end of zero interest rates in 2023, it’s as if there’s fewer ways to tempt Big Tech workers to switch jobs, so they remain in situ.

7. Interview preparation signups: what do they indicate? Companies doing mass layoffs tend to see the biggest surge in devs signing up to practice interviews. A list of the top 20 companies from where engineers are preparing to interview elsewhere.

8. Where engineers go after Big Tech. From Amazon, they go pretty much everywhere. From Google, Apple & Meta, it’s mostly to AI labs. Microsoft is where the most ex-workers become their own bosses by working for themselves next.

As a reminder, in Part 1, we covered:

• Software engineering recruitment: trending up, mostly

• Big Tech and publicly-traded companies

• Who’s hiring the most software engineers?

• AI engineering: explosive demand

• Who’s hiring the most AI engineers?

• Is AI engineering replacing software engineering hiring?

Let’s get into the latest data:

1. Top AI labs now more attractive than Big Tech

In Part 1 of this mini-series, we cover the exploding demand for AI engineering:

AI engineering job openings have increased 60% in the past year at top companies, while software engineering openings grew by 7% in the same places. We also found that Big Tech is significantly growing AI engineering headcount:

Anthropic: most in demand

New data suggests that the two biggest AI labs are attracting the most candidates to apply for their AI engineering roles, which is pretty predictable.

Interviewing.io is a job interview preparation service which offers coaching for clients who are getting ready for interviews at specific companies. Based on the number of mentions by clients, Anthropic is the one most candidates are preparing for with paid coaching, and it’s not even close:

It’s also notable that OpenAI (16% of candidates) gets around the same share as Google (17%) and other large tech companies (17%). Combined, Anthropic and OpenAI account for 51% of all interviewing.io coaching requests. For context, interviewing.io only added coaching for frontier labs this year!

There are a few potential causes of the surge of interest in Anthropic:

• OpenAI replaces Anthropic as AI supplier for the US’s novel ‘Department of War’. In early March, the US Government controversially declared Anthropic a “supply chain risk”, and appointed OpenAI as its AI supplier, after Anthropic raised concerns about the future use of AI in mass surveillance and fully autonomous weapons. This raised suspicions that OpenAI agreed to cross ‘red lines’.

• Anthropic’s market dominance continues. Claude Code is the most popular developer tool, as found by our AI tooling survey in February. It seems little has changed.

• Anthropic’s value exceeds OpenAI’s. In March, Anthropic raised a $65B funding round at a $965B valuation, making it more valuable than OpenAI for the first time.

• Anthropic files to go public first. Last week, Anthropic filed to go public, beating OpenAI which has done so a week later.

Anthropic also recruited the most in-demand AI researcher, Andrej Karpathy, in May. My sense is that between the two labs, Anthropic has more momentum for the time being, and has perhaps acquired a ‘halo effect’ with its seemingly principled stance. It’s not surprising that it’s attracting more candidates.

Where are AI labs hiring from?

We looked into the sources of recruits to the three most in-demand AI labs: Anthropic, OpenAI, and Google DeepMind. Here’s what we found:

Where Anthropic hires from, in order of popularity:

1. Google (often Google DeepMind)

2. Meta

3. Stripe

4. Microsoft

5. Amazon (AWS)

6. Databricks

OpenAI:

1. Google

2. Meta

3. Apple

4. Stripe

5. Statsig (after OpenAI acquired Statsig)

6. Microsoft

7. Amazon (mostly AWS)

8. Databricks

9. Airbnb

10. NVIDIA

Google DeepMind:

1. Internal transfer

2. Meta

3. Microsoft

4. Amazon

5. Windsurf

Anthropic has the highest retention rate of all AI labs. Data from SignalFire found the 2-year retention rate (percentage of employees who stay 2 years) is:

• OpenAI: 67%. This is consistent with the rest of Big Tech

• Google DeepMind: 78%. Well above the rest of Big Tech

• Anthropic: 80%. Standout, industry-wide!

Consistent with SignalFire’s 2025 finding, OpenAI 2-year retention was 67% (FAANG-level) versus Anthropic (80%), and DeepMind (78%).

2. Harder for graduates & interns to get hired

It’s well known that it’s getting harder to be hired as an early-career engineer, and new data underlines this.

Intern intakes down since 2022

Live Data Technologies looked at software engineer vs engineering intern hiring trends at 30-80 US-based tech companies, pinned to 2019 hiring numbers (100% being that year’s total number of hires). The spread is wide because Live Data Technologies selects the top few dozen companies that meet their criteria for a “large public tech company” in their database.

The findings:

Zooming into intern hiring, here’s a visualization of it as a percentage of all appointments:

Alex Hamilton, analyst at Live Data Technologies, says:

“We’ve seen overall software engineering hiring start to come back since the 2023 tough market. However, intern intake just kept falling alongside it, which isn’t what you’d normally expect.

Historically, intern programmes have tended to bounce back pretty quickly once companies start hiring again. That hasn’t happened this time, and 2024 and 2025 are the first years in the series where the two lines are moving in opposite directions.

Where you do see companies holding intern intake steady or growing it, it’s almost always a reflection of where they are as a business, be that earlier-stage or faster-growing companies, rather than any kind of broader market recovery”.

Graduate jobs trending down

Anecdotally, we hear new grads continue to have a hard time finding a position. Our new recruitment data on major US tech companies confirms it:

“New grads” in this data are software engineers who graduated less than a year before getting a job as a software engineer. In 2025, just one in 10 engineering hires at larger companies were recent grads, down from nearly three in 10 in 2023.

Pedigree matters more for new grads

We looked closely at the places from where new graduate software engineers are joining US-based tech companies, and found the share of successful candidates from “elite” universities is growing:

By “elite” universities, we mean one of the top 20 US colleges for computer science, such as MIT, Stanford, Carnegie Mellon, UC Berkeley, Harvard, Caltech, Georgia Tech, and Cornell.

Obviously, the influence of these places’ reputations is not a new thing, it’s what makes them “elite” universities, after all. But with new grad hiring down across tech, even graduates from these universities can expect fewer opportunities than before.

Even so, the pedigree that comes from graduating from a well-known university, or doing an internship at a well-known company, becomes ever more significant as the job market tightens.

3. Mobile and frontend demand drops, AI & FDE surges

Here’s interesting data showing the shifting prevalence of job titles on sites like LinkedIn over time:

Some takeaways:

• AI engineering’s on fire. This is not surprising and is evident throughout our study.

• Forward Deployed Engineers (FDE) are growing rapidly. We covered the sudden demand for FDEs in 2025, and this year we’re seeing the FDE role heat up again.

• Modest increase in sales engineers: Sales engineers help close large, B2B-type, deals, and are typical at companies selling to enterprises. The rise in prevalence of this position suggests more companies are targeting enterprise-scale clients. Also, my sense is that FDEs can operate like sales engineers.

• There are fewer native mobile engineers. In 2022, I observed a drop in demand for native iOS and Android engineers. Cross-platform frameworks being more capable today may contribute to fewer places investing in native applications, and a fall in demand for this discipline overall. Is the “golden age” of native mobile development over, with its standalone native iOS, native Android, and web teams for a single product?

• Frontend-only engineers are disappearing. This is one of the most interesting trends in the data. I’ve observed full-stack engineers become the norm at many places, who can do both frontend and backend development. Especially with AI, there is no reason a proficient frontend engineer should not work on backend as well, so, I expect “pure” frontend engineers will be employed only in larger companies, where demand exists for things like building a design system. We cover more on this topic in the deepdive, Design systems for software engineers.

4. AI engineering comp > software engineering comp

One poorly-kept secret in tech is that although software engineering compensation is very good at Big Tech and top startups, it’s superior for AI engineering jobs at the same places – and even better at leading AI labs:

Read more

Listen and watch now on YouTube, Spotify, and Apple. See the episode transcript at the top of this page, and timestamps for the episode at the bottom.

Brought to You by

• Antithesis – managing infra has gone through a mindset shift: from an imperative approach (with the likes of Puppet and Ansible) to a declarative one (with the likes of Terraform and Kubernetes). Software development is going through a similar shift with AI agents – and Antithesis is the declarative testing tool that can keep up with these AI agents. Learn more

• Buildkite – the CI used by companies like OpenAI, Anthropic, Uber, Shopify, Airbnb, Ramp and many more. Buildkite was stress-tested at the largest scale inside companies solving some of the hardest engineering problems. It’s built to absorb whatever your coding agents throw at the build queue, today. Learn more.

• Sentry – application monitoring software built by developers, for developers. With Sentry MCP and CLI, set up a helpful flow like “when a production error fires, have an agent investigate it, pulling all the error context that Sentry already has. Check out Sentry

In this episode

Kelsey Hightower went from a self-taught technician installing DSL modems to becoming one of Google’s elite Distinguished Engineers, whom the CEO of Microsoft personally tried to recruit. Hightower’s career achievements are rooted in hard work and self-directed learning, and today he’s one of the most influential voices in modern infrastructure, through his talks, open source work, and writing.

In this episode of The Pragmatic Engineer podcast, Kelsey and I cover his unconventional path into tech and the lessons he’s learned during three decades in the industry. We discuss his entrepreneurial years, building a reputation through open source, the rise of containers and Kubernetes, and his time at Google during one of the most consequential periods in cloud computing.

He recounts how a job offer from a big tech giant led to the biggest raise of his career, what prompted him to slow down after years of career acceleration, and we also discuss his perspective on AI. Throughout, Kelsey keeps a simple idea front of mind: that technology is ultimately about people. Whether it’s infrastructure, leadership, careers, or AI, he argues that the goal is not to build technology for its own sake; it’s to solve meaningful human problems.

My observations from the conversation with Kelsey

This is a long episode with many compelling, previously unshared stories. As such, there’s a lengthier-than-usual list of 15 interesting takeaways.

1. Kelsey’s career path is incredibly inspiring. From modest beginnings with no role models in technology, Kelsey worked his way up from technician, to software engineer, and grew into one of the most respected Distinguished Engineers at Google. His drive to improve and to always do his very best work is infectious. My sense is Kelsey would never be satisfied with “good enough” and has always aimed for standout work. This approach is rare and has opened doors that stay closed to average work.

2. Treat every public talk like a job interview. Kelsey’s career inflection points often came from people in audiences offering him jobs. He joined CoreOS because the team watched him PXE-boot CoreOS live on stage; afterwards, they wanted to recruit him.

3. “Some people have 20 years’ tenure – but only one year of experience.” Doing the exact same work for years does not advance your skillset. For example, Kelsey observed people in a call center doing identical manual, ticket-closing work for two decades, who never thought of automating themselves out of even some of it. Kelsey started doing this almost immediately and gained valuable new experience.

4. Side hustles and doing your own thing teach you business like no IC job can. Before becoming a software engineer at Google, Kelsey was a manager for his comedian friend, operated a computer store, and did IT contracting. These gigs taught him logistics, planning, and about money. All this helped him be far more effective at talking with executives and acting as an executive sponsor inside Google.

5. Business owners get paid last, but not employees. Kelsey ran a computer store in Atlanta that did well, but he closed it down to become an employee. As he puts it: “Employees get paid first, the owner gets paid last, and there are months where you get paid last, or you don’t get paid at all.” Kelsey hit a point in his life where he valued the predictability of a salary over volatile income.

6. Leading without influence: don’t tell people the answers, let them be discovered. Kelsey knew that Kubernetes was difficult to onboard and needed the Kubernetes team’s help to fix it. To get this done, he got the Kubernetes team to install K8s with no scripts, watched them struggle, and then guided them to what was missing. After people discovered the problems themselves, they set about fixing them. Kelsey noted that people uncovering problems on their own works better than an issue being handed down to them.

7. Can you explain what your startup does without mentioning AI? When Kelsey researches startups seeking his advice, he challenges founders to not say “AI” once. This means that they must explain the actual value their company creates. One unexpected benefit of this is that it often reveals there are easier, cheaper ways to achieve a goal than with AI.

8. “Look in the mirror”: AI’s impact on the software engineering profession. Kelsey says that when complaining about AI, engineers should bear in mind how their industry has disrupted and displaced jobs in other parts of the economy. It might be one reason why affected software engineers get seemingly little public sympathy.

9. Don’t let agents run loose on raw infra; provide guardrails and context. As Kelsey puts it, “I’ve seen what humans do when you just give them the AWS console. Watch what Claude’s going to do!”

10. It’s okay to interview when you’re happy in a job. Just put your ego aside and check out the market. Kelsey wasn’t looking and didn’t want to bother, but his wife pushed him, saying Kelsey should see what’s out there. She was right!

11. It’s very rare to get an extra zero put on your compensation figure – but it happened. Kelsey was a successful, well-paid Google engineer when Microsoft made him an offer that 10x’d his compensation. When Kelsey told Google he was planning to take the offer, it matched the offer, proving that his market value had massively increased. It shows that being well paid doesn’t necessarily mean you’re being paid at the correct market rate.

12. Satya Nadella: “We gave you an offer as if you were running away from something. We should have given you something to run towards.” Microsoft’s CEO himself acknowledged that when trying to recruit Kelsey, the Windows maker should’ve focused on the mission and growth opportunities, instead of just throwing money at him. It’s a lesson that goes well beyond this specific case.

13. Reframe money as “freedom tokens” instead of status. Once Kelsey stopped caring about impressing others with displays of wealth, money became a means to exit the game, not an end in itself. This reframing changed what he optimized for: to have enough money to not have to work for someone else.

14. Kelsey’s advisory setup: 1-year, no cliff, 10-year exercise window, plus a retainer. Kelsey advises select startups and has learned a few important things:

• Advisory shares alone are usually worth nothing, mostly due to dilution and the tax traps of exercising them.

• Cash retainers ensure he is not working for free. It also means the companies paying him expect impact, and real impact is worth paying for – by serious companies, that is.

Referrals are the fuel for winning more advisory work, and these are based on outcomes. Kelsey advised Pixie Labs, which was later acquired by New Relic, partly thanks to Kelsey’s involvement. Word got around that Kelsey’s advisory could make an impact, so more VCs and founders started reaching out to him.

15. Apply “intentional living” everywhere, not just where it’s comfortable. As a long-time minimalist, Kelsey is intentional about possessions, but realized he was being unintentional in other areas. For example, he now reads lyrics while listening to music to actually understand a song. Intentionality is a habit you extend, not a one-time setting.

The Pragmatic Engineer deepdives relevant for this episode

• Career paths for software engineers at large tech companies

• The past and future of modern backend practices

• How Kubernetes is built

• How Linux is built

• The Staff Engineer’s Path: You’re a role model now (sorry!)

Timestamps

00:00 Intro

03:34 Kelsey’s first job at McDonald’s

05:04 His non-traditional path into tech

11:45 Landing his first tech job with an A+ certification

15:33 His entrepreneurial years

19:45 Joining Google as a data center technician

27:48 Learning automation at a Rackspace spinoff

33:26 Moving into financial services

50:00 Building a reputation through open source

53:55 From configuration management to containers

1:08:20 The rise of Kubernetes

1:25:05 Why he almost joined NASA instead of Google

1:29:20 Defining DevRel at Google

1:38:20 Demonstrating impact at Google

1:41:20 Microsoft’s offer

1:55:20 Learning how to slow down

2:06:39 Advising and investing

2:15:03 A people-first view of GenAI

2:24:27 Using AI with guardrails

2:28:26 Matching AI to the task

2:36:06 Staying relevant in the AI era

References

Where to find Kelsey Hightower:

• X: https://x.com/kelseyhightower

• LinkedIn: https://www.linkedin.com/in/kelsey-hightower-849b342b1

Mentions during the episode:

• TI-BASIC: https://en.wikipedia.org/wiki/TI-BASIC

• Georgia HOPE scholarships: https://www.gafutures.org/hope-state-aid-programs/hope-zell-miller-scholarships/hope-scholarship

• BellSouth: https://en.wikipedia.org/wiki/BellSouth

• FreeBSD: https://en.wikipedia.org/wiki/FreeBSD

• Puppet: https://www.puppet.com

• Rackspace: https://www.rackspace.com

• TSYS: https://www.tsys.com

• James Turnbull’s website: https://jamesturnbull.net

• Kubernetes: https://kubernetes.io

• Red Hat: https://www.redhat.com

• Terraform: https://developer.hashicorp.com/terraform

• Docker: https://www.docker.com

• Mitchell Hashimoto’s new way of writing code: https://newsletter.pragmaticengineer.com/p/mitchell-hashimoto

• CoreOS: https://fedoraproject.org/coreos

• Mesos: https://en.wikipedia.org/wiki/Apache_Mesos

• Go: https://go.dev

• GopherCon: https://www.gophercon.com

• Rob Pike’s blog: https://commandcenter.blogspot.com

• Russ Cox’s website: https://swtch.com/~rsc/

• Brad Fitzpatrick’s website: https://bradfitz.com

• Erik St. Martin on X: https://x.com/erikstmartin

• Brian Ketelsen’s website: https://brian.dev

• Billie Cleek on LinkedIn: https://www.linkedin.com/in/billie-cleek-677b0830

• KubeCon: https://events.linuxfoundation.org/kubecon-cloudnativecon-north-america/

• How Kubernetes is Built with Kat Cosgrove: https://newsletter.pragmaticengineer.com/p/how-kubernetes-is-built-with-kat

• Kubernetes: Up and Running: Dive into the Future of Infrastructure: https://www.amazon.com/Kubernetes-Running-Dive-Future-Infrastructure/dp/1491935677

• Brian Grant on LinkedIn: https://www.linkedin.com/in/bgrant0607

• Eric Tune on LinkedIn: https://www.linkedin.com/in/eric-tune-3033693

• Dawn Chen on LinkedIn: https://www.linkedin.com/in/chendawnhomepage

• Satya Nadella on X: https://x.com/satyanadella

• Hit Refresh: The Quest to Rediscover Microsoft’s Soul and Imagine a Better Future for Everyone: https://www.amazon.com/Hit-Refresh-Rediscover-Microsofts-Everyone-ebook/dp/B01HOT5SQA

• Thomas Kurian on LinkedIn: https://www.linkedin.com/in/thomas-kurian-469b6219

• Liz Rice on LinkedIn: https://www.linkedin.com/in/lizrice

• Pixilabs: https://www.pixilabs.com

• Datadog: https://www.datadoghq.com

• Guillermo Rauch on X: https://x.com/rauchg

• Massdriver: https://www.massdriver.cloud

• Here’s everything the iPhone has replaced in the last 10 years: https://www.cnbc.com/2017/06/29/everything-the-iphone-has-destroyed-in-the-last-10-years.html

• Wix: https://www.wix.com

• Lambda: https://aws.amazon.com/lambda

—

Production and marketing by Pen Name.

I’m in Budapest, Hungary, this week, for Craft Conference, where I’ll be giving a keynote presentation alongside other speakers, including software engineering legend Kent Beck, who’s been on the podcast, Hillel Wayne, a formal methods expert and the author of ‘Logic for Programmers’, and Titus Winters, lead author of Software Engineering at Google.

The title of my keynote is “Slow down to speed up”, and I’ve been thinking about this topic a lot recently. Here are some things I’ve been seeing that I feel are relevant…

AI coding tools now used by pretty much all software engineers – that’s fast!

Read more

Today, we cover:

1. New trend? ROI questions for AI investments. I talked with engineering leaders at mid-sized and large companies, where spending on AI agents is being dampened via per-engineer mont…

Read more

Listen and watch now on YouTube, Spotify, and Apple. See the episode transcript at the top of this page, and timestamps for the episode at the bottom.

Brought to You by

• Antithesis – if you’re using agentic workflows, you need to be extremely clear about what you’re building and how your system should behave. Antithesis brings specification and verification together, making your agents faster, smarter, and safer. And when you’re using Antithesis, you’ll have greater clarity about your code as well. Learn more.

•WorkOS – The fastest AI-native teams have to slow down for the hard problems — WorkOS makes sure auth, for your app and your agents, is never one of them.

• turbopuffer – a search engine that companies like Cursor, Notion, and Linear use to index and retrieve every byte of context for their AI agents. It’s ridiculously scalable, built on object storage, with smart caching on NVMe SSDs so it’s very fast. It also offers many different search indexes and tools: check it out.

In this episode

OpenCode is one of the fastest-growing AI developer tools around, surging in just a few months from roughly 650,000 monthly active users to nearly 8 million, and almost 1M daily active users.

In this episode of The Pragmatic Engineer Podcast, we meet Dax Raad, co-founder of OpenCode, for a discussion about the gaps in developer tooling that led him to build OpenCode, the advantages of open source, and why taste and engineering judgment matter even more as AI becomes a core part of software development.

We also cover how OpenCode turned Anthropic’s blocking of integration with Claude Code into a massive growth lever by partnering with OpenAI and other model providers, why GPU demand is becoming a bottleneck everywhere, how come AI coding tools don’t automatically mean engineering teams move faster, and also why Dax is personally skeptical about predictions for the future of engineering and work, in general.

I found this conversation especially interesting because Dax displays a healthy skepticism toward the benefits of AI, even while building one of the most popular AI coding harnesses.

My observations from the conversation with Dax

Here are 14 of my most interesting takeaways from talking with Dax:

1. AI makes coding easier, but the hard parts of the job don’t vanish. Dax remarks that a lot of the job has become objectively easier with AI, but then follows up with a simple question: why does it feel like he is still having to think as hard as he ever did?

2. Thinking upfront beats building prototypes and seeing what sticks. This is especially true in the period before a product-market fit is found, Dax says. AI doesn’t help much in this early phase because the problem is figuring out what to build, not how fast you can build it, he says. Therefore, thinking hard about the right direction for development beats taking unfocused swings at different ideas.

3. Shipping 10x more features is a recipe for a Frankenstein-like product. It’s tempting to one-to-one prompt an agent for every user’s complaint or competitor’s feature. But the more features are jammed into a product, the worse it tends to become. Also, don’t forget that every shipped feature will need to be supported for as long as it’s part of the product!

4. No AI-native coding agent company is “winning” by being better with AI. Dax says that none of OpenCode’s competitors are crushing them, and that nobody is using AI so well that others cannot compete.

5. For OpenCode, product positioning beats speed of execution. A massive reason for OpenCode becoming the most popular open source AI coding harness is that they noticed no coding agent had successfully claimed the open source category. Dax was wondering why not, given that every market-leading dev tool across the industry is open source. So, he and the team focused on positioning and it paid off handsomely. He summarizes: “Get positioning right and the world just keeps handing you wins you didn’t expect.”

6. OpenCode’s “inverted” strategy: start with a good-enough product, then optimize. Dax admits their harness wasn’t ideal during OpenCode’s first five months, but it was still good enough. “Once we won enough market share, we went back and tried to make our harness good and smart.”

7. Most software engineers profit from AI as time gained, not increased output — unless you change incentives! Dax says the natural way for software engineers to “cash out” their AI tooling gains is with time savings, by doing the same work as before, but faster. Until compensation and motivation structures change, most teams should expect output to stay flat while engineers go home earlier. There’s nothing wrong with this, but AI vendors sell a different outcome to CFOs: increased output.

8. Motivated engineers who care about quality get buried by slop PRs from devs who don’t care. Dax has hired people from companies where they were one of the few who still cared about quality. In contrast, former colleagues just pumped out AI-generated code and focused on getting their tasks done, ignorant of the decreasing quality of code. Motivated devs feel they are drowning in garbage code and tech debt, and getting burnt out by trying to clean it up. Dax calls this an engineering leadership problem that most companies don’t notice.

9. AI code generation mutes the “guilt” of doing the wrong thing, but this builds up tech debt. Pre-AI, writing a hack felt bad, the second time it felt really bad, and by the third time you’d often just refactor in order to fix up the code. Now, the agent hides the hack, which skews devs’ judgment and results in less tech debt being cleaned up.

10. Dealing with tech debt is easier than ever, and teams should do more of it. Agents make refactoring across a codebase cheap: for example, ask an agent to implement a new pattern everywhere across the codebase. It’s very easy and cheap to clear up tech debt, today. So, do more of it!

11. AI has not really changed the thinking / doing ratio for Dax. “Pre-AI, I would spend 95% of my energy thinking about what to do and 5% on doing it. Now I spend 96% of my time thinking, and 4% on actually doing it. So, it’s like a 20% improvement [from 5% doing to 4% doing], but day to day, it feels as hard as ever.”

12. Confident predictions about AI are often forms of self-reassurance. A post went viral on X claiming that 24-29 year-old engineers will dominate in the future, which was written by – you can guess – someone in that exact age bracket. Dax says he sees this pattern a lot and frames such posts in terms of the author making themself feel better: “Someone like me has all the advantages. Someone unlike me has all the disadvantages”. Dax says he’s uninterested in predictions and just focuses on the next task, and the next day.

13. Old “enterprise” patterns are coming back in fashion for writing quality software, as agents are the new junior engineers. Dax says that things like domain-driven design and verbose design patterns went out of style over the past two decades because they’re tedious to type out. But they are actually very useful when there are junior devs on the team – or when there are agents that need strong guardrails. Dax is already using more such “old school” patterns.

14. The future-proof tech career: solid software engineering + deep industry expertise. Dax reckons engineers undervalue how easily they can become industry insiders compared to people who only focus on engineering, but never become an expert in one business area, as they go.

The Pragmatic Engineer deepdives relevant for this episode

• How Claude Code is built

• How Codex is built

• Real-world engineering challenges: building Cursor

• The AI Engineering stack

• How Uber uses AI for development: inside look

Timestamps

00:00 Intro

07:03 Dax’s path into tech

09:04 Early startup experience

13:16 Getting involved with open source

16:13 OpenCode

23:17 Anthropic banning OpenCode

30:34 From terminal to GUI

32:34 OpenCode’s business model

36:33 Why inference is profitable

39:11 GPU bottlenecks

40:54 AI hype

45:50 AI spending

48:47 Dax’s memo

55:41 Dax’s skepticism of predictions

58:58 Engineering culture at OpenCode

1:02:38 How building works at OpenCode

1:05:36 Taste and quality

1:11:32 Dax’s work setup

1:12:35 The role of engineers and EMs

1:15:50 Advice for engineers

1:18:12 Book recommendation

References

Where to find Dax Raad:

• X: https://x.com/thdxr

• Website: https://thdxr.com

Mentions during the episode:

• OpenCode: https://opencode.ai

• Minecraft: https://www.minecraft.net

• IRC: https://en.wikipedia.org/wiki/IRC

• Ride Health: https://www.ridehealth.com

• Serverless Stack: https://sst.dev

• OpenNext: https://opennext.js.org

• Vercel: https://vercel.com

• Red Hat: https://www.redhat.com

• Ubuntu: https://ubuntu.com

• Canonical: https://canonical.com

• OpenCode Zen: https://opencode.ai/zen

• Dax on X “inference is very profitable”:

• The history of servers, the cloud, and what’s next – with Oxide: https://newsletter.pragmaticengineer.com/p/the-history-of-servers-the-cloud

• Dax on X “everyone’s talking about their teams like they were at the peak of efficiency”:

• From IDEs to AI Agents with Steve Yegge: https://newsletter.pragmaticengineer.com/p/from-ides-to-ai-agents-with-steve

• Stripe: https://stripe.com

• TDD, AI agents and coding with Kent Beck: https://newsletter.pragmaticengineer.com/p/tdd-ai-agents-and-coding-with-kent

• How AI will change software engineering – with Martin Fowler: https://newsletter.pragmaticengineer.com/p/martin-fowler

• The third golden age of software engineering – thanks to AI, with Grady Booch: https://newsletter.pragmaticengineer.com/p/the-third-golden-age-of-software

• Mitchell Hashimoto’s new way of writing code: https://newsletter.pragmaticengineer.com/p/mitchell-hashimoto

• Arch Linux: https://archlinux.org

• tmux: https://github.com/tmux/tmux/wiki

• Neovim: https://neovim.io

• Skin in the Game: Hidden Asymmetries in Daily Life (Incerto): https://www.amazon.com/Skin-Game-Hidden-Asymmetries-Daily/dp/042528462X

• The Black Swan: The Impact of the Highly Improbable, second edition: https://www.amazon.com/Black-Swan-Improbable-Robustness-Fragility/dp/081297381X

—

Production and marketing by Pen Name.

A year ago, most engineers found it harder to get responses to job applications, while hiring managers found it harder to hire than before. We’ve now spent two months gathering fresh, never-before-shared data from a range of sources to find out if that paradoxical situation still exists or not.

We collected the data with software engineer Jessica Salmon – who’s joined the team as a tech industry analyst. A big thank you to partner teams who shared exclusive details for this deepdive:

• TrueUp: a platform that scans every open job in Big Tech, top startups, and scaleups, typically paying in the top two tiers of the trimodal software engineering compensation model. Thanks, Amit Taylor.

• Workforce.ai, built by Live Data Technologies, which monitors 1M+ job changes and 300M+ employment validations each month across companies, roles, levels, functions, industries, and locations. Special thanks to Alex Hamilton for his input.

Today, we cover:

1. Software engineering recruitment: trending up, mostly. More software engineering job postings in the UK and the US, and a decline in Germany and France. “Top” tech companies are hiring 20% more vs a year ago.

2. Big Tech and publicly-traded companies. Meta went on a monster hiring spree for the last two years until layoffs, last week. Microsoft and Amazon were flat, while Google and Apple consistently hired devs. Stripe, Shopify, and Atlassian all hired more than Big Tech.

3. Who’s hiring the most software engineers? Apple, Amazon, and IBM are the top 3 by number of positions listed, while Meta dropped off the top 20 with layoffs there. The fastest-growing companies seem to be in fintech, observability, and security.

4. AI engineering: explosive demand. Most tech companies seem to be prioritizing AI engineering recruitment over software engineers. Observability companies appear to be hiring fastest.

5. Who’s hiring the most AI engineers? Apple, Google, and TikTok have the most openings, and many larger tech companies have 50-100% more AI engineering job listings than a year ago.

6. Is AI engineering replacing software engineering hiring? The data doesn’t say, but are the basics of AI engineering becoming a baseline skill for software engineering job candidates?

For more on the tech jobs market of 2025, check out last year’s deepdive.

The bottom of this article could be cut off in some email clients. Read the full article uninterrupted, online.

Read the full article online

1. Software engineering recruitment: trending up, mostly

Top tech companies hiring more, slowly but surely

TrueUp tracks open positions at top-paying companies, which are usually US-based. Among this group, recruitment of software engineers has been steadily increasing since March 2023:

US growth, but less in Canada and Europe

Let’s take a look at broader industry trends via software developer jobs listed on Indeed.com, the world’s largest job aggregator site. As a “jobs aggregator”, Indeed not only displays paid-for ads but also crawls many other career sites and lists the vacancies.

Looking at trends over the last five years, the number of software engineering jobs listed is still lower than in May 2021:

Obviously, 2021 was the pandemic which created a commercial boom for the tech industry, so it’s not exactly an ideal comparison. Zooming into the last twelve months shows encouraging growth signs:

Outside of the US, let’s check out software engineering vacancy trends, per Indeed:

In this list, the US and the UK are the only two countries where vacancies are up; Canada is flat, while Germany and France have seen declines. To me, it suggests US-headquartered companies are hiring more devs, mostly in the US and some in the UK, whereas European-headquartered companies are more cautious about recruitment.

The software engineering industry still grows

Let’s ask how much the software engineer population has grown or shrunk in the last seven years. Here are some numbers from profiles, as tracked by Workforce.ai:

Let’s zoom into this chart:

There are a few interesting details:

• Hiring mostly occurs between the months of March and June, and there’s very little net growth in the second half of the year. This makes sense as tech companies set annual headcount budgets at the beginning of the year and commence hiring from then, with most hires evidently made in the following months. Hiring budgets are usually spent by mid-year.

• The industry seemingly shrank in 2023. That year, it was only in Q2 that more software engineers were hired than left their jobs. It was the worst period for software engineers in two decades.

…and also in 2024 and 2025 during Q3. Mass layoffs mostly happened between August and October, the charts show.

2. Big Tech and publicly-traded companies

Big Tech: modest jobs growth – but not at Meta

Looking at the two years since May 2024, of the five largest tech companies, only Meta grew headcount aggressively:

In that time, Meta increased its headcount by nearly 20%, so perhaps its recent 10% layoffs could be considered as not a total surprise. As for other tech giants, their software engineering headcount also changed in the last two years:

• Apple: +10%

• Google: +5%

• Microsoft: -1.1%

• Amazon: -1.3%

Visualizing this:

Apple and Google seem the most “stable” places when it comes to growing the software engineering function; Apple has not had mass layoffs for decades. In contrast, Meta is the most volatile place to work; hiring devs quickly and then letting them go. Elsewhere, Amazon and Microsoft are slowly employing more software folks.

Meta’s hire-then-fire rollercoaster

Meta warrants a closer look, where morale is reportedly as low as it’s ever been right now, due to layoffs of 10% of staff, while thousands of software engineers have been reassigned to manual data labeling from previous product development or infra work. From 2004 until 2022, Meta never did mass layoffs, but in the past three years there have been three sizeable redundancy rounds (at the end of 2022, early 2024, and last week), with the most rapid hiring growth of any tech giant sandwiched in between:

Before 2022, Meta ranked with Google as one of the most stable places to work in tech. Safe to say, the company is currently the riskiest place to be in job-security terms – even if Mark Zuckerberg has said there will be no more mass layoffs this year.

Publicly-traded tech companies: mostly flat recruitment

A couple of years ago, larger tech companies were hiring even more than most of Big Tech. Software engineering headcount grew in the past two years at Shopify (+36%), Stripe (+29%), Atlassian (+23%), Snap (+15%) and Spotify (+6%).

But that’s changed recently. In the past year, headcount growth has slowed at Stripe (+15%), Atlassian (+11%), Shopify (+6%), Snap (+2%), and Spotify (+1%),

This data does not capture Atlassian letting go of 10% of staff this month, and Snap laying off 16% in April.

Among this group, Stripe stands out as a place that keeps hiring at a higher pace than nearly all of Big Tech. Atlassian and Shopify have grown their software engineering headcount faster than Google, Microsoft, and Amazon did in the past two years.

3. Who’s hiring the most software engineers?

So, which companies are hiring the most? Let’s look at this via open positions and growth comparisons.

Companies with the most open roles

“Top” tech companies with the largest number of software engineering openings:

Compared to last year:

Observations:

• The top three are unchanged: It’s still Apple, IBM, and Amazon

• New entrants to the top 20: Accenture (consulting), Tesla, Cadence (hardware), Hewlett Packard Enterprise (HPE), and SpaceX.

• Dropouts: Meta and Oracle. Last year, Oracle was #4 on the list with 1,396 open software engineering roles, but the company announced up to 30,000 layoffs in March. Also no longer leading in jobs available are Speechify, NTT Data, Meta, and DXC Technology.

• Big Tech companies hiring more than last year: Apple and Google, which has +62% more engineering roles advertised than last year.

• Big companies hiring less vs last year: Meta, Oracle, and TikTok all have much fewer openings listed. For Meta and Oracle, there’s little surprise given recent layoffs. TikTok was in US regulatory limbo until January, when a new TikTok USDS joint venture was established.

• Hardware companies hiring more software talent: Micron, Qualcomm, and AMD all have considerably more software engineering openings than last year.

Where are the “top” software engineering jobs?

Most of these jobs are in the US, just as most companies leading in recruitment numbers are US-headquartered. Other leading locations for jobs are India, the UK, and Canada:

Fastest-growing: fintech and security

Then there are large companies where recruitment has been among the fastest anywhere in software engineering:

Software engineering headcount growth at some companies was significant over the past two years:

• Ramp (fintech): +94%

• Wiz (security): +84%

• Datadog (observability): +68%

• Rippling (fintech): +55%

• Figma (design tooling): +41%

• Netflix (streaming): +37%

Of this group, Datadog, Figma, and Netflix are publicly traded, and the largest by headcount:

• Datadog: AI agents are driving demand for observability through the roof, and Datadog is the leader of the pack. There’s demand to expand their infra and offerings. AI companies are also customers: OpenAI famously – or infamously! – spent around $170M on Datadog in 2025.

• Figma: the design tooling company went public in 2025 and is heavily investing in products related to AI, such as Figma AI.

• Netflix: the streaming behemoth is transitioning from “just” being a video streamer to becoming a broader entertainment platform, doing live events, advertising, gaming, and more. We did a podcast with Netflix CTO, Elizabeth Stone, in 2025 about Netflix’s engineering culture.

4. AI engineering: explosive demand

AI engineering demand is only going in one direction: upwards, at a fierce tick. Last September, our previous State of the Tech Job Market report found that AI Engineering was the hottest segment in tech for job listings growth. Today, the trend seems to continue:

Read more

Today, we cover:

1. Antigravity 2.0 takes the ‘IDE’ out of its new IDE. Feedback about the redesigned IDE is overwhelmingly negative due to bugs, poor UX & model support, and eating through Gemini token quotas. Also: a clue that Antigravity’s own devs use other tools for their work?

2. Why is Google’s product ecosystem chaotic? The range of products on display at the Google I/O conference made a messy, incoherent impression. But Google’s “let a thousand flowers bloom” approach might be giving the search giant an underrated advantage in the AI race that no other Big Tech giant has.

3. Meta cuts 8,000 jobs. Morale is very low inside the social media giant as thousands lose their jobs, just as revenue and profits hit record levels. Meanwhile, those assigned to dull data labeling work are spared the axe.

4. Industry pulse. Anthropic pays $15B/year for SpaceX compute, SpaceX’s financials and IPO filing, more woes for GitHub, court dismisses Elon Musk’s “hypocritical” OpenAI lawsuit, and Spain may stop blocking its internet during La Liga football games.

5. How to get a job at a frontier lab in 2026. A Distinguished Engineer at Google recommends focusing on developing particular skills

1. Antigravity 2.0 takes the ‘IDE’ out of its new IDE

Read more

Listen and watch now on YouTube, Spotify, and Apple. See the episode transcript at the top of this page, and timestamps for the episode at the bottom.

Brought to You by

• Antithesis – if you write Rust code, check out Hegel-Rust: a free, open-source property-based testing library for Rust, built by the team behind Hypothesis. It’s now maintained by Antithesis. If you try Hegel and like it, your Hegel tests will run in Antithesis as written, as well.

• Sentry – application monitoring software built by developers, for developers. I’ve used Sentry for 10 years now, starting back at Uber. It shows you the full context on issues: stack traces, user actions, environment details, and more. A new, recent feature: Seer, their debugging agent — I like it! Check out Sentry.

• Craft Conference: meet myself, Kent Beck, Hillel Wayne and other standout speakers on 4 June, in Budapest, Hungary, at the conference dedicated to the art and science of software delivery craft. See details.

In this episode

Rust is one of the most admired programming languages around – and also one of the hardest to learn. What makes developers stick with it?

In this episode of The Pragmatic Engineer Podcast, I sit down with Alice Ryhl, a software engineer on Google’s Android Rust team, and a core maintainer of Tokio, which is the most widely-used async runtime in Rust.

We discuss what makes Rust different from other languages like TypeScript, Go, and C++, and why so many developers say that “once it compiles, it works.” We go deep into memory safety, ownership, borrowing, unsafe Rust, and Cargo.

We also cover how Rust is governed by RFCs, feature flags, its six-week release cycle, how engineers get paid to work on the language, and also look into how Rust’s use inside the Linux kernel is progressing.

My observations from the conversation with Alice

Here are 12 of my most interesting takeaways from talking with Alice:

1. Open source contributions can open career doors. Alice landed her Google job not by sending in an application, but by spending years answering questions on Rust forums, and contributing to documentation and code. She also became a maintainer of Tokio while a college student. One day, an email arrived in her inbox, asking: “do you want to work on the Android Rust team?”

2. Reliability is the backend pitch for choosing Rust over TypeScript. Alice is adamant that Rust is a backend language and not a TypeScript replacement on the frontend. Rust’s design for minimizing errors and maximizing reliability can make it a better choice than TypeScript on the backend.

3. Rust over C++ for development is a strong choice for avoiding a host of security vulnerabilities. In C++, a trivial off-by-one error in an array can become a massive security vulnerability. In contrast, Rust’s memory safety eliminates an entire class of such bugs, unless you are brave – or foolhardy – enough to use an “unsafe” keyword, that is!

4. Rust was designed to turn implicit failures into compile errors. Where other languages allow you to forget something, Rust makes an omission into a compilation error for things like null checks, uninitialized variables, or error propagation with the ‘?’ character. If you mess something up, it’s almost certain your program will not compile. If it does, at the very least you should see a lint warning. We previously covered how Andrey Breslav, creator of Kotlin, purposely made null safety an important part of Kotlin.

5. The hardest part of learning Rust is not syntax, but data structure design. Alice observes that newcomers reflexively build cyclic object graphs, like a Book object referencing Page objects that refer back to the Book. Such cyclic graphs are possible in Rust, but its ownership model makes this hard, meaning that Rust novices end up battling the compiler. Obvious solutions are to use structs and understand how ownership works in Rust.

6. Refactoring in Rust is safe and easy, thanks to the compiler. Alice: “I change a return type or struct field, then just fix the compiler errors until the compiler stops shouting. And then once I’ve done that, I’ve updated every place I need to update.” Rust’s focus on correctness makes refactoring it more straightforward than dynamically-typed languages and Java-style typed ones are to refactor.

7. Rust may be optimal for AI agents because of the compiler’s high-quality feedback loop. Alice’s refactoring trick of just doing what the compiler says also applies to agents: they can talk to the compiler, be told what to fix, and iterate. Combined with Rust blocking entire bug classes by design, this makes it one of the better languages for agent-generated code.

8. “Editions” allow Rust to make breaking changes without ‘breaking’ anyone’s code. Rust editions (2015, 2018, 2021, 2024) can be mixed freely across crates. A library on the 2021 edition works seamlessly with a binary on the 2024 edition. This is how Rust evolves syntax (like adding async/await as keywords) without forcing an ecosystem-wide migration.

9. Rust’s governance precludes a “benevolent dictator for life”. Unlike with Python and Linux, teams in Rust self-organize and delegate to each other. Tough questions are hashed out at in-person events like ‘Rust All Hands’. It’s a good illustration that open source projects can thrive across different structures.

10. Rust in the Linux kernel has graduated from “experimental.” At December 2025’s Linux Kernel Maintainer Summit, the kernel community agreed Rust is no longer experimental. Combined with US Department of Defense regulations pushing agencies away from non-memory-safe languages, this means we should see more Rust in the Linux kernel and everywhere else, too.

11. AI code review may matter more than AI code generation in safety-critical codebases. The kernel community is experimenting with AI bots that review mailing-list patches. Maintainers reportedly find them impressive, especially for kernel code; an area where quality and reliability has always been more important than quantity. It’s interesting that AI might be helpful as an extra quality gate.

12. Risk of AI-assisted Rust: false fluency. Since Rust’s compiler is so strict, code that compiles can be assumed to be correct. However, Alice describes AI agents adding Rust versions of C build flags with no purpose! She also cautions that junior engineers using AI to learn Rust run the risk of not understanding why the compiler accepts the code they produce.

The Pragmatic Engineer deepdives relevant for this episode

• The past and future of modern backend practices

• How Kotlin was built with Andrey Breslav

• How Swift was built with Chris Lattner

• How Linux is built with Greg KH

Timestamps

(00:00) Intro

(04:09) Tokio: an overview

(05:11) What Alice likes about Rust

(12:48) Rust for TypeScript engineers

(13:51) Moving from C++ to Rust

(14:34) Memory safety

(18:12) Garbage collection tradeoffs

(21:46) Ownership, references, and borrowing

(26:59) Unsafe in Rust

(31:21) Crates and Cargo

(35:55) Language design and RFCs

(43:02) Building new features

(46:30) Editions vs. versions

(49:47) Getting paid to work on Rust

(51:27) Contributing to Rust

(53:03) Rust in the Linux kernel

(55:45) AI use cases for Rust

(1:01:35) Learning Rust

(1:03:54) Book recommendation

References

Where to find Alice Ryhl:

• LinkedIn: https://www.linkedin.com/in/aliceryhl

• Website: https://ryhl.io

Mentions during the episode:

• Rust: https://rust-lang.org

• Tokio: https://tokio.rs

• Minecraft: https://www.minecraft.net

• Rust Users Forum: https://users.rust-lang.org

• Null’s creator regrets inventing it: https://news.ycombinator.com/item?id=12427069

• PHP: https://www.php.net

• Go: https://go.dev

• TypeScript: https://www.typescriptlang.org

• C++: https://en.wikipedia.org/wiki/C%2B%2B

• Pip: https://pypi.org/project/pip

• Why Cargo Exists: https://doc.rust-lang.org/cargo/guide/why-cargo-exists.html

• Linus Torvalds: https://en.wikipedia.org/wiki/Linus_Torvalds

• Rust Week: https://2026.rustweek.org

• Inside Amazon’s Engineering Culture: https://newsletter.pragmaticengineer.com/p/amazon

• How Linux is built with Greg Kroah-Hartman: https://newsletter.pragmaticengineer.com/p/how-linux-is-built-with-greg-kroah

• Community Grants Program: https://rustfoundation.org/media/tag/community-grants-program

• Zulip: https://forge.rust-lang.org/platforms/zulip.html

• The Linux Kernel Archives: https://www.kernel.org

• Linux Plumbers Conference: https://lpc.events

• Gemini: https://gemini.google.com/app

• The Rust Programming Language: https://doc.rust-lang.org/book

• Rust for Rustaceans: Idiomatic Programming for Experienced Developers: https://www.amazon.com/Rust-Rustaceans-Programming-Experienced-Developers/dp/1718501854

• Rustlings: https://rustlings.rust-lang.org

—

Production and marketing by Pen Name.

1. Tradeoffs of AI tooling. Positive and negative sentiments, like how AI tools often mean less time spent on tedious, repetitive work, but that it often leads to unrealistic business expectations.

2. Adopting AI at scale is hard. Companies are struggling to achieve adoption that feels productive at the team and org levels. Our survey finds the benefits of AI heavily depend on the engineering culture that was in place before.

3. Impact on codebase quality. Codebase quality seems to be decreasing, but management at most places does not care. Meanwhile, maintenance duty is falling upon a shrinking number of engineers who still understand increasingly complex codebases.

4. Less experienced engineers and AI. For these folks, AI seems to be less helpful and they rack up higher AI token bills. Maybe they need more space to learn, mentorship, and support.

5. AI tooling ‘addiction’. Using AI agents “feels like a slot machine” encouraging “just one more prompt”-type behavior, while some folks think that the pricing of plans is built in a way to “lure” them to prompt more and more.

6. Changes since 2024. Fewer devs are negative about AI, but there’s not all that much more positivity. The models have become much higher quality, and better tooling improves trust.

7. Where are we headed? The concept of code ownership seems to be eroding, and collaboration within teams becomes less important.

Previously in this series, we covered:

• AI tooling for software engineers in 2026: a detailed summary of survey responses, covering the most-used AI tools, trends, AI agent usage, company size and usage, and tools engineers love.

• The impact of AI on software engineers in 2026: key trends. Part 1. Concerns about mounting AI costs, more engineers hitting usage limits, and AI tools having uneven effects upon different types of engineers

Full subscribers can access a more detailed report.

1. AI tooling tradeoffs

What can be said about the impact of AI tooling on engineers this year? Well, based on the responses that readers sent in to our survey, there are some common patterns:

Code and output tradeoffs

Productivity tradeoffs

Research published two weeks ago (5 May) by Microsoft claims AI expands the pool of people who can do high-value work. Our survey found similar, while MS’s findings are based on Microsoft 365 Copilot chat usage.

Obviously, Microsoft has an interest in showing the benefits of its AI tools, but in this specific area, it’s telling that both studies agree AI does allow devs and non-devs to expand the type of work they do.

Software engineering tradeoffs

With AI tools, their ‘mileage may vary’ among individuals. For example, some respondents told us they can spend much more time in a “flow state” thanks to AI tools, as they don’t have to wait for input from peers, can keep unblocking themselves, and have fewer interruptions.

But others say the opposite: that because they can start so many more tasks in parallel with AI tools, they keep context switching which knocks them out of a flow state!

The positives and negatives of these tools seem dependent on the environment respondents work in, individuals’ personality traits, and where users are on the AI learning curve.

2. Adopting AI at scale is hard

One pattern in the responses is that adopting AI at company-scale remains challenging, including:

• Costs: a growing concern covered in depth in Part 1

• Usage: getting people to use AI tools continuously is not always straightforward

• Onboarding and education: at larger companies, there needs to be support to help devs make the most of the tools

• Reviewing AI-generated output: code review is a particular pain point

• Integrating with internal systems: AI tools are more helpful when they are seamlessly integrated with internal systems, hence why many larger companies use in-house, deeply embedded coding agents

Adopting AI at company-level doesn’t lead to a magical fix for engineering problems, while the benefits of AI in an organizational sense seem to depend on what was in place before.

Case in point, AI seems to amplify pre-existing engineering culture. AI doesn’t change the underlying quality of an organization’s engineering culture. Teams with strong engineering practices get more positive benefits out of AI than those without. Teams that see benefits from AI tools already had:

• Guardrails: testing and automation around the codebase and deployments

• Documentation: they recorded their architectural decisions and engineering practices

• A quality codebase: AI agents will replicate patterns already in a codebase

A few quotes on this from respondents to our survey:

“AI is an amplifier, not a fixer. Good software engineering practices get multiplied. So do the bad ones. Embedding this properly in teams is exciting and important”. – Staff+ engineer at a large company in Europe

“I feel like AI allows both faster prototyping and increased velocity on iterations to production software; it relies on existing best practices / project templates our team already have”. – Solutions Architect at a small company in the US

A workflow that makes one dev “10x” more productive may not work for another. This is another reason why rolling out AI tools doesn’t seem to magically make everyone more productive. A senior engineer working at a large company in Canada told us:

“It feels like AI workflows are very idiosyncratic in that some people derive (I hate this framing, but…) 10x more productivity benefit from them than other apparently equally clever, educated, and diligent developers. It feels like finding a workflow that clicks with your own habits and heuristics is more important than finding a global optimum for everyone”.

AI amplifies individuals differently, so the team impact is messy to figure out. A US-based principal engineer at a large company reflected on feeling disconnected from colleagues because of how they use AI differently:

“I use AI in what I think is probably a more sophisticated way than most of my colleagues, so there can be a disconnect between my work and theirs, which is not good news because I am “The Principal” on the team”.

“The tool that works for you” approach can lead to tooling chaos, even at a team level. While it’s empowering to allow devs to choose the AI tools they feel are the most helpful, over time it becomes chaotic when teams can choose their own tech stacks, and when at large companies there are dozens of different technologies. A staff+ engineer at a 200-person business in the Middle East wrote in their response:

“We’re still trying to figure out how to deal with tooling consistency on a team level. It’s one of our biggest struggles, but possibly more due to company structure than anything else. Everyone is using different tools with little coherence. It’s been rough.”

Some companies have briefly rolled back AI to deal with the negative effects first. From an engineering lead at a 10,000+ person company in Europe:

“Since the AI boom, the quality of technical writing and reasoning from senior engineers in my org has significantly deteriorated. There’s an overwhelming volume of low-quality work product that is generated entirely or in part by AI, which has made it very difficult to conduct meaningful review of RFCs or code. We’ve also seen costly production incidents caused by code written and/or approved by AI, and – while my employer initially bought heavily into the hype – we have now rolled back some of our AI tools to deal with the drop in quality.”

3. Impact on codebase quality

A concrete pattern in our survey data is that codebase quality is decreasing due to AI. The contributing factors are not surprising:

• “AI slop”: more low-quality code generated, such as duplicated, verbose code, and poor abstractions

• Too many code reviews, which means review quality slips

• More bugs: due to faster code output and less strict reviews, more bugs sneak into codebases

We discuss the degradation of products and codebases in a recent deepdive, Are AI agents actually slowing us down?

One CTO at a European startup lists their negatives about increased AI usage:

1. “A lot of tiny bugs and low code quality if you are not careful, verify carefully, and have good structure and guardrails

2. AI agents generate too much and repetitive code, making systems harder to maintain

3. Developers lose understanding of the codebase and become numb to bad architecture and bad developer experience”

According to our survey, management often seems unfazed by decreasing quality, and instead focuses on the higher output. A principal DevOps engineer at a large European company said:

“In our company, we hand AI tools to inexperienced engineers who can’t distinguish good code from bad code and it’s falling on deaf ears in our leadership. They only seem to care about short to mid-term cost savings.”

The maintenance burden of AI-generated code is falling on the fewer engineers who understand and care. A staff engineer at a European company listed the problems that AI-generated code is causing:

• “Drive by” contributions are up: many more occasional non-core-engineer contributors adding code but not sharing the maintenance burden

• Contributing without adding guardrails: many engineers and most of engineering leadership are not using reasonable guardrails like tests

• AI slop from folks who have nothing to do with the codebase: huge volume of slop incoming from people who don’t understand the codebase, but will commit and create PRs without fully understanding what they’re doing

• Complexity is exploding: thanks to the above

The maintenance budget is falling upon fewer devs, while the task of refactoring bloated codebases and reducing complexity is left to those still sufficiently in touch with the codebase, thereby making the maintenance burden even worse.

But some leaders “get it.” A CEO at a 20-person company told us:

“While AI has made generating code ‘cheaper’, the monitoring and maintenance worry me; the things that have traditionally cost the most in software. We’re increasing the rate of shipping large amounts of code with less understanding and increasing the unpredictability, so how do we work the predictability back on top?”

There is industry pressure on companies to adopt AI tooling and impose its usage upon engineers, driven by a mix of factors:

• Seeing actual benefits of AI and hearing that other teams and companies enjoy success with it

• Fear of being left behind by competitors, or becoming less relevant

• Anxiety about investor interest if a company is seen as not adopting the latest AI tools.

This often leads to:

• Top-down mandates to use AI

• Expectation of headcount reduction, with smaller teams doing the same amount or more

• Management treating AI productivity gains as a baseline rather than a bonus.

One staff engineer at a 10,000-person US company explained it like this:

“AI is part of almost every work conversation. The entire company expects it to increase productivity and reduce the need to hire people. I keep trying to get better at using it and trying to make it more reliable so I can do more. I do worry about the quality of the work and atrophy of certain skills. It’s unclear to me if those skills even matter anymore.”

Pushing AI adoption blindly triggers red flags. Respondents shared what makes them worry about things going south in their workplaces:

• Focus on tracking AI usage, but not the quality of the output. This will likely lead to product regressions and unhappy customers

• Pushing for universal adoption. Some companies target 50%, 80%, or 100% AI usage for certain tasks, seemingly blind to how some targets can worsen the quality of output, or simply create wasteful usage. See the trend of tokenmaxxing.

• A focus on velocity, but without recognition of quality work. Expecting more velocity and output seems to be the baseline, and there is no recognition for work truly well done.

The “move fast and break things” mantra famously championed by Facebook seems widespread across the industry with AI tools. A senior manager working at a large, European-headquartered company told us:

“I see a trend: move fast and break things, and end up breaking things too often. We have to learn to focus on testing and resiliency a lot more, as with AI-driven development we introduce more bugs than before. But the velocity gain is bigger for now”.

Output over quality is leading to the death of code review at some places. As a lead engineer working at a small company summarized:

“We’re at the death of code review. I used to do very deep code reviews where I’d take the time to understand the architecture and organization and provide feedback on maintainability and efficiency. I have no motivation in spending that time to review a giant PR where it’s clear that even the original author didn’t bother to do that”.

4. Less experienced engineers and AI

The first generation of software engineers who have never developed without the help of AI are now entering the industry. Here is a response from a young engineer working at a startup as an intern:

“I have never worked as a developer without AI. Writing this scares me a bit, actually, but it’s the truth!”

But this will be the new reality for those joining the industry. So, what needs to happen to help a new generation of “AI-native” grad engineers grow professionally?

AI is an amplifier which could amplify the lack of experience. A staff engineer in the US at a large company told us:

“Agentic AI is a fascinating mirror. It can code as well as the user who drives it. If that user is a junior engineer, now you have a faster junior engineer. If the user is a staff engineer, now you have a faster staff engineer.

What agentic AI doesn’t do is magically convert a junior engineer into a staff engineer, because the user driving it still needs enough experience to know what a good solution looks like”.

A junior engineer in Australia shares their experience of how frustrating working with AI tools is:

“I think AI agents are great for vibe coding or prototypes where the code quality and functionality doesn’t matter that much. I think it’s also useful for senior engineers who know what they’re doing.

For junior engineers like myself, these AI tools are stressful to use. I don’t have the experience or knowledge to tell AI exactly what to do or quickly confirm its output, so I spend a lot of time on just triple checking and redoing stuff. I’m overall frustrated, but I’m trying to embrace it as we’ve been asked to by the company”.

Less experienced engineers seem to use more AI tokens and rack up higher bills. Several respondents observed this: director-level folks noted that junior engineers are in the top-spender category in their orgs, and it is junior devs who spend tokens on unproductive use cases.

There should be more space for junior folks to grow because they use AI more, one staff engineer respondent said:

“Companies need to give some breathing room to Junior engineers and help them learn and acquire knowledge using AI tools as a booster and not as a replacement”.

Junior folks seem to be delegated fewer opportunities that could help them grow. This is because senior people can turn to AI for tasks, including those which they would have previously delegated to an intern or new grad. A few responses mention this:

“AI allows me to have work done that I would usually delegate to a junior or pay a SaaS for; e.g., writing drafts, summarizing the news. “ DevSecOps lead at a small company, Europe

“I’ve begun to automate any repetitive task that we previously relied on juniors and offshore contractors for.” - Engineering manager, at a large company, US

“I no longer have to delegate work by writing a very long document and briefing a junior engineer.” - Principal engineer, large company, Europe

Why not consider mentoring junior devs in your organization? It’s clear that less experienced engineers are having a rocky start to their careers, so delegating stuff to them instead of to AI could be of high value for newer generations of talent.

5. AI tooling ‘addiction’

It seems that the rapid feedback loops of AI-assisted development create addictive tendencies, and there’s a noteworthy presence of “addiction lingo” in some responses to our survey:

Read more

Today, we cover:

1. Forward deployed engineering heats up again. Massive demand for the role at Google, OpenAI, and Anthropic. The latest version of the FDE role looks like the consultant / solution a…

Read more

Listen and watch now on YouTube, Spotify, and Apple. See the episode transcript at the top of this page, and timestamps for the episode at the bottom.

Brought to You by

• Antithesis – verify your system’s correctness in the development loop. Build, extend and deploy mission-critical systems: without human review or traditional integration tests, all while avoiding embarrassing outages. Antithesis goes far beyond code review and runs your complete system, exhaustively analyzing its behavior, and exposes bugs as quickly as agents introduce them. Learn more.

• WorkOS – Anders has spent decades thinking in programming languages — WorkOS speaks the same language when it comes to enterprise infrastructure: SSO, SCIM, RBAC, and more, built right so you never have to. See how.

• turbopuffer – a ridiculously scalable, fast and cheap vector and full-text search engine. Cursor moved to turbopuffer after their existing vector database couldn’t keep up with the number of codebases they were adding. This move cut their semantic search costs by 95%, and they have so many good things to say about turbopuffer. Check it out.

In this episode

Anders Hejlsberg is a living legend and one of the most influential programming language designers of all time. He created Turbo Pascal, Delphi, C#, and also TypeScript. As well as that, he spent nearly a decade at the pioneering dev tools company, Borland, and is now in his 30th year of working at Microsoft, where he’s a Technical Fellow.

In this episode, we discuss what it takes to build programming languages that developers love to use, and trace his career from writing his first compiler to creating Turbo Pascal and Delphi, and helping to pioneer modern software development through C# and TypeScript.

Anders details how C# was designed by a small group of experienced language designers who met a few hours each week, and he explains why tooling was just as important as the language for TypeScript’s success, and what he has learned from building languages which stay relevant for decades.

We also look into how Anders uses AI today, which language features suit AI-assisted development, and what he thinks is changing in the craft of software engineering as developers move further away from writing code line by line.

It was special for me as well to talk with Anders: because my first-ever programming language was Turbo Pascal, and my first paying job was at a C# shop, back in the day. And I write most of my backend services using TypeScript, today.

My observations from the conversation with Anders

Here are 14 of my most interesting takeaways from talking with Anders:

1. The first computers had zero abstractions on top of the hardware. Anders started programming on an HP 2100 with 32K of memory and a paper tape boot loader. “Things were so simple. You could see all the way to the bottom. There was just no layering. It was right on top of the hardware,” he says. The past 50 years of computing have been a process of adding layers above the hardware.

2. Turbo Pascal won by being more than “just” a compiler. Anders also built an IDE to edit Turbo Pascal programs in, and a debugger to troubleshoot them with. In later languages which Anders designed, he always focused on the entire developer workflow, not just compilation.

3. “10x better for 1/10th of the price” is a proven winner. This is what Turbo Pascal did: it sold for $49.95 when competing compilers cost $500, and it was faster and more interactive than competitors’ products. Conveniently, the low price tag also killed off piracy

4. The first Turbo Pascal “debugger” was an elegant hack. The compiler printed the program counter address on a runtime error. Then you could re-run the compiler in a mode that stopped at that address, and the IDE would show which line was being syntactically processed at that point. There were no line maps or debuggers built for this debugging experience: Constraints force creativity!

5. C# might have not existed without a famous court case. Microsoft originally hired Anders to architect its Java tools (Visual J++), but the Sun versus Microsoft lawsuit (1997-2001) meant Microsoft could not build on top of Java, as the company that owned Java’s IP (Sun) sued MS for alleged unauthorized changes to the Java language. Microsoft realized it had to build a new language that combined VB’s productivity with C++’s power. This led to C# and .NET.

6. The original C# design team numbered only six. They held three meetings per week which lasted two hours each, during which they debated what to build. Anders believes that designing even large languages does not require a large team; it’s the right experience that matters most. In that group, this meant folks who had built languages before.

7. C# introduced the async/await pattern that many languages later ran with. Anders said this pattern spread to so many other languages because compilers can generate state machines that humans hate writing. Manual state machines require moving all stack states into objects and wrapping logic in a giant case statement. Devs generally loath doing this by hand, and async/await lets developers write sequential-looking code while the compiler does the painful rewrite behind the scenes.

8. TypeScript exists because Anders refused to build Script# for the Outlook.com team. Microsoft’s Outlook.com team asked Anders’ C# team to productize “Script#,” a language to cross-compile C# to JavaScript. Anders and the C# team pushed back, suggesting that a better approach was to fix JavaScript. Anders felt strongly that to be attractive to the best-of-breed developers in the JavaScript ecosystem, you want people to write JavaScript, and not another language like C#.

9. Open development on GitHub made TypeScript much better. TypeScript was open source from the beginning in 2012 on CodePlex, Microsoft’s open source platform. There was not much community activity there, and in 2014, the project moved to GitHub with its large, active community. “Open development” on GitHub is what Anders credits for making TypeScript as good as it has become.

10. The TypeScript compiler breaks many traditional practices. The compiler is built to support lazy evaluations, and deferred imports, and is functional by necessity. For example, with 500 files open, the compiler keeps abstract syntax trees (ASTs) cached for 499, and rebuilds just the one being edited. It only resolves the minimum types needed for the cursor’s current position.

11. Training data volume is what makes AI great at TypeScript and Python. Anders says the language best suited for AI is the one that AI has seen the most in its training set. How well AI agents work in a specific language largely depends on how much of that language exists on the internet. It has less to do with the design of a language.

12. But AI is limited for writing compilers – for now, at least. On Anders’ team, AI is limited in terms of seeing the “big picture”; for example, how types, symbols, binding, and parsing all relate. It’s because LLM training sets don’t contain much about compilers as yet.

13. Reviewing code could be the future of the craft. Anders predicts we’ll all be project managers in the future, managing armies of junior programmers, aka agents, which generate reams of code. Anders admits he is less interested in reviewing code, personally, but reckons code review could be made much more interesting; for example, AI generating commentary that guides reviewers through changes.

14. Designing a programming language is a 10-year play. As Anders puts it:

“Version one is great, but has all sorts of issues. You’ve got to do version two, but it’s not until version three that it really starts to be great. Then you’ve got to convince people to adopt it.”

The Pragmatic Engineer deepdives relevant for this episode

• Microsoft’s developer tools roots

• 50 Years of Microsoft and developer tools with Scott Guthrie

• How Linux is built with Greg Kroah-Hartman

• How will AI change operating systems? Part 1: Ubuntu and Linux

• How Uber uses AI for development: inside look

Timestamps

(00:00) Intro

(02:48) How Anders got into programming

(05:40) Building his first compiler

(07:44) Turbo Pascal

(12:25) Delphi

(14:53) Joining Microsoft

(19:41) Building C#

(29:11) Async/await

(34:01) The rise of JavaScript

(37:52) Building TypeScript

(42:58) How the TypeScript compiler works

(48:30) JavaScript’s strengths and weaknesses

(52:18) How Anders uses AI

(56:03) What language features work well with AI

(1:02:49) How software craftsmanship is changing

(1:07:49) Performance and efficiency

(1:09:29) Anders’ tool stack

(1:11:30) A 30-year career at Microsoft

(1:13:40) Book recommendation

References

Where to find Anders Hejlsberg:

• X: https://x.com/ahejlsberg

• LinkedIn: https://www.linkedin.com/in/ahejlsberg

• GitHub: https://github.com/ahejlsberg

Mentions during the episode:

• Turbo Pascal: https://en.wikipedia.org/wiki/Turbo_Pascal

• Borland: https://en.wikipedia.org/wiki/Borland

• Delphi: https://en.wikipedia.org/wiki/Delphi_(software)

• Visual Basic: https://learn.microsoft.com/en-us/dotnet/visual-basic

• Skype: https://en.wikipedia.org/wiki/Skype

• J++: https://en.wikipedia.org/wiki/Visual_J%2B%2B

• Java: https://en.wikipedia.org/wiki/Java_(programming_language)

• Legal Newsroom Archive: Sun Microsystems Inc. and Microsoft: https://news.microsoft.com/case-archives/sun-microsystems-inc-and-microsoft

• .NET: https://dotnet.microsoft.com

• Roslyn: https://en.wikipedia.org/wiki/Roslyn_(compiler)

• Async/await: https://en.wikipedia.org/wiki/Async/await

• Go: https://go.dev/

• JavaScript: https://en.wikipedia.org/wiki/JavaScript

• Brendan Eich’s website: https://brendaneich.com

• Steve Ballmer on X: https://x.com/Steven_Ballmer

• Luke Hoban’s website: https://lukehoban.com

• TypeScript: https://www.typescriptlang.org

• GitHub: https://github.com

• Language Server Protocol: https://en.wikipedia.org/wiki/Language_Server_Protocol

• Caml: https://caml.inria.fr

• PHP: https://www.php.net

• Book recommendation: Algorithms + Data structures by Niklaus Wirth = Programs. Full book: https://www.cl72.org/110dataAlgo/Algorithms%20%20%20Data%20Structures%20=%20Programs%20%5BWirth%201976-02%5D.pdf Amazon: https://www.amazon.nl/-/en/Niklaus-Wirth/dp/0130220051

—

Production and marketing by Pen Name.

Four decades ago, the writer of ‘The Mythical Man-Month’ (1975), drew on folklore about werewolves to publish a paper about the prospects of a so-called silver bullet for software development that would make professionals much more productive at their craft.

Frederick P. Brooks published “No Silver Bullet – Essence and Accident in Software Engineering” in 1986, and as the title suggests, it is pessimistic about the existence of any silver bullets. The term refers to a super weapon capable of dropping otherwise near-unstoppable werewolves and other creepy supernatural beings in European folk tales.

Since its release, this paper might have become even better-known than Mythical Man-Month (MMM). In 1995, the second edition of that book included Brooks’ later essay as chapter 17, along with an additional chapter of reflections.

In this article, we look into whether the essay was correct in its disbelief in silver bullets, or whether any did indeed slay the beast of unproductivity for developers over the course of time. Also, how does AI agents generating so much code, as of today, challenge the entire premise – or not?

We cover:

1. “No silver bullets” – why has it held up? No single new technology or methodological breakthrough by itself introduced magnitudes-of-improvement to the areas that really matter in software engineering. Is that unusual?

2. Is SRE a silver bullet? Google’s Search team introduced the SRE discipline, and won orders-of-magnitude superior reliability to its competitors. But why only Google Search?

3. Was open source + GitHub a silent silver bullet? No development had a bigger impact on the wider tech industry than the open source wave since the 2010s. Has it been a silent silver bullet, an overlooked cause?

4. Could AI be a silver bullet? At first glance, AI generates 100x-or-more code output. But productivity, reliability, and simplicity improvements are a bit unimpressive – at least for now.

Brooks was a computer scientist who led IBM’s System/360 and OS/360 operating systems development, ‘The Mythical Man-Month’ was published in 1975. Last year, we did a deepdive into this engineering classic (Part 1, Part 2, Part 3, Part 4), delving into its predictions and legacy.

1. No silver bullets?

The paper delves into folklore for its motif, a ‘silver bullet,’ and uses it to pose the question of whether there would be any “silver bullets” on the horizon (in 1986) that could be similarly fatal to software engineering complexity. From the paper (emphasis mine:)

“Of all the monsters who fill the nightmares of our folklore, none terrify more than werewolves, because they transform unexpectedly from the familiar into horrors. For these, one seeks bullets of silver that can magically lay them to rest.

The familiar software project has something of this character (at least as seen by the non-technical manager), usually innocent and straightforward, but capable of becoming a monster of missed schedules, blown budgets, and flawed products. So, we hear desperate cries for a silver bullet, something to make software costs drop as rapidly as computer hardware costs do.

But, as we look to the horizon of a decade hence, we see no silver bullet. There is no single development, in either technology or management technique, which by itself promises even one order of magnitude improvement in productivity, in reliability, in simplicity.

Skepticism is not pessimism, however. Although we see no startling breakthroughs, and indeed, believe such to be inconsistent with the nature of software, many encouraging innovations are under way. A disciplined, consistent effort to develop, propagate, and exploit them should indeed yield an order-of-magnitude improvement. There is no royal road, but there is a road.

The first step toward the management of disease was replacement of demon theories and humor theories by the germ theory. That very step, the beginning of hope, in itself dashed all hopes of magical solutions. It told workers that progress would be made stepwise, at great effort, and that a persistent, unremitting care would have to be paid to a discipline of cleanliness. So it is with software engineering today.”

In 1995, Brooks revisited his idea that silver bullets weren’t real in the software domain. From the Mythical Man-Month’s anniversary edition:

“No Silver Bullet” asserts and argues that no single software engineering development will produce an order-of-magnitude improvement in programming productivity within ten years (from the paper’s publication in 1986). We are now nine years into that decade, so it is timely to see how this prediction is holding up.

Whereas The Mythical Man-Month generated many citations but little argument, “No Silver Bullet” has occasioned rebuttal papers, letters to journal editors, and letters and essays that continue to this day.

Most of these attack the central argument that there is no magical solution, and my clear opinion that there cannot be one. Most agree with most of the arguments in “NSB,” but then go on to assert that there is indeed a silver bullet for the software beast, which the author has invented. As I reread the early responses today, I can’t help noticing that the nostrums pushed so vigorously in 1986 and 1987 have not had the dramatic effects claimed.”

Brooks re-concluded that there had been no technological breakthroughs of the type postulated in NSB.

But motivation can also have silver bullet-like effects and always has had, he found via more research into scientific evidence that motivation can boost productivity. In his own words:

“Since “NSB,” Bruce Blum has drawn my attention to the 1959 work of Herzberg, Mausner, and Sayderman.

They find that motivational factors can increase productivity. On the other hand, environmental and accidental factors, no matter how positive, cannot; but these factors can decrease productivity when negative. “NSB” argues that much software progress has been the removal of such negative factors: stunningly awkward machine languages, batch processing with long turnaround times, poor tools, and severe memory constraints.”

Today, it’s a long time since the mid-nineties; with the benefit of hindsight, were there any silver bullets flying between then and 2022, which fit the bill as slayers of unproductiveness? I suggest a few, below. If you can name other silver bullets since the launch of Windows 95, please do so in the comments!

• Version control: (late 1990s.) CVS, Subversion, and later, Git. Version control allowed engineers to collaborate much more fluently, leading to more teamwork and – in some cases – less full-on solo labor.

• IDEs: (early 2000s). Modern IDEs like Visual Studio, IntelliJS, and others make context-rich editing easy and fast. They also allow for faster, less error-prone refactoring and more efficient debugging.

• CI/CD and automated testing: (mid-2000s). CI systems started to spread during the 2000s with the likes of CruiseControl (2000s) → Jenkins and SaaS CI solutions from the 2010s (e.g., Travis CI, CircleCI, GitLab CI, GitHub Actions).

• Open source and package managers: (2010s). Open source has been around for decades, but GitHub’s rapid adoption made it easier to create and discover, coupled with package managers in the Node, Python, and other language ecosystems to build on top of open source solutions.

• StackOverflow: (2010s). The popular programming Q&A site made it easier to get unstuck by finding solutions to common problems, with the capability to ask questions and get responses from the large user community within hours. By 2025, the site was pretty much dead.

• Cloud: (early 2010s). AWS launched in 2006, then Azure and Google Cloud in 2008, and they went mainstream in the 2010s. Today, cloud is everywhere, and increasingly more infrastructure startups build on top of hyperscalers (the biggest public cloud platforms), such as Vercel / Netlify (platform-as-a-service), Supabase / Turbopuffer (databases) and others.

• Platform teams & DevEx teams: (mid-2010s). Mid-sized and larger tech companies created dedicated platform teams to own infrastructure and internal platforms. In the 2020s, larger tech companies have created ‘Developer Experience’ teams to build better internal dev tools and workflows. We previously covered Uber’s program/platform split, and Uber’s developer experience evolution.

• SRE: the Site Reliability Engineer (SRE) profession grew during the 2010s, much influenced by Google.

These technologies increased developer efficiency and productivity, but none by itself was a productivity accelerator in isolation.

Obviously, by 2022 the craft of building software had developed greatly since ‘No Silver Bullets’ came out; and was more efficient, faster, and more collaborative than ever. One highly anecdotal way to identify this is via the disappearance of cake from some tech workplaces. Back in the day, cake was distributed at work for major product milestones being hit: the shipping of a new product was often marked with awards and tasty baked treats – at least on teams building browsers, like the IE and Firefox teams.

But by the 2010s, shipping frequency had increased by so much and was an everyday, unremarkable occurrence at some places, according to Matt Brubeck, a former engineer on the Firefox team:

“Back when Firefox 2 was released (six years ago this week!), the Internet Explorer team started a friendly tradition of sending Mozilla a cake as congratulations. This continued for Firefox 3 and Firefox 4. After Firefox switched from major releases once or twice a year to incremental updates every six weeks, they sent us a cupcake for the next few updates instead. :)” Mozilla engineer, Matt Brubeck

Today, Firefox ships a stable version about once a month, as does Chrome. In this context, marking each release with more cake could inadvertently cause some health issues on the team – too much cake, that is! From this September, Chrome will switch to shipping every two weeks.

Agile and Scrum is worth a mention; not as a technology, but a methodology: Scrum encourages teams to move in smaller cycles and deliver more frequently, via sprints that typically range from a week to a month. In the early 2000s, this methodology spread quickly and brought efficiency improvements to many tech companies. However, by the early 2020s, many startups and some of Big Tech had moved on, as covered in How Big Tech runs tech projects and the curious absence of Scrum:

“Scrum got in the way of shipping on a daily basis. The whole idea of Scrum revolves around Sprints, of committing to tasks at the beginning of the sprint, working on these during the sprint, and demoing what we did at the end.

The process felt unnatural and like it had been forced on a fast-moving web team. We soon moved to a more fluid way of working, taking the Kanban approach. We stopped caring about sprints, and dropped most rituals that come with Scrum. We just cared about knowing what we’re working on now, and what it was we’d get done next.”

Basically, Scrum worked and still does so for teams wanting to shorten shipping cadence from months to weeks. But for teams shipping daily, it often gets in the way.

One area that improved significantly has been the pace of shipping incremental software. In 1975, shipping software several times per day with elements like version control, CI/CD, feature flags and engineers being oncall, might have sounded far-fetched. Back then, software delivery was measured in months and years. In this way, we’ve perhaps made improvements overall in the regions of 10x to 100x over the years.

But that came via combinations of new tools like version control and CI/CD, new approaches & methodologies, and testing – and also from shifting constraints; for example, it’s now possible to revert backend changes rapidly, and code shipped in binaries can be controlled by feature flags in many cases.

Even so, improvements were mainly in iteration speed and not necessarily in the complexity of the software shipped. With all that progress, shipping complex and high-quality software still takes comparable time, often years, as 50 years ago. A prime example is the upcoming video game, Grand Theft Auto VI, probably by now the most highly-anticipated game ever, which is set to launch in November, after at least six years – and potentially 12 – of total development time:

• Initial planning started in 2014 (12 years of development)

• Development started in earnest in 2020 (circa six years of full-on development)

• The studio, Rockstar, confirmed development was underway in February 2022 (at least 4.5 years of full development)

The video game development timeline is as long as it ever was, and even longer, as developers strive to meet players’ expectations on things like graphics, lighting, and physics. GTA 6 looks like being the most complex installment in the long-running series. So, perhaps there’s not been much change in software delivery timelines because when we have more capabilities to work with, the goals get more ambitious and the bar for “standout” software keeps rising.

2. Is SRE a silver bullet?

Brooks’s definition of a silver bullet:

“A single development, in either technology or management technique, which by itself promises even one order of magnitude improvement in productivity, in reliability, in simplicity.”

In simplicity and productivity terms, I struggle to name a single approach that delivered a 10x-or-more improvement by itself. But in the area of reliability, one company that has pioneered novel approaches since the 2000s is Google. Google.com is probably the single most reliable piece of internet software of all. In the last 15 years, Google Search has suffered a single outage, on 8 August 2022, which lasted around an hour. Otherwise, there have been no global outages (of course, there have been several for other Google services).

In 2003, Google created the ‘Site Reliability Engineering’ (SRE) role. SRE veteran, Dave O’Connor, shared with us:

“The fervent belief of Google’s founders was that speed and reliability mattered more than features. This belief was coupled with the understanding that we couldn’t achieve it traditionally, which made it an existential issue. The level of investment in building out all layers of the serving stack was a case of “because we can”, but also “because we have to, as nowhere else does what we need”.

There was never a question of whether traditional ‘ops’ would work at Google. We needed a specialized role, staffed by folks familiar with the problem space and engineering methods required to make it work.

In 2003, the SRE role was born. Ben Treynor Sloss had been tasked with building Google’s “production team” and in his own words, he built “what happens when you ask a software engineer to design an operations team.” This turned into the birth of the SRE function at Google. From the outset, SRE was staffed in varying measures by systems/operations experts and software engineers. A large part of the remit of the team was to build the tools and practices required to operate Google’s fleet.”

Over time, the rest of the industry caught on to SRE and DevOps. From our SRE deepdive:

“Eventually, other companies caught onto the scaling issues, especially the hyperscalers. Each had their own approach, but over time, the notion grew industry-wide that making things reliable was a real-life engineering discipline, not simply ‘ops’.

This step saw a number of terms coined to describe this engineering, including ‘DevOps’. At its core, this was the notion that the disciplines and practices of reliability engineering should be ingrained into the overall engineering organization. At places other than Google, this mostly took the form of combined developer/operations roles (i.e. “you build it, you run it”), which differed from Google’s implementation, but the practices were similar.

Around this time, Google started opening up about SRE, eventually publishing the first SRE book, and follow ups. Conferences such as USENIX SRECon, Devops Days, and other movements have solidified reliability engineering as a discipline that scales well beyond Google. Indeed, the company has become a consumer of many state-of-the-art developments.”

So, at Google Search, the SRE role could be described as a genuine silver bullet for the tech giant. The company’s obsession with reliability helped it build what is probably the most reliable public-facing service of all. On the assumption that SRE plays a significant role in the approach, I would feel comfortable with calling SRE a silver bullet for Google Search.

SRE, as a concept, is commonplace across Google, but the reliability of its other services is not so impressive. For example, Google Cloud has had many outages, and Gmail also goes down every now and then. I’m sure that without SRE, reliability would be worse, but in general, Google services’ availability these days is probably a magnitude higher than the availability of most online services in the 2000s.

Similarly, GitHub has an SRE role but the service is at zero nines of availability, partially explained by a 3.5x increase in load in two years. But in other ways, the zero nines is likely self-inflicted.

This makes me wonder if the existence of silver bullets depends greatly on teams and individual contexts. SRE seems like a good case to consider:

• a “silver bullet” for Google Search

• … but not for other Google services

• … and definitely not for the broader industry

Could it be that when implemented in the right place, in the right way, and with the correct investment, then SRE – and an incredible focus on reliability – will yield a 10x-or-higher increase in reliability?

My hunch is that Google Search has such standout reliability not just because of SRE, but because Search might be the only organization in Google with reliability as a founding value, embedded in the team’s culture, with unmatched investments of time and money.

Google has published several books that explain their techniques and practices, but for other teams to get those results, they would need to invest similarly in reliability.

3. Was open source + GitHub a silent silver bullet?

Perhaps there’s a silver bullet which is easily missed: open source. In the first-ever Pragmatic Engineer Podcast episode, I asked software engineer Simon Willison what the biggest “productivity leaps” have been during his career. He named open source:

Read more

Today, we cover:

1. Did Anthropic turn hostile on devs because capacity was running low? For the past few weeks, Anthropic has continually upset devs with its “dumber” model, and by removing Claude Code access from some paid accounts. After securing lots of compute from SpaceX, could the reason have been to conceal capacity issues?

2. Amazon finally allows Claude Code and Codex usage. The online retail giant wanted to improve its own coding agent, Kiro, so banned other AI coding tools. But that ban is now lifted.

3. Meta forcefully assigns engineers to data labelling ahead of job cuts. In several teams, 20-40% of engineers are given menial, data labelling work. Could that actually boost people’s job security – for now?

4. New trend: small “AI-forward” teams. Meta and Amazon’s CEOs say teams of 5-10 devs do better work than teams of 50. There are important caveats: it’s unclear what they’ll do with the “excess,” and if it’s limited to “mechanical” work like rewrites.

5. Industry Pulse. Why Meta tracks employees’ computer activity, OpenAI starts to move off Datadog, Apple lets slip it uses Claude Code, GitHub → Xbox transfers at Microsoft, VS Code inserted “coathored by Copilot” even when Copilot did nothing, analysis of the Coinbase layoffs.

Before we start, last week I covered big pricing changes coming from GitHub, but it seems I underplayed how big they will be. Reader Julien has helpfully clarified the actual impact (thank you!)

• The multipliers will be increased for Pro and Pro+ plans on annual renewal (roughly a ~3x increase on average)

• With other plans (Pro, Pro+, Business, Enterprise) it’s more drastic; they will adopt API token-based pricing and drop request-based pricing.

This is indeed a massive change; GitHub has heavily subsidized its usage, relative to API billing, and this change will make pricing unpredictable as of 1 June – at least initially. As Julien put it:

“We are basically waiting to see how much we will be able to use our actual subscription with the new pricing 😇”

1. Did Anthropic turn hostile on devs because capacity was running low?

Read more

Nine years later, he felt the time was ripe for an updated edition, with cloud computing much more widespread than in 2016. So, Martin teamed up with software engineer and investor, Chris Riccomini, a former colleague at LinkedIn and the author of The Missing README, for a full refresh of the book which brings it right up to date for the present day.

Martin was recently on The Pragmatic Engineer Podcast, where we discussed this updated volume and many related cloud computing matters. We also looked into some topics that have become less relevant over time, like details on MapReduce.

I asked Martin if this newsletter could share an excerpt of the updated edition of the book about a timeless, important topic, and he generously agreed. So, today we cover:

1. Cloud versus self-hosting tradeoffs

2. Doing the right thing as a software engineer

These excerpts are only part of the book; the first edition has been on my shelf for years and is now in well-worn condition. I jumped at the chance to get the second edition, and if you’re interested in building resilient systems, I recommend it as an excellent resource.

Get the second edition of the book

My usual disclaimer: as with all my recommendations, I was not paid for this article, and none of the links are affiliates. See my ethics statement for more.

The excerpt below is from “Designing Data-Intensive Applications” second edition, by Martin Kleppmann and Chris Riccomini. Copyright © 2026 Martin Kleppmann, Chris Riccomini. Published by O’Reilly Media, Inc. Used with permission.

1. Cloud versus self-hosting tradeoffs

This excerpt is from Chapter 1: “Trade-Offs in Data Systems Architecture”

For anything that an organization needs to do, one of the first questions is whether it should be done in-house or outsourced. That is, should you build or should you buy?

Ultimately, this is a question about business priorities. A common rule of thumb is that things that are a core competency or a competitive advantage of your organization should be done in-house, whereas things that are non-core, routine, or commonplace should be left to a vendor [20]. To give an extreme example, most companies do not fabricate their own CPUs, since it is cheaper to buy them from the semiconductor manufacturers.

With software, two important decisions to be made are who builds the software and who deploys it. The spectrum of possibilities is illustrated in Figure 1-2. At one extreme is bespoke software that you write and run in-house; at the other extreme are widely-used cloud services or SaaS products that are implemented and operated by an external vendor and that you access only through a web interface or API.

The middle ground is off-the-shelf software (open source or commercial) that you self-host, or deploy yourself – for example, if you download MySQL and install it on a server you control. This could be on your own hardware (often called ‘on-premises,’ even if the server is in a rented datacenter rack and not literally on your own premises), or on a virtual machine (VM) in the cloud (infrastructure as a service, or IaaS). There are more points along this spectrum, such as taking open source software and running a modified version of it.

A related question is how you deploy services, either in the cloud or on premises – for example, whether you use an orchestration framework such as Kubernetes. However, choice of deployment tooling is beyond the scope of this book, since other factors have a greater influence on the architecture of data systems.

Pros & Cons of Cloud Services

Using a cloud service, rather than running comparable software yourself, essentially outsources the operation of that software to the cloud provider. There are good arguments for and against this approach. Cloud providers claim that using their services saves time and money and allows you to move faster compared to setting up your own infrastructure.

Whether using a cloud service is actually cheaper and easier than self-hosting depends very much on your skills and the workload on your systems, however. If you already have experience of setting up and operating the systems you need, and if your load is quite predictable (i.e., the number of machines you need does not fluctuate wildly), then it’s often cheaper to buy your own machines and run the software on them yourself [21, 22].

On the other hand, if you need a system that you don’t already know how to deploy and operate, adopting a cloud service is often easier and quicker than learning to manage the system. Hiring and training staff specifically to maintain and operate the system can get very expensive. You still need an operations team when you’re using the cloud, but outsourcing the basic system administration can free up your team to focus on higher-level concerns.

Outsourcing the operation of a system to a company that specializes in running it can potentially result in better service, since the provider gains operational expertise from providing the service to many customers. On the other hand, if you run the service, you can configure and tune it to perform well on your particular workload. A cloud service would likely be unwilling to make such customizations on your behalf.

Cloud services are particularly valuable if the load on your systems varies a lot over time. If you provision your machines to be able to handle peak load, but those computing resources are idle most of the time, the system becomes less cost-effective. In this situation, cloud services have the advantage that they can make it easier to scale your computing resources up or down in response to changes in demand.

For example, analytical systems often have extremely variable load. Running a large analytical query quickly requires a lot of computing resources in parallel, but once the query completes, those resources sit idle until a user makes the next query. Predefined queries (e.g., for daily reports) can be enqueued and scheduled to smooth out the load, but for interactive queries, the faster you want them to complete, the more variable the workload becomes. If your dataset is so large that querying it quickly requires significant computing resources, using the cloud can save money as you can return unused resources to the provider rather than leaving them idle. For smaller datasets, this difference is less significant.

The biggest downside of a cloud service is that you have no control over it:

• If it is lacking a feature you need, all you can do is politely ask the vendor whether they will add it; you generally cannot implement it yourself.

• If the service goes down, all you can do is wait for it to recover.

• If you are using the service in a way that triggers a bug or causes performance problems, diagnosing the issue will be difficult. With software that you run yourself, you can get performance metrics and debugging information from the operating system to help you understand its behavior, and you can look at the server logs. With a service hosted by a vendor, you usually do not have access to these internals.

• If the service shuts down or becomes unacceptably expensive, or if the vendor changes their product in a way you don’t like, you are at their mercy; continuing to run an old version of the software is usually not an option, so you’ll be forced to migrate to an alternative service [23]. This risk is mitigated if alternative services expose a compatible API, but for many cloud services there are no standard APIs, which raises the cost of switching, making vendor lock-in a problem.

• If the cloud provider is in another country and a political conflict arises between that country and your own, you risk being locked out of the service due to imposed sanctions.

• The cloud provider needs to be trusted to keep the data secure, which can complicate the process of complying with privacy and security regulations.

Despite all these risks, it has become more and more popular for organizations to build new applications on top of cloud services, or to adopt a hybrid approach in which cloud services are used for some aspects of a system. However, cloud services will not subsume all in-house data systems. Many older systems predate the cloud, and for any services that have specialist requirements that existing cloud services cannot meet, in-house systems remain necessary. For example, very latency-sensitive applications such as high-frequency trading require full control of the hardware.

Cloud-Native System Architecture

Besides having a different economic model (subscribing to a service instead of buying hardware and licensing software to run on it), the rise of the cloud has also had a profound effect on how data systems are implemented on a technical level. The term “cloud native” is used to describe an architecture that is designed to take advantage of cloud services.

In principle, almost any software that you can self-host could also be provided as a cloud service, and indeed, such managed services are now available for many popular data systems. However, systems that have been designed from the ground up to be cloud native have been shown to have several advantages: better performance on the same hardware, faster recovery from failures, being able to quickly scale computing resources to match the load, and supporting larger datasets [24, 25, 26]. Table 1-2 lists some examples of both types of systems.

Layering of cloud services

Many self-hosted data systems have simple system requirements; they run on a conventional operating system such as Linux or Windows, they store their data as files on the filesystem, and they communicate via standard network protocols such as TCP/IP. A few systems depend on special hardware such as GPUs (for ML) or remote direct memory access (RDMA) network interfaces, but on the whole, self-hosted software tends to use generic computing resources: CPUs, RAM, a filesystem, and an IP network.

In a cloud, this type of software can be run in an IaaS environment, using one or more VMs (or instances) with a certain allocation of CPUs, memory, disk, and network bandwidth. Compared to physical machines, cloud instances can be provisioned faster and come in a greater variety of sizes, but otherwise they are similar to traditional computers: you can run any software you like on them, but you are responsible for administering it yourself.

In contrast, the key idea of cloud-native services is not only to use the computing resources managed by your operating system, but also to build upon lower-level cloud services to create higher-level services. For example:

• Object storage services such as Amazon S3, Azure Blob Storage, and Cloudflare R2 store large files. They provide more limited APIs than a typical filesystem (basic file reads and writes), but they have the advantage that they hide the underlying physical machines; the service automatically distributes the data across many machines so that you don’t have to worry about running out of disk space on any one machine. Even if some machines or their disks fail entirely, no data is lost.

• Many other services are, in turn, built upon object storage and other cloud services. For instance, Snowflake is a cloud-based analytical database (data warehouse) that relies on S3 for data storage [26], and some other services, in turn, build upon Snowflake.

As always with abstractions in computing, there is no one right answer to what you should use. As a general rule, higher-level abstractions tend to be more oriented toward particular use cases. If your needs match the situations for which a higher-level system is designed, using the existing higher-level system will probably meet your needs with much less hassle than building it yourself from lower-level systems would. On the other hand, if no high-level system meets your needs, building it yourself from lower-level components is the only option.

Separation of storage and compute

In traditional computing, disk storage is regarded as durable (we assume that once something is written to disk, it will not be lost). To tolerate the failure of an individual hard disk, RAID (redundant array of independent disks) is often used to maintain copies of the data on several disks attached to the same machine. RAID can be implemented either in hardware or in software by the operating system, and it is transparent to the applications accessing the filesystem.

In the cloud, compute instances (VMs) may also have local disks attached, but cloud-native systems typically treat these disks more like an ephemeral cache and less like long-term storage. This is because the local disk becomes inaccessible if the associated instance fails, or if the instance is replaced with a bigger or a smaller one (on a different physical machine) to adapt to changes in load.

As an alternative to local disks, cloud services also offer virtual disk storage that can be detached from one instance and attached to a different one (e.g., Amazon EBS, Azure managed disks, and persistent disks in Google Cloud). Such a virtual disk is not a physical disk, but rather a cloud service provided by a separate set of machines that emulates the behavior of a disk (a block device, where each block is typically 4 KiB in size). This technology makes it possible to run traditional disk-based software in the cloud, but the block device emulation introduces overheads that can be avoided in systems that are designed from the ground up for the cloud [24]. The use of virtual disks also makes the application very sensitive to network glitches, since every I/O operation on the virtual block device is a network call [27].

To address this problem, cloud-native services generally avoid using virtual disks and instead build on dedicated storage services that are optimized for particular workloads. Object storage services such as S3 are designed for long-term storage of fairly large files, ranging from hundreds of kilobytes to several gigabytes in size. The individual rows or values stored in a database are typically much smaller than this; cloud databases therefore typically manage smaller values in a separate service and store larger data blocks (containing many individual values) in an object store [25, 28].

In traditional systems architecture, the same computer is responsible for both storage (disk) and computation (CPU and RAM), but in cloud-native systems, these two responsibilities have become somewhat separated, or disaggregated [9, 26, 29, 30]: for example, S3 only stores files, and if you want to analyze that data, you will have to run the analysis code somewhere outside of S3. This implies transferring the data over the network.

Furthermore, cloud-native systems are often multitenant, which means that rather than having a separate machine for each customer, data and computation from several customers are handled on the same shared hardware by the same service [31]. Multitenancy can enable better hardware utilization, easier scalability, and easier management by the cloud provider, but it also requires careful engineering to ensure that one customer’s activity does not affect the performance or security of the system for other customers [32].

Operations in the Cloud Era

Traditionally, the people managing an organization’s server-side data infrastructure were known as database administrators (DBAs), or system administrators (sysadmins). More recently, many organizations have tried to integrate the roles of software development and operations into teams with a shared responsibility for both backend services and data infrastructure; the DevOps philosophy has guided this trend. Site reliability engineers (SREs) are Google’s implementation of this idea [33].

The role of operations is to ensure that services are reliably delivered to users (including configuring infrastructure and deploying applications) and to ensure a stable production environment (including monitoring and diagnosing any problems that may affect reliability). For self-hosted systems, operations traditionally involve a significant amount of work at the level of individual machines, such as capacity planning (e.g., monitoring available disk space and adding more disks before you run out of space), provisioning new machines, moving services from one machine to another, and installing operating system patches.

Many cloud services present an API that hides the individual machines implementing the service. For example, cloud storage replaces fixed-size disks with metered billing, where you can store data without planning your capacity needs in advance, and you are then charged based on the space used. Moreover, many cloud services remain highly available, even when individual machines have failed.

This shift in emphasis from individual machines to services has been accompanied by a change in the role of operations. The high-level goal of providing a reliable service remains the same, but the processes and tools have evolved.

The DevOps/SRE philosophy places greater emphasis on the following:

• Setting up automation; preferring repeatable processes over manual one-off jobs

• Using ephemeral VMs and services rather than long-running servers

• Enabling frequent application updates

• Learning from incidents

• Preserving the organization’s knowledge about the system, even as individuals come and go [34]

With the rise of cloud services, a bifurcation of roles has occurred. Operations teams at infrastructure companies specialize in the details of providing a reliable service to a large number of customers, while the customers of the service spend as little time and effort as possible on infrastructure [35].

Customers of cloud services still require operations, but they focus on different aspects, such as choosing the most appropriate service for a given task, integrating services with each other, and migrating from one service to another. Even though metered billing removes the need for capacity planning in the traditional sense, it’s still important to know what resources you are using for which purpose so that you don’t waste money on cloud resources that are not needed. Capacity planning becomes financial planning, and performance optimization becomes cost optimization [36]. Additionally, cloud services do have resource limits or quotas (such as the maximum number of processes you can run concurrently), which you need to know about and plan for before you run into them [37].

Adopting a cloud service can be easier and quicker than provisioning and running your own infrastructure, although you still have to learn how to use the cloud service and perhaps work around its limitations. Integration among services becomes a particular challenge as a growing number of vendors offer an ever-broader range of cloud services targeting different use cases [38, 39]. ETL is only part of the story; operational cloud services also need to be integrated with each other. At present, we lack standards to facilitate this sort of integration, so it often involves significant manual effort.

Other operational aspects that cannot fully be outsourced to cloud services include maintaining the security of an application and the libraries it uses, managing the interactions between your own services, monitoring the load on your services, and tracking down the cause of problems such as performance degradations or outages. While the cloud is changing the role of operations, the need for operations is as great as ever.

2. Doing the right thing as a software engineer

The excerpt below is a section from Chapter 14, “Doing the Right Thing”

In the final chapter of this book, let’s take a step back. Throughout, we have examined a wide range of architectures for data systems, evaluated their pros and cons, and explored techniques for building reliable, scalable, and maintainable applications. However, we have left out a fundamental part of the discussion, which we should now fill in.

Every system is built for a purpose; every action we take has both intended and unintended consequences. The purpose may be as simple as making money, but the consequences may be far-reaching. We, the engineers building these systems, have a responsibility to carefully consider those consequences and to ensure that our decisions do not cause harm.

We talk about data as an abstract thing, but remember that many datasets are about people: their behavior, their interests, their identities. We must treat such data with humanity and respect. Users are humans too, and human dignity is paramount [1].

Software development increasingly involves making important ethical choices. There are guidelines to help software engineers navigate these issues, such as the ACM Code of Ethics and Professional Conduct [2], but they are rarely discussed, applied, or enforced in practice. As a result, engineers and product managers sometimes take a cavalier attitude to privacy and the potential negative consequences of their products [3, 4].

A technology is not good or bad in itself – what matters is how it is used and how it affects people. This is true of a software system such as a search engine in much the same way as it is for a weapon like a gun. The ethical responsibility is ours to bear; it is not sufficient for software engineers to focus exclusively on the technology and ignore its consequences.

In contrast to much of computing, however, the concepts at the heart of ethics are not fixed or determinate in their precise meaning; they require interpretation, which may be subjective [5]. What makes something “good” or “bad” is not well defined, and serious discourse on the subject among computing professionals is lacking [6]. Reasoning about ethics is difficult, but also too important to ignore. What does this entail? “Ethics” are not a checklist with which to comply; it’s a participatory and iterative process of reflection, in dialogue with people involved and accountability for the results [7].

Predictive Analytics

Predictive analytics is a major part of why people are excited about big data and AI. It’s also an area that is fraught with ethical dilemmas. Using data analysis to predict the weather, or the spread of diseases, is one thing [8]; it is another matter to predict whether a convict is likely to reoffend, whether an applicant for a loan is likely to default, or whether an insurance customer is likely to make expensive claims [9]. The latter have a direct effect on people’s lives.

Naturally, payment networks want to prevent fraudulent transactions, banks want to avoid bad loans, airlines want to avoid hijackings, and companies want to avoid hiring ineffective or untrustworthy people. From their point of view, the cost of a missed business opportunity is low, but the cost of a bad loan or a problematic employee is much higher, so it is expected for organizations to be cautious. If in doubt, they are better off saying “no”.

However, as algorithmic decision making becomes more widespread, someone who has (accurately or falsely) been labeled as risky by an algorithm may suffer a large number of “no” decisions. Systematically being excluded from jobs, air travel, insurance coverage, property rental, financial services, and other key aspects of society is such a large constraint on an individual’s freedom that it has been called “algorithmic prison” [10]. In countries that respect human rights, the criminal justice system presumes innocence until proven guilty; on the other hand, automated systems can systematically and arbitrarily exclude a person from participating in society without any proof of guilt and little chance of appeal.

Bias & discrimination

Decisions made by an algorithm are not necessarily any better or any worse than those made by a human. Everyone is likely to have biases, even if they actively try to counteract them, and discriminatory practices can become culturally institutionalized. There is hope that basing decisions on data, rather than subjective and instinctive human assessments, could be more fair and give a better chance to people who are often overlooked or disadvantaged in the traditional system [11].

When we develop predictive analytics and AI systems, we are not merely automating a human’s decision by using software to specify the rules for when to say “yes” or “no”; we are leaving the rules themselves to be inferred from data. However, the patterns learned by these systems are opaque: even if the data indicates a correlation, we may not know why. If the input to an algorithm carries a systematic bias, the system will most likely learn and amplify that bias in its output [12].

In many countries, anti-discrimination laws prohibit treating people differently depending on protected traits such as ethnicity, age, gender, sexuality, disability, or beliefs. Other features of a person’s data may be analyzed, but what happens if they are correlated with protected traits? For example, in racially segregated neighborhoods, a person’s postal code or even their IP address is a strong predictor of race. Put like this, it seems ridiculous to believe that an algorithm could somehow take biased data as input and produce fair and impartial output from it [13, 14]. Yet this belief often seems to be implied by proponents of data-driven decision making; an attitude that has been satirized as “machine learning is like money laundering for bias” [15].

Predictive analytics systems merely extrapolate from the past; if the past is discriminatory, they codify and amplify that discrimination [16]. If we want the future to be better than the past, moral imagination is required, and that’s something only humans can provide [17]. Data and models should be our tools, not our masters.

Responsibility and Accountability

Automated decision-making raises the question of responsibility and accountability [17]. If a human makes a mistake, they can be held accountable, and the person affected by the decision can appeal. Algorithms make mistakes too, but who is accountable when they go wrong? [18] When a self-driving car causes an accident, who is responsible? If an automated credit scoring algorithm systematically discriminates against people of a particular race or religion, is there any recourse? If a decision by your ML system comes under judicial review, can you explain to the judge how the algorithm made its decision? People should not be able to evade responsibility by blaming an algorithm.

Credit rating agencies are a classic example of collecting data to make decisions about people. A bad credit score makes life difficult, but at least a credit score is normally based on relevant facts about a person’s actual borrowing history, and any errors in the record can be corrected (although the agencies normally do not make this easy). Scoring algorithms based on machine learning, however, typically use a much wider range of inputs and are much more opaque, making it harder to understand how a particular decision has come about and whether someone is being treated in an unfair or discriminatory way [19].

A credit score summarizes “how did you behave in the past?” whereas predictive analytics usually work on the basis of “who is similar to you, and how did people like you behave in the past?” Drawing parallels to others’ behavior implies stereotyping people; for example, based on where they live (a close proxy for race and socioeconomic class). What about people put in the wrong bucket? Furthermore, if a decision is incorrect because of erroneous data, recourse is almost impossible [17].

Much data is statistical in nature, which means that even if the probability distribution on the whole is correct, individual cases may well be wrong. For example, if the average life expectancy in your country is 80 years, that doesn’t mean you’re expected to drop dead on your 80th birthday. From the average and the probability distribution, you can’t say much about the age to which someone will live. Similarly, the output of a prediction system is probabilistic and may well be wrong in individual cases.

A blind belief in the supremacy of data for making decisions is not only delusional, but also positively dangerous. As data-driven decision making becomes more widespread, we will need to figure out how to avoid reinforcing existing biases, how to make algorithms accountable and transparent, and how to fix them when they inevitably make mistakes.

We will also need to figure out how to realize the positive potential of data and prevent it from being used to harm people. For example, analytics can reveal financial and social characteristics about personal lives. On the one hand, this power could be used to focus aid and support to help those who need it most. On the other hand, it is sometimes used by predatory businesses seeking to identify vulnerable people and sell them risky products such as high-cost loans or worthless college degrees [17, 20].

Feedback loops

Even with predictive applications with less immediately far-reaching effects on people, such as recommendation systems, there are difficult issues that we must confront. When services become good at predicting the content users want to see, they may end up showing them only opinions they already agree with, leading to echo chambers in which stereotypes, misinformation, and polarization can breed. We already know the impact that social media echo chambers can have on election campaigns.

When predictive analytics affect people’s lives, particularly pernicious problems arise because of self-reinforcing feedback loops. For example, consider the case of employers using credit scores to evaluate potential hires. You may be a good worker with a good credit score, but suddenly find yourself in financial difficulties due to a misfortune beyond your control. As you miss payments on your bills, your credit score suffers, and you will be less likely to find work. Joblessness pushes you toward poverty, which further worsens your score, making it even harder to find employment [17]. It’s a downward spiral due to poisonous assumptions, hidden behind a camouflage of mathematical rigor and data.

As another example of a feedback loop, economists found that when gas stations in Germany introduced algorithmic prices, competition was reduced and prices for consumers went up because the algorithms learned to collude [21].

We can’t always predict when such feedback loops may happen. However, many consequences can be predicted by thinking about an entire system (not just the computerized parts, but also the people interacting with it), in an approach known as “systems thinking” [22]. We can try to understand how a data analysis system responds to different behaviors, structures, or characteristics. Does the system reinforce and amplify existing differences between people (e.g., making the rich richer or the poor poorer), or does it try to combat injustice? Even with the best intentions, we must beware of the possibility of unintended consequences.

Surveillance

The excerpt below is from another section in Chapter 14, “Doing the Right Thing”

As a thought experiment, try replacing the word “data” with “surveillance”, and observe whether common phrases still sound so good [23]. How about this: “In our surveillance-driven organization we collect real-time surveillance streams and store them in our surveillance warehouse. Our surveillance scientists use advanced analytics and surveillance processing in order to derive new insights.”

This thought experiment is unusually polemical for this book, “Designing Surveillance-Intensive Applications”, but strong words are needed to emphasize this point. In our attempts to make software “eat the world” [24], we have built the greatest mass surveillance infrastructure ever seen. We are rapidly approaching a world in which every inhabited space contains at least one internet-connected microphone, in the form of smartphones, smart TVs, voice-controlled assistant devices, baby monitors, and even children’s toys that use cloud-based speech recognition. Many of these devices have terrible security track records [25].

What is new compared to the past is that digitization has made it easy to collect large amounts of data about people. Surveillance of our location and movements, our social relationships and communications, our purchases and payments, and our health data has become almost unavoidable. A surveillance organization may end up knowing more about a person than that person knows about themselves; for example, identifying illnesses or economic problems before that individual is aware of them.

Even the most totalitarian, repressive regimes of the past could only dream of putting a microphone in every room and forcing every person to constantly carry a device capable of tracking their location and movements. Yet the benefits that we get from digital technology are so great that we now voluntarily accept this state of total surveillance. The difference is just that the data is being collected by corporations to provide us with services, rather than government agencies seeking control [26].

Not all data collection necessarily qualifies as surveillance, but examining it as such can help us understand our relationship with the data collector. Why are we seemingly happy to accept surveillance by corporations? Perhaps you feel you have nothing to hide; in other words, you are totally in line with existing power structures, you are not a marginalized minority, and you needn’t fear persecution [27]. Not everyone is so fortunate. Or perhaps it’s because the purpose seems benign; it’s not overt coercion and conformance, merely better recommendations and more personalized marketing. However, combined with the discussion of predictive analytics from the last section, that distinction seems less clear.

We are already seeing behavioral data about driving, tracked by vehicles without drivers’ consent, affecting their insurance premiums [28], and health insurance coverage that depends on people wearing a fitness tracking device. When surveillance is used to make decisions that hold sway over important aspects of life, such as insurance coverage or employment, it starts to appear less benign. Data analysis can also reveal surprisingly intrusive things; for example, the movement sensor in a smartwatch or fitness tracker can be used to work out what you are typing (e.g., passwords) with fairly good accuracy [29]. Sensor accuracy and algorithms for analysis are only going to get better.

Takeaways

Thanks to Martin for writing this book, and to himself and Chris for doing a revamp for the second edition. The volume is now even more relevant to how we build systems in 2026 and beyond. You can purchase a hard copy from the publisher’s website or Amazon.

The first edition has a timeless quality because it focused on the fundamentals of large systems, and the new second edition follows the same approach, as laid out in its preface:

“Although the landscape of technologies for processing and storing data is diverse and fast-changing, the underlying principles endure. If you understand those principles, you’re in a position to see where each tool fits in, how to make good use of it, and how to avoid its pitfalls. This book focuses on those principles.”

Since the first edition appeared nine years ago, some things have changed in the tech industry:

• Much greater focus on the cloud. Building large systems on top of cloud infrastructure is more common. This brings lower complexity as cloud primitives hide a lot of implementation complexity, but it also means accepting more risk because when the cloud is down, so is your system.

• Systems which AI tools build upon are more relevant. Vector databases, DataFrames (for training datasets), and the processing of large amounts of training data with batch processing systems are relevant to anyone building production AI systems.

• Local-first software. Martin focuses on this area in his work, and with AI, we could see more demand for running models locally. Operating systems like Ubuntu are also focusing on this.

• Formal methods. The advent of AI-generated code means this topic is getting more attention industry-wide, and the second edition covers it.

• Regulation and legal context. Regulations like the EU’s General Data Protection Regulation (GDPR) are something software engineers increasingly need to know about, and the book now covers it.

If I had to summarize the evolution of the book in its second edition, it would be more focus on cloud and AI, and more on local-first software, testing, and how regulations affect engineers. Interestingly, this mirrors how the tech industry has developed over time, too.

I very much appreciate that the book closes with the final chapter focused on “doing the right thing” as a software engineer. Software systems have wide-ranging societal impact, and engineers working on these systems have a great say in what gets built, and how it gets built. As engineers, we owe it the very least to ourselves to consider the broader impact of our decisions — and doing so might also force us to make important ethical choices. There’s less discussion of the ethics angle on software engineering: and I’m glad that Martin and Chris did not shy away from going deeper into this topic.

If you’d like to get more background on the book – and on the hard parts of building large-scale systems – check out our podcast episode with Martin Kleppmann.

Today, we cover:

1. Load from AI breaks GitHub – but why not other vendors? GitHub’s reliability is less than one nine, and getting worse. Prolific open source contributor, Mitchell Hashimoto, is quitting GitHub because he thinks it’s not suited for professional work. GitHub’s leadership blames the 3.5x increase in service load as the cause of degradation – or it might be self-inflicted.

2. Anthropic’s speedrun to destroy trust. Anthropic could do no wrong until recently, but in the past month, that’s all changed. Silently nerfing Claude Code, banning companies from Claude, and baffling price rises all add to a sense that Anthropic is in its “extraction” era of generating more revenue for the same or worse service.

3. Industry pulse. Dramatic price increases at GitHub Copilot, explosive growth at Codex, Google scrambling to build a good coding model, Cursor might be bought by SpaceX, AI agent deletes car business, and more.

4. Mitchell Hashimoto & the “building block economy.” Ghostty’s creator finds that open source “building blocks” are the best way to win massive adoption by software components – but it’s got harder to build a business on top of open building blocks.

The bottom of this article could be cut off in some email clients. Read the full article uninterrupted, online.

1. Load from AI breaks GitHub – but why not other vendors?

GitHub’s reliability has been beyond unacceptable recently: last month, third party measurements pinned it at one nine (right at 90%). This month, reliability has been down to zero nines – 86% – as per a third-party tracker, and last week, things got even worse: a frankly embarrassing data integrity incident, more outages, and a partial explanation from GitHub, eventually.

Data integrity incident

Read more

Listen and watch now on YouTube, Spotify, and Apple. See the episode transcript at the top of this page, and timestamps for the episode at the bottom.

Brought to You by

• Statsig – ⁠ The unified platform for flags, analytics, experiments, and more. Stop switching between different tools, and have them all in one place.

• Sonar — The makers of SonarQube, the industry standard for code verification and automated code review. As AI agents generate extreme volumes of code, verification can’t be optional: SonarQube acts as the independent, zero‑trust, multi-layered verification engine that checks every line of code against your quality, security, and architectural standards, so only safe, reliable, and auditable code reaches production. Try it out for yourself.

• WorkOS – Designing large systems is about tradeoffs. But one thing isn’t a tradeoff: enterprise features. WorkOS gives you APIs to ship enterprise features – SSO, directory sync, RBAC, audit logs – in days, not months. Visit WorkOS.com to learn more.

In this episode

Mario Zechner is the creator of Pi, a minimalist, self-modifying AI coding agent, that is the foundation upon which OpenClaw (created by Peter Steinberger) is built. Meanwhile, Armin Ronacher is the creator of Flask, and a longtime user of Pi. The pair are also friends.

I sat down with Mario and Armin for the latest episode of the Pragmatic Engineer Podcast for an interesting conversation about AI and their reservations about it – even though both are heavily invested in building AI-powered tools.

Mario explains why he built Pi, and gives his take on why it has become so popular. Armin walks us through how he uses AI tools, including building a game with Pi, and why he always puts human judgment firmly at the heart of his approach.

We cover the risks of over-automation, the limits of agentic workflows, and why strong engineers with informed judgment still matter. We also get into the challenges of working with code written by non-engineers, and whether open source can withstand a tidal wave of agent-generated code.

My observations from the conversation with Mario and Armin

Here are 9 of my most interesting takeaways from talking with Armin and Mario:

1. Pi was built because Claude Code became unpredictable. Mario was a big fan of Claude Code at first. But as the team behind it pushed velocity and added features, he found that bugs multiplied and the tool’s behavior started to change. Mario wanted an AI harness that behaves in a stable, consistent way. He observed that the addition of new features caused Claude Code to act unpredictably, so resolved to add as few features as possible to Pi.

2. It should be MUCH easier to build specialized tools for specific tasks. Different projects need different harness types because, as Mario points out, the same hammer is not ideal for every single construction job. As such, Pi is built with the goal of allowing the creation of specialized harnesses. It can modify itself so that a user can create the bespoke harness needed for any task. Mario believes it’s a preview of how self-modifiable software might look in the future.

3. Automation bias is one of the biggest risks of working with AI agents. Once devs confirm that an AI agent can produce acceptable code, they start to review its output less often, even though agents can – and do! – produce slop. Mario advises being far more sceptical with agents, and cautions that the quality of their output isn’t guaranteed, however well they performed previously.

4. AI agents decrease code quality, but this is not on purpose. From talking with 30+ engineering teams, Armin found that code quality is down everywhere, and serious projects are shipping with “vibe slop.” A potential cause of this is that keeping agentic output clean and of high quality takes deliberate effort, but it’s not clear to many devs exactly how to do this. There’s also PR review fatigue and automation bias (the assumption that AI agents invariably generate good code).

5. New trend: AI makes it harder for senior engineers to reject pointless complexity. Historically, senior engineers kept software complexity at bay simply by saying “no” a lot. But Armin observes that these days, more junior engineers and product managers deploy agent-scripted counterarguments when a senior colleague kicks an idea to the curb. This makes decision-making exhausting, and more bad ideas make it into production as a result.

6. Junior engineers > AI agents. Mario points out that, unlike humans, agents don’t retain lessons in the same way, nor feel the pain of bad code. Junior engineers do, and the pain of maintenance teaches them to simplify interfaces and avoid bad abstractions – which are both qualities of an effective senior engineer. In this way, a junior engineer is more valuable than an AI agent!

7. Agents refactor less because they feel no “pain.” Humans rewrite bad interfaces because maintaining them hurts, whereas agents will obliviously churn out and extend a terrible structure, ad infinitum. This is a big reason why AI agents keep adding more tech debt.

8. Frictionless shipping can actually be harmful. Armin notes that some friction is desirable; for example, multi-reviewer approvals on critical services, SLO gates (different gates based on the service level objective offered), and migration checklists. The good thing about friction is that it makes humans stop and think.

9. Does not being in San Francisco help people stay grounded about AI? I asked Mario how he keeps level-headed about AI while building one of the most popular AI agent harnesses. In response, he credits living in Austria, being a father, and enjoying the great outdoors, as his antidotes to all the hype.

The Pragmatic Engineer deepdives relevant for this episode

• The creator of OpenClaw: “I ship code that I don’t read”

• Building great SDKs

• What is inference engineering? Deepdive

• The impact of AI on software engineers in 2026: key trends

• Cycles of disruption in the tech industry

• The AI engineering stack

Timestamps

(00:00) Intro

(07:30) How Mario, Armin, and Peter Steinberger met

(15:15) How 30 dev teams use AI agents: learnings

(21:50) The importance of judgment

(24:26) Challenges when non-engineers write code

(28:30) Downsides of over-automation

(32:18) Pi

(48:09) OpenClaw + Pi

(50:54) “Clankers”

(57:32) Open source and AI

(1:00:22) Complexity as the enemy

(1:02:50) Building an AI-native startup

(1:11:52) “Slow the F down”

(1:16:40) MCPs vs. CLI

(1:25:03) Predictions and staying up to date

References

Where to find Mario Zechner:

• X: https://x.com/badlogicgames

• LinkedIn: https://www.linkedin.com/in/mariozechner

• Website: https://mariozechner.at

Where to find Armin Ronacher:

• X: https://x.com/mitsuhiko

• LinkedIn: https://www.linkedin.com/in/arminronacher

• Website: https://mitsuhiko.at

• Blog: https://lucumr.pocoo.org

Mentions during the episode:

• Python, Go, Rust, TypeScript and AI with Armin Ronacher: https://newsletter.pragmaticengineer.com/p/python-go-rust-typescript-and-ai

• Pi: https://pi.dev

• OpenClaw: https://openclaw.ai

• Flask: https://flask.palletsprojects.com/en/stable

• The creator of Clawd: “I ship code that I don’t read”: https://newsletter.pragmaticengineer.com/p/the-creator-of-clawd-i-ship-code

• Amiga 500: https://en.wikipedia.org/wiki/Amiga_500

• i486: https://timeline.intel.com/1989/meet-the-i486

• Peter Steinberger on X: https://x.com/steipete

• Sentry: https://sentry.io

• Nat Friedman on X: https://x.com/natfriedman

• Chroma: https://www.trychroma.com

• Siemens: https://www.siemens.com

• Y Combinator: https://www.ycombinator.com

• The Final Bottleneck: https://lucumr.pocoo.org/2026/2/13/the-final-bottleneck

• Children’s Learning With Tablet Technology is Often Too Passive: https://news.utexas.edu/2017/08/22/childrens-learning-with-tablet-technology-is-often-passive

• Amp: https://ampcode.com

• OpenCode: https://opencode.ai

• Agent Design Is Still Hard: https://lucumr.pocoo.org/2025/11/21/agents-are-hard

• How Linux is built with Greg Kroah-Hartman: https://newsletter.pragmaticengineer.com/p/how-linux-is-built-with-greg-kroah

• Mario’s post on X about complexity:

• VibeTunnel: https://vibetunnel.sh

• Thoughts on slowing the F down: https://mariozechner.at/posts/2026-03-25-thoughts-on-slowing-the-fuck-down

• StackOverflow: https://stackoverflow.com

• David Cramer on LinkedIn: https://www.linkedin.com/in/dmcramer

• Stainless: https://www.stainless.com

—

Production and marketing by Pen Name.

Today’s article focuses on Linux and Ubuntu, and we’ll cover Windows in a follow-up issue. Obviously, I reached out to Apple but heard nothing back, unsurprisingly. If you’re reading this and happen to work at Apple, it’d be great to learn more!

Jon Seager is VP of Engineering at Canonical – the company behind Ubuntu – and has provided new details about what the team there has built for AI support, and some new ideas that they’re brewing up. Today, we cover:

1. Hardware enablement: support for GPUs, NPUs and DPUs. When you turn on a machine with AI accelerators, Ubuntu aims for the hardware to perform at its full potential. This means having proper driver support for PCs and cloud data centers’ computing units.

2. Hardware partnerships. Working closely with NVIDIA, AMD, and Intel means Ubuntu can support those vendors’ new hardware from release day.

3. CPU architecture variants. New versions in a CPU family add to, or change, features. An operating system needs to support a new version of the CPU architecture variant in order to fully utilize it. Ubuntu does this for the x86‑64 family, making it a lot more performant on newer CPUs – while still supporting older CPUs.

4. Local-first bet & plans for agentic workflows. There’s a big focus on running local models and using “inference snaps” which help choose the right model with the right quantization. There is the intention to support agentic workflows at the OS level, one day, which is currently at the early exploration stage.

5. Developer ecosystem. There’s a plan to add more support for AI dev tools, a focus on sandboxing at the OS level, a push to support ARM64 laptops more, and we touch on the popularity of Windows Subsystem for Linux (WSL).

6. Engineering culture. A skeptical attitude to AI at Canonical has given way to one where experimentation is encouraged and devs lean into AI tools, but there are no targets for token usage or amounts of AI-generated code.

7. What other Linux distributions are doing. Arch Linux takes the “DIY your AI setup” approach, Omarchy makes it easy to install AI tools, while Red Hat Enterprise Linux ships with AI integrated into the command-line and support for AI accelerators & popular AI tools.

The bottom of this article could be cut off in some email clients. Read the full article uninterrupted, online.

Read the full article online

1. Hardware enablement: support for GPUs, NPUs & DPUs

Jon mentioned he detects a “Dotcom Boom”-era vibe in the industry, like around when “web 1.0” was created, and indeed, lots of startups today aim to be the Google-style success story of this “AI era”. At Canonical, the team asked: what does that mean for Ubuntu as an operating system?

For instance, should Ubuntu join the competition and try to position itself closer to AI, or keep focusing on what they’ve done for decades: build an operating system? Jon said:

“We need to make sure to remain a relatable and accessible system. I don’t think we should blur the line between application features and the OS itself. So, the most powerful thing we can do is hardware enablement.”

Hardware enablement means that if a computer (typically, a laptop) has AI-related hardware, Ubuntu should allow it to make full use of it. This involves adding support for GPUs, NPUs, DPUs and other types of accelerator cards. Let’s briefly go through each.

GPUs

As is likely widely known by readers, ‘GPU’ stands for Graphics Processing Unit. Originally built for graphics rendering, its #1 use case is no longer in video games but for AI training and inference. GPUs come in two forms:

• Integrated GPUs: located on the same die (integrated circuit) as the CPU, like GPUs on Apple’s M-series processors

• Discrete GPUs: separate chips on their own board; often for gaming, or in standalone GPU rigs for AI and ML workloads

NVIDIA leads the market in discrete GPUs for rigs with its Blackwell family, and in standalone GPU cards with the NVIDIA RTX series. Other vendors like AMD offer GPUs for data centers (like the Instinct MI300 Series) and for PCs with the AMD Radeon series.

NPUs

Neural Processing Units (NPUs) are also called “AI accelerators.” This is a dedicated block on the System-on-a-chip (SoC), on modern processors especially designed for running AI inference efficiently on‑device. Since 2022, many modern processors have had a dedicated NPU block, including all Apple’s M-series chips (from M1 and up), Intel’s Core Ultra and Core Ultra “Series 2”, AMD’s Ryzen AI 300 series, and also Qualcomm’s Snapdragon X Elite and Snapdragon X Plus.

A number shared for each NPU is TOPS. TOPS means Tera (trillions) of Operations Per Second, and the said operation is a “multiply-accumulate” (MAC) one, which Qualcomm describes as:

“A multiply-accumulate (MAC) operation executes the mathematical formulas at the core of AI workloads. A matrix multiply consists of a series of two fundamental operations: multiplication and addition to an accumulator. A MAC unit can, for example, run one of each per clock cycle, meaning it executes two operations per clock cycle. A given NPU has a set number of MAC units that can operate at varying levels of precision, depending on the NPU’s architecture.”

How TOPS is calculated: TOPS = 2 × MAC unit count × Frequency / 1 trillion.

“Frequency” refers to the clock speed (cycles per second) at which an NPU and its MAC units (as well as a CPU or GPU) operate, which directly influences overall performance. Processors at higher frequencies allow for more operations, but higher frequencies also mean more energy consumed, heat generated, and battery life decreased. The TOPS number that’s quoted for processors is generally the peak operating frequency.

NPUs are often ideal for low-power, local inference, and for running smaller, local models. They can be useful for things like Local speech‑to‑text (dictation, captions, meeting transcription), video background blur/replacement or auto‑framing, small local language summarization, etc. NPUs are more typical of laptop and PC processors, although some phone processors ship with them like the iPhone (A-series chips) and Google’s Tensor processor in Pixel phones. Basically, NPUs promise to bring efficiently-running local models on laptops one step closer.

DPUs

Data Processing Units (DPUs) are typically found in data centers, moving massive amounts of data fast. NVIDIA’s explanation:

“The CPU is for general-purpose computing, the GPU is for accelerated computing, and the DPU, which moves data around the data center, does data processing.

A DPU is a new class of programmable processor that combines three key elements. A DPU is a system on a chip, or SoC, that combines:

• An industry-standard, high-performance, software-programmable, multi-core CPU, typically based on the widely used Arm architecture, tightly coupled to the other SoC components.

• A high-performance network interface capable of parsing, processing and efficiently transferring data at line rate, or the speed of the rest of the network, to GPUs and CPUs.

• A rich set of flexible and programmable acceleration engines that offload and improve applications’ performance for AI and machine learning, zero-trust security, telecommunications, and storage, among others.”

Several major chipmakers manufacture DPUs, of which NVIDIA’s BlueField family is the most widespread. Others include AMD Pensando DPUs (Elba, Giglio), and Intel IPU / DPU cards (E2100, E2200 series).

DPUs are most commonly deployed inside Hyperscale cloud providers (AWS, Azure, GCP, OCI), or in AI and high-performance computing (HPC) data centers, or larger private clouds. DPUs make sense when GPU traffic is huge, or when the network telemetry overhead is so great that it could overwhelm the CPUs processing the data transfer.

2. Hardware partnerships

It’s easiest to add support to hardware by working with leading chip manufacturers, so Ubuntu has relationships with hardware vendors for that reason. As a result, the OS sometimes offers day-one support for cutting-edge AI supercomputers.

Partnership with NVIDIA

In September 2025, Canonical announced it would package and distribute the full NVIDIA CUDA toolkit directly within Ubuntu’s repositories. This deal collapsed into a single standard apt install, something that had previously been a multi-step manual installation process of downloading from NVIDIA’s site, importing GPG keys, pinning a separate APT repo – and praying nothing broke.

Packaging and distributing the CUDA toolkit makes developing with CUDA easier. From Jon:

“One of the trickiest things for developers who have to use this tech is the dance of matching the right version of Python, with the right version of CUDA, with the right driver. Projects end up with different versions of CUDA, and then machines end up breaking because the driver configuration gets inadvertently broken along the way.

The number one thing we can do as an operating system is to make this setup as easy as possible.”

Ubuntu’s strategy of working directly with chipmakers seems to be working. NVIDIA recently discontinued its custom NVIDIA DGX OS — a modified Ubuntu it maintained for years — and now ships plain Ubuntu. Jon:

“Previously, NVIDIA shipped NVIDIA DGX OS for which NVIDIA had an agreement with Canonical where they could take Ubuntu, modify it with the kernel modules and software they needed, do some product-specific optimization, and ship that as NVIDIA DGX OS.

This more recent development sees NVIDIA just shipping Ubuntu as it comes.

When NVIDIA released the DGX Spark, a $4,000 AI workstation with an ARM64 chipset, it shipped running vanilla Ubuntu as the only supported operating system.”

At CES 2026 in January, Canonical announced Ubuntu support for the NVIDIA Vera Rubin NVL72 rack-scale architecture, with day-one platform readiness in Ubuntu, version 26.04 LTS (Long-Term Support: at least 15 years for enterprise customers).

AMD and Intel

It’s clear Ubuntu and NVIDIA enjoy a strong partnership, but Canonical aims to remain neutral, Jon says:

“We have an amazing partnership with NVIDIA, but we do the same with Intel, the same with AMD, the same with Qualcomm, and the same with MediaTek because in reality there is hardware being released every day, and if we don’t maintain those partnerships, the ecosystem becomes even more fragmented than it already naturally is.”

Last December, Ubuntu announced native support for AMD ROCm, and also ships with Intel’s OpenVINO toolkit. Ubuntu 26.04 LTS will be the first major distribution to natively package all three GPU compute stacks — NVIDIA, AMD, and Intel — with long-term enterprise support. Under Ubuntu Pro, ROCm LTS releases receive up to 15 years of security maintenance.

Security maintenance means that if vulnerabilities or critical incompatibilities are discovered in an LTS version, Canonical will patch them even if the upstream vendor no longer supports those versions and no longer backports security patches.

AMD Instinct accelerators are gaining traction in HPCs and sovereign AI deployments, as enterprises look for alternatives to CUDA-locked hardware. AMD’s SVP and Chief Software Officer, Andrej Zdravkovic, said the partnership would make it “easier for developers and enterprises to deploy AMD solutions on supported systems.”

Chip vendors want to collaborate because it means less work for them to add operating system-level support. Jon:

“It’s a win-win on both ends. Silicon companies are in the business of building the best chips they can, and partnering with Canonical means they have to concentrate on fewer things which are not their core focus. My hope is that partnering with Canonical helps them to focus on what they’re best at, while enabling us to help with what we’re best at: integrating, shipping and maintaining a Linux distribution.”

3. Architecture variants

Modern x86 processors support multiple instruction set generations: x86_64 v1, v2, v3, v4, and v5. ARM has a similar hierarchy. Each generation adds capabilities, such as AVX-512 instructions that accelerate machine learning workloads.

Let’s take the x86_64 instruction set. The instruction set is versioned. These are the versions:

• For x86_64: v1, v2, v3, v4, v5…

• For ARM: ARM v8.2, v8.3, v9…

Until recently, Ubuntu ran slower on newer CPUs in order to keep supporting older ones. So, when installing Ubuntu compiled for AMD64, the OS supported architecture variants for AMD64 v1.

Supporting v1 has the advantage that the oldest of AMD64 processors can run this Ubuntu version. But if Ubuntu decided to support v2 instructions, then v1 processors could not run the OS! The OS did not use the new instructions; for example, a modern processor with hardware accelerators like AVX-512, didn’t use them.

Canonical has reworked its build infrastructure to produce binaries with specific architecture variant support. So, in the case of running an x86_64 v3 compatible processor, you can download an Ubuntu OS variant that’s compiled specifically for x86_64 v3.

One tradeoff the Ubuntu team had to make was building binaries several times, which takes up more processing time and storage at their end. Then again, the Ubuntu team doing this once means that users don’t need to do recompilation, which made it an easy tradeoff, Jon told me.

Now, Ubuntu supports x86_64 v3 as an architecture variant and plans to do more. Jon says:

“Today, we’ve released x86_64 v3 as a variant, but the capability in our build and delivery pipelines unlocks the ability to add variants for the next RISC-V RVA versions, for ARMv9, ARMv10, ARMv11 and so on.

We will start now onboarding variants to make sure that when you go and buy your latest Snapdragon laptop, your operating system and all of the parts of it are using the silicon to its fullest.”

Adding support for architecture variants was a significant undertaking. Jon explains:

“This work was especially complex because combined with having the hardware physically available in the build farm, Canonical also needed to make the build scheduler aware, and thread the capability through the build systems of Debian packages, Snaps, OCI images, virtual machine images, etc. As it stands, the capability exists for Debian packages, and support for further package types will land shortly.

In addition to the build infrastructure, work needed to be done on downstream package managers (apt, snap, …) and schedulers to ensure they pull the right version of packages, and consideration needs to be given to what happens if a VM containing x86_64 v3 code ends up trying to boot on v1 hardware, and so on.”

4. Betting on local-first & plans for agentic workflows

If you’ve tried to run an LLM locally on your machine, you’ll know it comes with friction. Jon:

Read more

Let’s get to today’s topics:

1. New trend: token spend breaks budgets – what next? In the past 2-3 months, spending on AI agents has exploded at many tech companies, and the ramifications of this are starting to dawn on engineering leaders. We’ve sourced details from 15 companies, including the different ways they are coping with this realization.

2. New trend: more AI vendors can’t keep up with demand. Related to massively increased spending, GitHub Copilot and Anthropic are starting to limit less-profitable individual users, so they can serve business users whose spend has easily 10x’d in the last few months. The exception is OpenAI and Codex.

3. Morale at Meta hits all-time low? Business is booming but devs at Meta are furious and worried due to looming layoffs, and an invasive tracking program rolled out to all US employees.

1. New trend: token spend breaks budgets – what next?

Read more

Listen and watch now on YouTube, Spotify, and Apple. See the episode transcript at the top of this page, and timestamps for the episode at the bottom.

Brought to You by

• Statsig – ⁠ The unified platform for flags, analytics, experiments, and more. Stop switching between different tools, and have them all in one place.

• Sonar – The makers of SonarQube, the industry standard for code verification and automated code review. Sonar helps teams close the “architecture gap” by preventing code complexity and structural decay. Learn how Sonar is empowering the Agent Centric Development Cycle with new architecture management capabilities that ensure both humans and AI agents respect your system’s blueprint.

• WorkOS – Designing large systems is about tradeoffs. But one thing isn’t a tradeoff: enterprise features. WorkOS gives you APIs to ship enterprise features – SSO, directory sync, RBAC, audit logs – in days, not months. Visit WorkOS.com to learn more.

In this episode

Martin Kleppmann is a researcher and the author of Designing Data-Intensive Applications, one of the most influential books on modern distributed systems. As of this month, the second, heavily updated edition of the book is out.

In this episode of Pragmatic Engineer, we discuss Martin’s career in tech building startups, how he ended up writing this iconic book, and what he’s focused on, these days, after moving from industry, into academia.

We talk about the tradeoffs behind modern infrastructure, how the cloud has changed what it means to scale, and the thinking behind Designing Data-Intensive Applications, including what’s changing in the second edition.

Martin reflects on lessons from building startups like Rapportive, which he sold to LinkedIn, and shares how his experience in both academia and industry shaped his perspective.

We also explore what’s ahead: why formal verification may become more important in an AI-assisted world, the challenges of building local-first software, and his recent research into using cryptography to improve transparency in supply chains without exposing sensitive data.

Key observations from Martin

Here are 12 of my most interesting takeaways from talking with Martin:

1. Seeing Kafka as it was built at LinkedIn heavily shaped the ideas behind the book. Kafka (a popular event streaming platform) was open-sourced while Martin was at LinkedIn. Seeing this large system up close helped Martin build a mental model of how various data systems fit together, what they have in common, and their fundamental principles.

2. Martin wrote the book because he wished he had this resource when they were “drowning” in design decisions at his startup. At Rapportive, they hit database performance problems and were searching in the dark, with no idea what to do, because they lacked foundations. Martin wrote the book, so hopefully others won’t have to learn the fundamentals the hard way that his team did.

3. Knowing system internals as a superpower for application developers. Martin maintains that Designing Data-Intensive Applications is not a book for people who build databases or even infrastructure, but it’s helpful for application developers to develop an intuition for making good design decisions and debugging performance issues they will encounter.

4. Multi-region and multi-cloud are risk/cost trade-offs, not best practices. Martin does not believe that there is a “best practice” in deciding whether to go multi-region or multi-cloud. This decision is a tradeoff between risk and costs. It’s a business decision to be made. Designing Data-Intensive Applications gives engineers the vocabulary to articulate the tradeoffs, not to dictate answers.

5. Scaling down can be as challenging as scaling up. When talking about scaling systems, most engineers associate this with scaling up. But building a system that can operate efficiently and scale down when there’s less traffic is an exciting (and challenging) problem as well! Solutions like Serverless are valuable building blocks for scaling down efficiently.

6. Replication for fault tolerance is more relevant these days than sharding. Though the book has a full chapter on sharding, Martin said that the cloud has reduced the need for manual sharding for the majority of teams. This is also because machines are increasingly bigger, and more workloads fit on a single machine. Sharding across machines is increasingly a specialist concern; replication for fault tolerance, however, is still relevant at every scale.

7. MapReduce might be “dead,” but it is still worth knowing about. The second edition of the book cut most MapReduce coverage because Martin observed that, these days, practically nobody uses it: technologies like Spark and Flink have replaced MapReduce. The second edition of the book has a reference to MapReduce purely as a learning tool, for understanding partitioned batch systems.

8. Distributed systems theory makes deliberately paranoid assumptions: this is on purpose! The theory assumes that there’s no upper bound on how long it might take for a message to go over the network: it might arrive in 100 microseconds or 10 years. Clocks, crashes, and network delays all get similarly worst-case treatment. Occasionally, reality will hit some of these extremes!

9. An engineer’s job is increasingly about surfacing risks — including societal ones — to decision-makers. Martin believes that engineers need to articulate tradeoffs in a way that enables business leaders to make informed decisions. These tradeoffs include reputational and societal risks, not just technical ones.

10. Formal verification was too expensive to use across the industry, and LLMs may change this. Martin said that he never used formal verification in his time in the industry because it was too time-consuming. Now he sees two things happening at once:

• LLMs are producing so much code that human review becomes the bottleneck

• LLMs are getting good at writing formal proofs as well

Put both together, and we might see more formal verification happening!

11. Building local-first software has difficult engineering challenges. Decentralized access control sounds trivial, but it becomes pretty hard without a single server to arbitrate. For example, a revoked user can make a concurrent edit, and different devices will disagree about what happened. Martin is currently working in this problem space.

​12. Industry and academia dismiss each other, and this is not great for either field! The tech industry calls academia “theoretical” and misses useful research. Academia, in turn, often calls industry work just engineering and misses the interesting problems they solve. Martin has worked in both industry and academia, and would like to build better respect in both directions. The best PhD students he works with have a few years of real engineering experience.

The Pragmatic Engineer deepdives relevant for this episode

• Building Bluesky: a distributed social network (Martin is an advisor at Bluesky)

• Inside Uber’s move to the cloud

• The history of servers, the cloud, and what’s next

• The past and future of modern backend practices

• How Kubernetes is built

Timestamps

(00:00) Early career

(05:46) Building Rapportive

(10:47) Working at LinkedIn

(14:09) Writing Designing Data-Intensive Applications

(23:00) Reliability, scalability, and repeatability

(26:24) DDIA: the second edition

(30:50) Tradeoffs of using cloud services

(39:02) How the cloud changed scaling

(42:53) The trouble with distributed systems

(49:02) Ethics for software engineers

(52:45) Formal verification

(1:00:12) Academia vs. industry

(1:03:50) Local-first software

(1:09:50) Computer science education

(1:18:32) Martin’s current research and advice

References

Where to find Martin:

• LinkedIn: https://www.linkedin.com/in/martinkleppmann

• Bluesky: https://bsky.app/profile/martin.kleppmann.com

• Website: https://martin.kleppmann.com

• Distributed Systems lecture series: https://www.youtube.com/playlist?list=PLeKd45zvjcDFUEv_ohr_HdUFe97RItdiB

• Designing Data Intensive Applications, 2nd edition: https://learning.oreilly.com/library/view/designing-data-intensive-applications/9781098119058

Mentions during the episode:

• Selenium: https://www.selenium.dev

• SauceLabs: https://saucelabs.com

• Rapportive on YC’s website: https://www.ycombinator.com/companies/rapportive

• Kafka: https://engineering.linkedin.com/teams/data/data-infrastructure/streams/kafka

• The Log: What every software engineer should know about real-time data’s unifying abstraction: https://engineering.linkedin.com/teams/data/data-infrastructure/streams/kafka

• Materialized View:

Materialized View

Software infrastructure hot takes, projects, papers, developer interviews, and deep dives. Brought to you by Chris Riccomini.

By Chris

• The Missing README: A Guide for the New Software Engineer: https://www.amazon.com/Missing-README-Guide-Software-Engineer/dp/1718501838

• How AWS S3 is built: https://newsletter.pragmaticengineer.com/p/how-aws-s3-is-built

• MapReduce: https://en.wikipedia.org/wiki/MapReduce

• Prediction: AI will make formal verification go mainstream: https://martin.kleppmann.com/2025/12/08/ai-formal-verification.html

• Isabelle proof assistant: https://isabelle.in.tum.de

• Rocq: https://rocq-prover.org

• Lean: https://lean-lang.org

• TLA+: https://github.com/tlaplus

• FizzBee: https://fizzbee.io

• Local-First Software: You Own Your Data, in spite of the Cloud: https://martin.kleppmann.com/papers/local-first.pdf

• How AI assistance impacts the formation of coding skills: https://www.anthropic.com/research/AI-assistance-coding-skills

• Cryptography: https://en.wikipedia.org/wiki/Cryptography

—

Production and marketing by Pen Name.