Nine years later, he felt the time was ripe for an updated edition, with cloud computing much more widespread than in 2016. So, Martin teamed up with software engineer and investor, Chris Riccomini, a former colleague at LinkedIn and the author of The Missing README, for a full refresh of the book which brings it right up to date for the present day.

Martin was recently on The Pragmatic Engineer Podcast, where we discussed this updated volume and many related cloud computing matters. We also looked into some topics that have become less relevant over time, like details on MapReduce.

I asked Martin if this newsletter could share an excerpt of the updated edition of the book about a timeless, important topic, and he generously agreed. So, today we cover:

1. Cloud versus self-hosting tradeoffs

2. Doing the right thing as a software engineer

These excerpts are only part of the book; the first edition has been on my shelf for years and is now in well-worn condition. I jumped at the chance to get the second edition, and if you’re interested in building resilient systems, I recommend it as an excellent resource.

Get the second edition of the book

My usual disclaimer: as with all my recommendations, I was not paid for this article, and none of the links are affiliates. See my ethics statement for more.

The excerpt below is from “Designing Data-Intensive Applications” second edition, by Martin Kleppmann and Chris Riccomini. Copyright © 2026 Martin Kleppmann, Chris Riccomini. Published by O’Reilly Media, Inc. Used with permission.

1. Cloud versus self-hosting tradeoffs

This excerpt is from Chapter 1: “Trade-Offs in Data Systems Architecture”

For anything that an organization needs to do, one of the first questions is whether it should be done in-house or outsourced. That is, should you build or should you buy?

Ultimately, this is a question about business priorities. A common rule of thumb is that things that are a core competency or a competitive advantage of your organization should be done in-house, whereas things that are non-core, routine, or commonplace should be left to a vendor [20]. To give an extreme example, most companies do not fabricate their own CPUs, since it is cheaper to buy them from the semiconductor manufacturers.

With software, two important decisions to be made are who builds the software and who deploys it. The spectrum of possibilities is illustrated in Figure 1-2. At one extreme is bespoke software that you write and run in-house; at the other extreme are widely-used cloud services or SaaS products that are implemented and operated by an external vendor and that you access only through a web interface or API.

The middle ground is off-the-shelf software (open source or commercial) that you self-host, or deploy yourself – for example, if you download MySQL and install it on a server you control. This could be on your own hardware (often called ‘on-premises,’ even if the server is in a rented datacenter rack and not literally on your own premises), or on a virtual machine (VM) in the cloud (infrastructure as a service, or IaaS). There are more points along this spectrum, such as taking open source software and running a modified version of it.

A related question is how you deploy services, either in the cloud or on premises – for example, whether you use an orchestration framework such as Kubernetes. However, choice of deployment tooling is beyond the scope of this book, since other factors have a greater influence on the architecture of data systems.

Pros & Cons of Cloud Services

Using a cloud service, rather than running comparable software yourself, essentially outsources the operation of that software to the cloud provider. There are good arguments for and against this approach. Cloud providers claim that using their services saves time and money and allows you to move faster compared to setting up your own infrastructure.

Whether using a cloud service is actually cheaper and easier than self-hosting depends very much on your skills and the workload on your systems, however. If you already have experience of setting up and operating the systems you need, and if your load is quite predictable (i.e., the number of machines you need does not fluctuate wildly), then it’s often cheaper to buy your own machines and run the software on them yourself [21, 22].

On the other hand, if you need a system that you don’t already know how to deploy and operate, adopting a cloud service is often easier and quicker than learning to manage the system. Hiring and training staff specifically to maintain and operate the system can get very expensive. You still need an operations team when you’re using the cloud, but outsourcing the basic system administration can free up your team to focus on higher-level concerns.

Outsourcing the operation of a system to a company that specializes in running it can potentially result in better service, since the provider gains operational expertise from providing the service to many customers. On the other hand, if you run the service, you can configure and tune it to perform well on your particular workload. A cloud service would likely be unwilling to make such customizations on your behalf.

Cloud services are particularly valuable if the load on your systems varies a lot over time. If you provision your machines to be able to handle peak load, but those computing resources are idle most of the time, the system becomes less cost-effective. In this situation, cloud services have the advantage that they can make it easier to scale your computing resources up or down in response to changes in demand.

For example, analytical systems often have extremely variable load. Running a large analytical query quickly requires a lot of computing resources in parallel, but once the query completes, those resources sit idle until a user makes the next query. Predefined queries (e.g., for daily reports) can be enqueued and scheduled to smooth out the load, but for interactive queries, the faster you want them to complete, the more variable the workload becomes. If your dataset is so large that querying it quickly requires significant computing resources, using the cloud can save money as you can return unused resources to the provider rather than leaving them idle. For smaller datasets, this difference is less significant.

The biggest downside of a cloud service is that you have no control over it:

• If it is lacking a feature you need, all you can do is politely ask the vendor whether they will add it; you generally cannot implement it yourself.

• If the service goes down, all you can do is wait for it to recover.

• If you are using the service in a way that triggers a bug or causes performance problems, diagnosing the issue will be difficult. With software that you run yourself, you can get performance metrics and debugging information from the operating system to help you understand its behavior, and you can look at the server logs. With a service hosted by a vendor, you usually do not have access to these internals.

• If the service shuts down or becomes unacceptably expensive, or if the vendor changes their product in a way you don’t like, you are at their mercy; continuing to run an old version of the software is usually not an option, so you’ll be forced to migrate to an alternative service [23]. This risk is mitigated if alternative services expose a compatible API, but for many cloud services there are no standard APIs, which raises the cost of switching, making vendor lock-in a problem.

• If the cloud provider is in another country and a political conflict arises between that country and your own, you risk being locked out of the service due to imposed sanctions.

• The cloud provider needs to be trusted to keep the data secure, which can complicate the process of complying with privacy and security regulations.

Despite all these risks, it has become more and more popular for organizations to build new applications on top of cloud services, or to adopt a hybrid approach in which cloud services are used for some aspects of a system. However, cloud services will not subsume all in-house data systems. Many older systems predate the cloud, and for any services that have specialist requirements that existing cloud services cannot meet, in-house systems remain necessary. For example, very latency-sensitive applications such as high-frequency trading require full control of the hardware.

Cloud-Native System Architecture

Besides having a different economic model (subscribing to a service instead of buying hardware and licensing software to run on it), the rise of the cloud has also had a profound effect on how data systems are implemented on a technical level. The term “cloud native” is used to describe an architecture that is designed to take advantage of cloud services.

In principle, almost any software that you can self-host could also be provided as a cloud service, and indeed, such managed services are now available for many popular data systems. However, systems that have been designed from the ground up to be cloud native have been shown to have several advantages: better performance on the same hardware, faster recovery from failures, being able to quickly scale computing resources to match the load, and supporting larger datasets [24, 25, 26]. Table 1-2 lists some examples of both types of systems.

Layering of cloud services

Many self-hosted data systems have simple system requirements; they run on a conventional operating system such as Linux or Windows, they store their data as files on the filesystem, and they communicate via standard network protocols such as TCP/IP. A few systems depend on special hardware such as GPUs (for ML) or remote direct memory access (RDMA) network interfaces, but on the whole, self-hosted software tends to use generic computing resources: CPUs, RAM, a filesystem, and an IP network.

In a cloud, this type of software can be run in an IaaS environment, using one or more VMs (or instances) with a certain allocation of CPUs, memory, disk, and network bandwidth. Compared to physical machines, cloud instances can be provisioned faster and come in a greater variety of sizes, but otherwise they are similar to traditional computers: you can run any software you like on them, but you are responsible for administering it yourself.

In contrast, the key idea of cloud-native services is not only to use the computing resources managed by your operating system, but also to build upon lower-level cloud services to create higher-level services. For example:

• Object storage services such as Amazon S3, Azure Blob Storage, and Cloudflare R2 store large files. They provide more limited APIs than a typical filesystem (basic file reads and writes), but they have the advantage that they hide the underlying physical machines; the service automatically distributes the data across many machines so that you don’t have to worry about running out of disk space on any one machine. Even if some machines or their disks fail entirely, no data is lost.

• Many other services are, in turn, built upon object storage and other cloud services. For instance, Snowflake is a cloud-based analytical database (data warehouse) that relies on S3 for data storage [26], and some other services, in turn, build upon Snowflake.

As always with abstractions in computing, there is no one right answer to what you should use. As a general rule, higher-level abstractions tend to be more oriented toward particular use cases. If your needs match the situations for which a higher-level system is designed, using the existing higher-level system will probably meet your needs with much less hassle than building it yourself from lower-level systems would. On the other hand, if no high-level system meets your needs, building it yourself from lower-level components is the only option.

Separation of storage and compute

In traditional computing, disk storage is regarded as durable (we assume that once something is written to disk, it will not be lost). To tolerate the failure of an individual hard disk, RAID (redundant array of independent disks) is often used to maintain copies of the data on several disks attached to the same machine. RAID can be implemented either in hardware or in software by the operating system, and it is transparent to the applications accessing the filesystem.

In the cloud, compute instances (VMs) may also have local disks attached, but cloud-native systems typically treat these disks more like an ephemeral cache and less like long-term storage. This is because the local disk becomes inaccessible if the associated instance fails, or if the instance is replaced with a bigger or a smaller one (on a different physical machine) to adapt to changes in load.

As an alternative to local disks, cloud services also offer virtual disk storage that can be detached from one instance and attached to a different one (e.g., Amazon EBS, Azure managed disks, and persistent disks in Google Cloud). Such a virtual disk is not a physical disk, but rather a cloud service provided by a separate set of machines that emulates the behavior of a disk (a block device, where each block is typically 4 KiB in size). This technology makes it possible to run traditional disk-based software in the cloud, but the block device emulation introduces overheads that can be avoided in systems that are designed from the ground up for the cloud [24]. The use of virtual disks also makes the application very sensitive to network glitches, since every I/O operation on the virtual block device is a network call [27].

To address this problem, cloud-native services generally avoid using virtual disks and instead build on dedicated storage services that are optimized for particular workloads. Object storage services such as S3 are designed for long-term storage of fairly large files, ranging from hundreds of kilobytes to several gigabytes in size. The individual rows or values stored in a database are typically much smaller than this; cloud databases therefore typically manage smaller values in a separate service and store larger data blocks (containing many individual values) in an object store [25, 28].

In traditional systems architecture, the same computer is responsible for both storage (disk) and computation (CPU and RAM), but in cloud-native systems, these two responsibilities have become somewhat separated, or disaggregated [9, 26, 29, 30]: for example, S3 only stores files, and if you want to analyze that data, you will have to run the analysis code somewhere outside of S3. This implies transferring the data over the network.

Furthermore, cloud-native systems are often multitenant, which means that rather than having a separate machine for each customer, data and computation from several customers are handled on the same shared hardware by the same service [31]. Multitenancy can enable better hardware utilization, easier scalability, and easier management by the cloud provider, but it also requires careful engineering to ensure that one customer’s activity does not affect the performance or security of the system for other customers [32].

Operations in the Cloud Era

Traditionally, the people managing an organization’s server-side data infrastructure were known as database administrators (DBAs), or system administrators (sysadmins). More recently, many organizations have tried to integrate the roles of software development and operations into teams with a shared responsibility for both backend services and data infrastructure; the DevOps philosophy has guided this trend. Site reliability engineers (SREs) are Google’s implementation of this idea [33].

The role of operations is to ensure that services are reliably delivered to users (including configuring infrastructure and deploying applications) and to ensure a stable production environment (including monitoring and diagnosing any problems that may affect reliability). For self-hosted systems, operations traditionally involve a significant amount of work at the level of individual machines, such as capacity planning (e.g., monitoring available disk space and adding more disks before you run out of space), provisioning new machines, moving services from one machine to another, and installing operating system patches.

Many cloud services present an API that hides the individual machines implementing the service. For example, cloud storage replaces fixed-size disks with metered billing, where you can store data without planning your capacity needs in advance, and you are then charged based on the space used. Moreover, many cloud services remain highly available, even when individual machines have failed.

This shift in emphasis from individual machines to services has been accompanied by a change in the role of operations. The high-level goal of providing a reliable service remains the same, but the processes and tools have evolved.

The DevOps/SRE philosophy places greater emphasis on the following:

• Setting up automation; preferring repeatable processes over manual one-off jobs

• Using ephemeral VMs and services rather than long-running servers

• Enabling frequent application updates

• Learning from incidents

• Preserving the organization’s knowledge about the system, even as individuals come and go [34]

With the rise of cloud services, a bifurcation of roles has occurred. Operations teams at infrastructure companies specialize in the details of providing a reliable service to a large number of customers, while the customers of the service spend as little time and effort as possible on infrastructure [35].

Customers of cloud services still require operations, but they focus on different aspects, such as choosing the most appropriate service for a given task, integrating services with each other, and migrating from one service to another. Even though metered billing removes the need for capacity planning in the traditional sense, it’s still important to know what resources you are using for which purpose so that you don’t waste money on cloud resources that are not needed. Capacity planning becomes financial planning, and performance optimization becomes cost optimization [36]. Additionally, cloud services do have resource limits or quotas (such as the maximum number of processes you can run concurrently), which you need to know about and plan for before you run into them [37].

Adopting a cloud service can be easier and quicker than provisioning and running your own infrastructure, although you still have to learn how to use the cloud service and perhaps work around its limitations. Integration among services becomes a particular challenge as a growing number of vendors offer an ever-broader range of cloud services targeting different use cases [38, 39]. ETL is only part of the story; operational cloud services also need to be integrated with each other. At present, we lack standards to facilitate this sort of integration, so it often involves significant manual effort.

Other operational aspects that cannot fully be outsourced to cloud services include maintaining the security of an application and the libraries it uses, managing the interactions between your own services, monitoring the load on your services, and tracking down the cause of problems such as performance degradations or outages. While the cloud is changing the role of operations, the need for operations is as great as ever.

2. Doing the right thing as a software engineer

The excerpt below is a section from Chapter 14, “Doing the Right Thing”

In the final chapter of this book, let’s take a step back. Throughout, we have examined a wide range of architectures for data systems, evaluated their pros and cons, and explored techniques for building reliable, scalable, and maintainable applications. However, we have left out a fundamental part of the discussion, which we should now fill in.

Every system is built for a purpose; every action we take has both intended and unintended consequences. The purpose may be as simple as making money, but the consequences may be far-reaching. We, the engineers building these systems, have a responsibility to carefully consider those consequences and to ensure that our decisions do not cause harm.

We talk about data as an abstract thing, but remember that many datasets are about people: their behavior, their interests, their identities. We must treat such data with humanity and respect. Users are humans too, and human dignity is paramount [1].

Software development increasingly involves making important ethical choices. There are guidelines to help software engineers navigate these issues, such as the ACM Code of Ethics and Professional Conduct [2], but they are rarely discussed, applied, or enforced in practice. As a result, engineers and product managers sometimes take a cavalier attitude to privacy and the potential negative consequences of their products [3, 4].

A technology is not good or bad in itself – what matters is how it is used and how it affects people. This is true of a software system such as a search engine in much the same way as it is for a weapon like a gun. The ethical responsibility is ours to bear; it is not sufficient for software engineers to focus exclusively on the technology and ignore its consequences.

In contrast to much of computing, however, the concepts at the heart of ethics are not fixed or determinate in their precise meaning; they require interpretation, which may be subjective [5]. What makes something “good” or “bad” is not well defined, and serious discourse on the subject among computing professionals is lacking [6]. Reasoning about ethics is difficult, but also too important to ignore. What does this entail? “Ethics” are not a checklist with which to comply; it’s a participatory and iterative process of reflection, in dialogue with people involved and accountability for the results [7].

Predictive Analytics

Predictive analytics is a major part of why people are excited about big data and AI. It’s also an area that is fraught with ethical dilemmas. Using data analysis to predict the weather, or the spread of diseases, is one thing [8]; it is another matter to predict whether a convict is likely to reoffend, whether an applicant for a loan is likely to default, or whether an insurance customer is likely to make expensive claims [9]. The latter have a direct effect on people’s lives.

Naturally, payment networks want to prevent fraudulent transactions, banks want to avoid bad loans, airlines want to avoid hijackings, and companies want to avoid hiring ineffective or untrustworthy people. From their point of view, the cost of a missed business opportunity is low, but the cost of a bad loan or a problematic employee is much higher, so it is expected for organizations to be cautious. If in doubt, they are better off saying “no”.

However, as algorithmic decision making becomes more widespread, someone who has (accurately or falsely) been labeled as risky by an algorithm may suffer a large number of “no” decisions. Systematically being excluded from jobs, air travel, insurance coverage, property rental, financial services, and other key aspects of society is such a large constraint on an individual’s freedom that it has been called “algorithmic prison” [10]. In countries that respect human rights, the criminal justice system presumes innocence until proven guilty; on the other hand, automated systems can systematically and arbitrarily exclude a person from participating in society without any proof of guilt and little chance of appeal.

Bias & discrimination

Decisions made by an algorithm are not necessarily any better or any worse than those made by a human. Everyone is likely to have biases, even if they actively try to counteract them, and discriminatory practices can become culturally institutionalized. There is hope that basing decisions on data, rather than subjective and instinctive human assessments, could be more fair and give a better chance to people who are often overlooked or disadvantaged in the traditional system [11].

When we develop predictive analytics and AI systems, we are not merely automating a human’s decision by using software to specify the rules for when to say “yes” or “no”; we are leaving the rules themselves to be inferred from data. However, the patterns learned by these systems are opaque: even if the data indicates a correlation, we may not know why. If the input to an algorithm carries a systematic bias, the system will most likely learn and amplify that bias in its output [12].

In many countries, anti-discrimination laws prohibit treating people differently depending on protected traits such as ethnicity, age, gender, sexuality, disability, or beliefs. Other features of a person’s data may be analyzed, but what happens if they are correlated with protected traits? For example, in racially segregated neighborhoods, a person’s postal code or even their IP address is a strong predictor of race. Put like this, it seems ridiculous to believe that an algorithm could somehow take biased data as input and produce fair and impartial output from it [13, 14]. Yet this belief often seems to be implied by proponents of data-driven decision making; an attitude that has been satirized as “machine learning is like money laundering for bias” [15].

Predictive analytics systems merely extrapolate from the past; if the past is discriminatory, they codify and amplify that discrimination [16]. If we want the future to be better than the past, moral imagination is required, and that’s something only humans can provide [17]. Data and models should be our tools, not our masters.

Responsibility and Accountability

Automated decision-making raises the question of responsibility and accountability [17]. If a human makes a mistake, they can be held accountable, and the person affected by the decision can appeal. Algorithms make mistakes too, but who is accountable when they go wrong? [18] When a self-driving car causes an accident, who is responsible? If an automated credit scoring algorithm systematically discriminates against people of a particular race or religion, is there any recourse? If a decision by your ML system comes under judicial review, can you explain to the judge how the algorithm made its decision? People should not be able to evade responsibility by blaming an algorithm.

Credit rating agencies are a classic example of collecting data to make decisions about people. A bad credit score makes life difficult, but at least a credit score is normally based on relevant facts about a person’s actual borrowing history, and any errors in the record can be corrected (although the agencies normally do not make this easy). Scoring algorithms based on machine learning, however, typically use a much wider range of inputs and are much more opaque, making it harder to understand how a particular decision has come about and whether someone is being treated in an unfair or discriminatory way [19].

A credit score summarizes “how did you behave in the past?” whereas predictive analytics usually work on the basis of “who is similar to you, and how did people like you behave in the past?” Drawing parallels to others’ behavior implies stereotyping people; for example, based on where they live (a close proxy for race and socioeconomic class). What about people put in the wrong bucket? Furthermore, if a decision is incorrect because of erroneous data, recourse is almost impossible [17].

Much data is statistical in nature, which means that even if the probability distribution on the whole is correct, individual cases may well be wrong. For example, if the average life expectancy in your country is 80 years, that doesn’t mean you’re expected to drop dead on your 80th birthday. From the average and the probability distribution, you can’t say much about the age to which someone will live. Similarly, the output of a prediction system is probabilistic and may well be wrong in individual cases.

A blind belief in the supremacy of data for making decisions is not only delusional, but also positively dangerous. As data-driven decision making becomes more widespread, we will need to figure out how to avoid reinforcing existing biases, how to make algorithms accountable and transparent, and how to fix them when they inevitably make mistakes.

We will also need to figure out how to realize the positive potential of data and prevent it from being used to harm people. For example, analytics can reveal financial and social characteristics about personal lives. On the one hand, this power could be used to focus aid and support to help those who need it most. On the other hand, it is sometimes used by predatory businesses seeking to identify vulnerable people and sell them risky products such as high-cost loans or worthless college degrees [17, 20].

Feedback loops

Even with predictive applications with less immediately far-reaching effects on people, such as recommendation systems, there are difficult issues that we must confront. When services become good at predicting the content users want to see, they may end up showing them only opinions they already agree with, leading to echo chambers in which stereotypes, misinformation, and polarization can breed. We already know the impact that social media echo chambers can have on election campaigns.

When predictive analytics affect people’s lives, particularly pernicious problems arise because of self-reinforcing feedback loops. For example, consider the case of employers using credit scores to evaluate potential hires. You may be a good worker with a good credit score, but suddenly find yourself in financial difficulties due to a misfortune beyond your control. As you miss payments on your bills, your credit score suffers, and you will be less likely to find work. Joblessness pushes you toward poverty, which further worsens your score, making it even harder to find employment [17]. It’s a downward spiral due to poisonous assumptions, hidden behind a camouflage of mathematical rigor and data.

As another example of a feedback loop, economists found that when gas stations in Germany introduced algorithmic prices, competition was reduced and prices for consumers went up because the algorithms learned to collude [21].

We can’t always predict when such feedback loops may happen. However, many consequences can be predicted by thinking about an entire system (not just the computerized parts, but also the people interacting with it), in an approach known as “systems thinking” [22]. We can try to understand how a data analysis system responds to different behaviors, structures, or characteristics. Does the system reinforce and amplify existing differences between people (e.g., making the rich richer or the poor poorer), or does it try to combat injustice? Even with the best intentions, we must beware of the possibility of unintended consequences.

Surveillance

The excerpt below is from another section in Chapter 14, “Doing the Right Thing”

As a thought experiment, try replacing the word “data” with “surveillance”, and observe whether common phrases still sound so good [23]. How about this: “In our surveillance-driven organization we collect real-time surveillance streams and store them in our surveillance warehouse. Our surveillance scientists use advanced analytics and surveillance processing in order to derive new insights.”

This thought experiment is unusually polemical for this book, “Designing Surveillance-Intensive Applications”, but strong words are needed to emphasize this point. In our attempts to make software “eat the world” [24], we have built the greatest mass surveillance infrastructure ever seen. We are rapidly approaching a world in which every inhabited space contains at least one internet-connected microphone, in the form of smartphones, smart TVs, voice-controlled assistant devices, baby monitors, and even children’s toys that use cloud-based speech recognition. Many of these devices have terrible security track records [25].

What is new compared to the past is that digitization has made it easy to collect large amounts of data about people. Surveillance of our location and movements, our social relationships and communications, our purchases and payments, and our health data has become almost unavoidable. A surveillance organization may end up knowing more about a person than that person knows about themselves; for example, identifying illnesses or economic problems before that individual is aware of them.

Even the most totalitarian, repressive regimes of the past could only dream of putting a microphone in every room and forcing every person to constantly carry a device capable of tracking their location and movements. Yet the benefits that we get from digital technology are so great that we now voluntarily accept this state of total surveillance. The difference is just that the data is being collected by corporations to provide us with services, rather than government agencies seeking control [26].

Not all data collection necessarily qualifies as surveillance, but examining it as such can help us understand our relationship with the data collector. Why are we seemingly happy to accept surveillance by corporations? Perhaps you feel you have nothing to hide; in other words, you are totally in line with existing power structures, you are not a marginalized minority, and you needn’t fear persecution [27]. Not everyone is so fortunate. Or perhaps it’s because the purpose seems benign; it’s not overt coercion and conformance, merely better recommendations and more personalized marketing. However, combined with the discussion of predictive analytics from the last section, that distinction seems less clear.

We are already seeing behavioral data about driving, tracked by vehicles without drivers’ consent, affecting their insurance premiums [28], and health insurance coverage that depends on people wearing a fitness tracking device. When surveillance is used to make decisions that hold sway over important aspects of life, such as insurance coverage or employment, it starts to appear less benign. Data analysis can also reveal surprisingly intrusive things; for example, the movement sensor in a smartwatch or fitness tracker can be used to work out what you are typing (e.g., passwords) with fairly good accuracy [29]. Sensor accuracy and algorithms for analysis are only going to get better.

Takeaways

Thanks to Martin for writing this book, and to himself and Chris for doing a revamp for the second edition. The volume is now even more relevant to how we build systems in 2026 and beyond. You can purchase a hard copy from the publisher’s website or Amazon.

The first edition has a timeless quality because it focused on the fundamentals of large systems, and the new second edition follows the same approach, as laid out in its preface:

“Although the landscape of technologies for processing and storing data is diverse and fast-changing, the underlying principles endure. If you understand those principles, you’re in a position to see where each tool fits in, how to make good use of it, and how to avoid its pitfalls. This book focuses on those principles.”

Since the first edition appeared nine years ago, some things have changed in the tech industry:

• Much greater focus on the cloud. Building large systems on top of cloud infrastructure is more common. This brings lower complexity as cloud primitives hide a lot of implementation complexity, but it also means accepting more risk because when the cloud is down, so is your system.

• Systems which AI tools build upon are more relevant. Vector databases, DataFrames (for training datasets), and the processing of large amounts of training data with batch processing systems are relevant to anyone building production AI systems.

• Local-first software. Martin focuses on this area in his work, and with AI, we could see more demand for running models locally. Operating systems like Ubuntu are also focusing on this.

• Formal methods. The advent of AI-generated code means this topic is getting more attention industry-wide, and the second edition covers it.

• Regulation and legal context. Regulations like the EU’s General Data Protection Regulation (GDPR) are something software engineers increasingly need to know about, and the book now covers it.

If I had to summarize the evolution of the book in its second edition, it would be more focus on cloud and AI, and more on local-first software, testing, and how regulations affect engineers. Interestingly, this mirrors how the tech industry has developed over time, too.

I very much appreciate that the book closes with the final chapter focused on “doing the right thing” as a software engineer. Software systems have wide-ranging societal impact, and engineers working on these systems have a great say in what gets built, and how it gets built. As engineers, we owe it the very least to ourselves to consider the broader impact of our decisions — and doing so might also force us to make important ethical choices. There’s less discussion of the ethics angle on software engineering: and I’m glad that Martin and Chris did not shy away from going deeper into this topic.

If you’d like to get more background on the book – and on the hard parts of building large-scale systems – check out our podcast episode with Martin Kleppmann.

Today’s article focuses on Linux and Ubuntu, and we’ll cover Windows in a follow-up issue. Obviously, I reached out to Apple but heard nothing back, unsurprisingly. If you’re reading this and happen to work at Apple, it’d be great to learn more!

Jon Seager is VP of Engineering at Canonical – the company behind Ubuntu – and has provided new details about what the team there has built for AI support, and some new ideas that they’re brewing up. Today, we cover:

1. Hardware enablement: support for GPUs, NPUs and DPUs. When you turn on a machine with AI accelerators, Ubuntu aims for the hardware to perform at its full potential. This means having proper driver support for PCs and cloud data centers’ computing units.

2. Hardware partnerships. Working closely with NVIDIA, AMD, and Intel means Ubuntu can support those vendors’ new hardware from release day.

3. CPU architecture variants. New versions in a CPU family add to, or change, features. An operating system needs to support a new version of the CPU architecture variant in order to fully utilize it. Ubuntu does this for the x86‑64 family, making it a lot more performant on newer CPUs – while still supporting older CPUs.

4. Local-first bet & plans for agentic workflows. There’s a big focus on running local models and using “inference snaps” which help choose the right model with the right quantization. There is the intention to support agentic workflows at the OS level, one day, which is currently at the early exploration stage.

5. Developer ecosystem. There’s a plan to add more support for AI dev tools, a focus on sandboxing at the OS level, a push to support ARM64 laptops more, and we touch on the popularity of Windows Subsystem for Linux (WSL).

6. Engineering culture. A skeptical attitude to AI at Canonical has given way to one where experimentation is encouraged and devs lean into AI tools, but there are no targets for token usage or amounts of AI-generated code.

7. What other Linux distributions are doing. Arch Linux takes the “DIY your AI setup” approach, Omarchy makes it easy to install AI tools, while Red Hat Enterprise Linux ships with AI integrated into the command-line and support for AI accelerators & popular AI tools.

The bottom of this article could be cut off in some email clients. Read the full article uninterrupted, online.

Read the full article online

1. Hardware enablement: support for GPUs, NPUs & DPUs

Jon mentioned he detects a “Dotcom Boom”-era vibe in the industry, like around when “web 1.0” was created, and indeed, lots of startups today aim to be the Google-style success story of this “AI era”. At Canonical, the team asked: what does that mean for Ubuntu as an operating system?

For instance, should Ubuntu join the competition and try to position itself closer to AI, or keep focusing on what they’ve done for decades: build an operating system? Jon said:

“We need to make sure to remain a relatable and accessible system. I don’t think we should blur the line between application features and the OS itself. So, the most powerful thing we can do is hardware enablement.”

Hardware enablement means that if a computer (typically, a laptop) has AI-related hardware, Ubuntu should allow it to make full use of it. This involves adding support for GPUs, NPUs, DPUs and other types of accelerator cards. Let’s briefly go through each.

GPUs

As is likely widely known by readers, ‘GPU’ stands for Graphics Processing Unit. Originally built for graphics rendering, its #1 use case is no longer in video games but for AI training and inference. GPUs come in two forms:

• Integrated GPUs: located on the same die (integrated circuit) as the CPU, like GPUs on Apple’s M-series processors

• Discrete GPUs: separate chips on their own board; often for gaming, or in standalone GPU rigs for AI and ML workloads

NVIDIA leads the market in discrete GPUs for rigs with its Blackwell family, and in standalone GPU cards with the NVIDIA RTX series. Other vendors like AMD offer GPUs for data centers (like the Instinct MI300 Series) and for PCs with the AMD Radeon series.

NPUs

Neural Processing Units (NPUs) are also called “AI accelerators.” This is a dedicated block on the System-on-a-chip (SoC), on modern processors especially designed for running AI inference efficiently on‑device. Since 2022, many modern processors have had a dedicated NPU block, including all Apple’s M-series chips (from M1 and up), Intel’s Core Ultra and Core Ultra “Series 2”, AMD’s Ryzen AI 300 series, and also Qualcomm’s Snapdragon X Elite and Snapdragon X Plus.

A number shared for each NPU is TOPS. TOPS means Tera (trillions) of Operations Per Second, and the said operation is a “multiply-accumulate” (MAC) one, which Qualcomm describes as:

“A multiply-accumulate (MAC) operation executes the mathematical formulas at the core of AI workloads. A matrix multiply consists of a series of two fundamental operations: multiplication and addition to an accumulator. A MAC unit can, for example, run one of each per clock cycle, meaning it executes two operations per clock cycle. A given NPU has a set number of MAC units that can operate at varying levels of precision, depending on the NPU’s architecture.”

How TOPS is calculated: TOPS = 2 × MAC unit count × Frequency / 1 trillion.

“Frequency” refers to the clock speed (cycles per second) at which an NPU and its MAC units (as well as a CPU or GPU) operate, which directly influences overall performance. Processors at higher frequencies allow for more operations, but higher frequencies also mean more energy consumed, heat generated, and battery life decreased. The TOPS number that’s quoted for processors is generally the peak operating frequency.

NPUs are often ideal for low-power, local inference, and for running smaller, local models. They can be useful for things like Local speech‑to‑text (dictation, captions, meeting transcription), video background blur/replacement or auto‑framing, small local language summarization, etc. NPUs are more typical of laptop and PC processors, although some phone processors ship with them like the iPhone (A-series chips) and Google’s Tensor processor in Pixel phones. Basically, NPUs promise to bring efficiently-running local models on laptops one step closer.

DPUs

Data Processing Units (DPUs) are typically found in data centers, moving massive amounts of data fast. NVIDIA’s explanation:

“The CPU is for general-purpose computing, the GPU is for accelerated computing, and the DPU, which moves data around the data center, does data processing.

A DPU is a new class of programmable processor that combines three key elements. A DPU is a system on a chip, or SoC, that combines:

• An industry-standard, high-performance, software-programmable, multi-core CPU, typically based on the widely used Arm architecture, tightly coupled to the other SoC components.

• A high-performance network interface capable of parsing, processing and efficiently transferring data at line rate, or the speed of the rest of the network, to GPUs and CPUs.

• A rich set of flexible and programmable acceleration engines that offload and improve applications’ performance for AI and machine learning, zero-trust security, telecommunications, and storage, among others.”

Several major chipmakers manufacture DPUs, of which NVIDIA’s BlueField family is the most widespread. Others include AMD Pensando DPUs (Elba, Giglio), and Intel IPU / DPU cards (E2100, E2200 series).

DPUs are most commonly deployed inside Hyperscale cloud providers (AWS, Azure, GCP, OCI), or in AI and high-performance computing (HPC) data centers, or larger private clouds. DPUs make sense when GPU traffic is huge, or when the network telemetry overhead is so great that it could overwhelm the CPUs processing the data transfer.

2. Hardware partnerships

It’s easiest to add support to hardware by working with leading chip manufacturers, so Ubuntu has relationships with hardware vendors for that reason. As a result, the OS sometimes offers day-one support for cutting-edge AI supercomputers.

Partnership with NVIDIA

In September 2025, Canonical announced it would package and distribute the full NVIDIA CUDA toolkit directly within Ubuntu’s repositories. This deal collapsed into a single standard apt install, something that had previously been a multi-step manual installation process of downloading from NVIDIA’s site, importing GPG keys, pinning a separate APT repo – and praying nothing broke.

Packaging and distributing the CUDA toolkit makes developing with CUDA easier. From Jon:

“One of the trickiest things for developers who have to use this tech is the dance of matching the right version of Python, with the right version of CUDA, with the right driver. Projects end up with different versions of CUDA, and then machines end up breaking because the driver configuration gets inadvertently broken along the way.

The number one thing we can do as an operating system is to make this setup as easy as possible.”

Ubuntu’s strategy of working directly with chipmakers seems to be working. NVIDIA recently discontinued its custom NVIDIA DGX OS — a modified Ubuntu it maintained for years — and now ships plain Ubuntu. Jon:

“Previously, NVIDIA shipped NVIDIA DGX OS for which NVIDIA had an agreement with Canonical where they could take Ubuntu, modify it with the kernel modules and software they needed, do some product-specific optimization, and ship that as NVIDIA DGX OS.

This more recent development sees NVIDIA just shipping Ubuntu as it comes.

When NVIDIA released the DGX Spark, a $4,000 AI workstation with an ARM64 chipset, it shipped running vanilla Ubuntu as the only supported operating system.”

At CES 2026 in January, Canonical announced Ubuntu support for the NVIDIA Vera Rubin NVL72 rack-scale architecture, with day-one platform readiness in Ubuntu, version 26.04 LTS (Long-Term Support: at least 15 years for enterprise customers).

AMD and Intel

It’s clear Ubuntu and NVIDIA enjoy a strong partnership, but Canonical aims to remain neutral, Jon says:

“We have an amazing partnership with NVIDIA, but we do the same with Intel, the same with AMD, the same with Qualcomm, and the same with MediaTek because in reality there is hardware being released every day, and if we don’t maintain those partnerships, the ecosystem becomes even more fragmented than it already naturally is.”

Last December, Ubuntu announced native support for AMD ROCm, and also ships with Intel’s OpenVINO toolkit. Ubuntu 26.04 LTS will be the first major distribution to natively package all three GPU compute stacks — NVIDIA, AMD, and Intel — with long-term enterprise support. Under Ubuntu Pro, ROCm LTS releases receive up to 15 years of security maintenance.

Security maintenance means that if vulnerabilities or critical incompatibilities are discovered in an LTS version, Canonical will patch them even if the upstream vendor no longer supports those versions and no longer backports security patches.

AMD Instinct accelerators are gaining traction in HPCs and sovereign AI deployments, as enterprises look for alternatives to CUDA-locked hardware. AMD’s SVP and Chief Software Officer, Andrej Zdravkovic, said the partnership would make it “easier for developers and enterprises to deploy AMD solutions on supported systems.”

Chip vendors want to collaborate because it means less work for them to add operating system-level support. Jon:

“It’s a win-win on both ends. Silicon companies are in the business of building the best chips they can, and partnering with Canonical means they have to concentrate on fewer things which are not their core focus. My hope is that partnering with Canonical helps them to focus on what they’re best at, while enabling us to help with what we’re best at: integrating, shipping and maintaining a Linux distribution.”

3. Architecture variants

Modern x86 processors support multiple instruction set generations: x86_64 v1, v2, v3, v4, and v5. ARM has a similar hierarchy. Each generation adds capabilities, such as AVX-512 instructions that accelerate machine learning workloads.

Let’s take the x86_64 instruction set. The instruction set is versioned. These are the versions:

• For x86_64: v1, v2, v3, v4, v5…

• For ARM: ARM v8.2, v8.3, v9…

Until recently, Ubuntu ran slower on newer CPUs in order to keep supporting older ones. So, when installing Ubuntu compiled for AMD64, the OS supported architecture variants for AMD64 v1.

Supporting v1 has the advantage that the oldest of AMD64 processors can run this Ubuntu version. But if Ubuntu decided to support v2 instructions, then v1 processors could not run the OS! The OS did not use the new instructions; for example, a modern processor with hardware accelerators like AVX-512, didn’t use them.

Canonical has reworked its build infrastructure to produce binaries with specific architecture variant support. So, in the case of running an x86_64 v3 compatible processor, you can download an Ubuntu OS variant that’s compiled specifically for x86_64 v3.

One tradeoff the Ubuntu team had to make was building binaries several times, which takes up more processing time and storage at their end. Then again, the Ubuntu team doing this once means that users don’t need to do recompilation, which made it an easy tradeoff, Jon told me.

Now, Ubuntu supports x86_64 v3 as an architecture variant and plans to do more. Jon says:

“Today, we’ve released x86_64 v3 as a variant, but the capability in our build and delivery pipelines unlocks the ability to add variants for the next RISC-V RVA versions, for ARMv9, ARMv10, ARMv11 and so on.

We will start now onboarding variants to make sure that when you go and buy your latest Snapdragon laptop, your operating system and all of the parts of it are using the silicon to its fullest.”

Adding support for architecture variants was a significant undertaking. Jon explains:

“This work was especially complex because combined with having the hardware physically available in the build farm, Canonical also needed to make the build scheduler aware, and thread the capability through the build systems of Debian packages, Snaps, OCI images, virtual machine images, etc. As it stands, the capability exists for Debian packages, and support for further package types will land shortly.

In addition to the build infrastructure, work needed to be done on downstream package managers (apt, snap, …) and schedulers to ensure they pull the right version of packages, and consideration needs to be given to what happens if a VM containing x86_64 v3 code ends up trying to boot on v1 hardware, and so on.”

4. Betting on local-first & plans for agentic workflows

If you’ve tried to run an LLM locally on your machine, you’ll know it comes with friction. Jon:

Read more

Tech interviews have two parts: the technical interview – with a focus on things like coding, software architecture, problem solving – and the behavioral part – with a focus on past experience, and the situations that show you’d be a good fit at the company you’re interviewing with, along with things like attitude, motivation, culture fit. Technical interviews are going through a big change, thanks to AI tools: some companies are bringing in new, AI-assisted types of interviews, while others are trying to make “pre-AI” type interviews work.

What doesn’t seem to be changing is the second type of interviews: the behavioral ones. I’ve found the topic of behavioral interviews from a software engineer’s perspective somewhat under-discussed – even though this interview carries huge weight in securing an offer and what level you come in at. No matter how strong your technical skills are, especially at mid-sized and larger companies, you are unlikely to get an offer if you are deemed to not be a fit for what the company is looking for.

Steve Huynh was an engineer at Amazon for 17 years – I previously did a podcast episode with him on the reality of being a principal engineer at Amazon. During this time, Steve conducted nearly 1,000 interviews, of which around 600 were Bar Raiser ones. Bar Raiser interviews are unique to Amazon: it’s an interview conducted by someone outside of the hiring team, with the goal of ensuring that the new hire raises the company’s talent bar.

After leaving the e-commerce giant, Steve spent 2 years researching and writing the book Technical Behavioral Interview: An Insider’s Guide.

Today, we cover two topics on interviews and behavioral interviews:

1. Learnings from conducting ~1,000 behavioral interviews at Amazon. Steve reflects of major observations from his 17 years at Amazon, covering:

• You’re over-prepared for one interview and unprepared for the other

• How you deliver the story matters as much as the story itself

• The interview is an audition for what it’s like to work with you

2. What companies are looking for during behavioral interviews. An excerpt from Steve’s new book, Technical Behavioral Interview, covering ~75% of a full chapter of the book (out of the 14 total chapters.) We get into:

• Understanding fit: role and company

• The four dimensions that determine your level

• What each level looks like

• Reading and calibrating your own level

• Researching what companies really value

Longtime readers might remember Steve from my podcast with him a year back: What is a Principal Engineer at Amazon? With Steve Huynh

My usual disclaimer: as with all my recommendations, I was not paid for this article, and none of the links are affiliates. See my ethics statement for more.

With this, it’s over to Steve:

1. Learnings from conducting ~1,000 behavioral interviews at Amazon

A Bar Raiser is a specially trained interviewer whose job is to ensure that every hire raises the average talent level at Amazon. I had veto power over any candidate. I sat on nearly a thousand interview loops across every level from intern to Principal Engineer.

After 50 or so interviews as a Bar Raiser, the patterns became impossible to miss. And this was the biggest one:

The candidates who didn’t get offers seldom failed because they lacked technical skill. They failed because of how they presented themselves.

For sure, technical preparation is crucial, and I’m not telling you to skip it. But most candidates have massive blind spots when it comes to non-technical matters, which is a big problem. Why? Because that blind spot is where most hiring decisions are made.

The Bar Raiser who trained me put it this way:

Technical skills are the ante. They get you into the game. But they’re not what wins you the hand.

I didn’t fully appreciate what that meant until I’d seen candidates who were technically very strong get rejected because of everything else.

Think about it. By the time you’re sitting in a final round of interviews, you’ve already passed at least one technical screen or take-home assignment. The company already knows you could probably do the job. They already know you want to work with them.

But that’s not what the final round is for.

The final round is when the team figures out whether they want to work with you. Being technically proficient is part of it, but it’s not all of it. Can you explain your thinking clearly when you’re stumped? How do you handle it when things go wrong? Can they picture you in a design review or in a tough conversation with a partner team?

Fit.

Fit is what decides most hiring outcomes, yet it’s the thing most candidates spend the least time preparing for. After nearly a thousand interviews, I can tell you exactly where the gap is and how you can close it.

Learning #1: You’re over-prepared for one interview and unprepared for the other

The average candidate preparing for a tech interview probably spends 95% of their time on technical preparation and 5% on everything else. Some spend literally zero on everything else.

I get why. Technical preparation feels concrete. You can grind coding problems and measure your progress. You can study system design patterns and feel yourself getting sharper. There’s a clear input/output relationship. Do more problems, get better at problems.

For most technical interviews, even if you haven’t seen the exact problem before, you can still do a decent job. It’s simply not possible to prepare for every problem, so it’s expected that you can reason through an unfamiliar coding question and pick up on hints the interviewer gives you. You can work through a system design problem by applying fundamentals you already know. It’s expected that you will encounter new questions during an interview, so it isn’t fatal if you’re a competent engineer who can think on your feet.

However, the non-technical rounds are the opposite. You cannot wing them and expect to do well. When an interviewer says, “Tell me about a time something went wrong on a project and how you handled it” and you haven’t thought about that question before, there is no hint they can give you. There’s no reasoning your way through it in real time. You either have a prepared story ready to go, or you’re going to mumble your way through a word salad while the interviewer watches.

I’ve seen this play out hundreds of times. A candidate would crush the coding round, then I would ask them about a difficult decision they made, and they would fall apart. They would pick a half-remembered example, start rambling, backtrack to add context they forgot, in the process losing track of the question. Then, five minutes later, they would land on something like, “So, yeah, it worked out in the end.”

These candidates were often strong coders, but that didn’t matter. At the debriefs, the feedback was always some version of “I couldn’t get a concrete answer about their experience. Every story was vague and unconvincing.” We couldn’t extend an offer when a candidate couldn’t articulate how they worked.

The technical bar was met, but the hiring decision was made in the behavioral round.

Here’s what’s frustrating about this. Non-technical preparation takes a fraction of the time for technical.

If you’re going to spend 80 to 100 hours preparing for an interview cycle, spending a single weekend on your stories might be the highest-leverage investment you make.

Ten hours of story prep can completely change the outcome of your behavioral rounds. Meanwhile, your 80th hour of LeetCode will give you almost nothing you didn’t already have at 60.

The returns on technical prep diminish rapidly. The returns on story prep are exponential because almost nobody does it at all.

What to do: How are you currently splitting your interview prep time? If it’s 99% technical and 1% everything else, you’re over-indexed on the part with diminishing returns and under-indexed on the part where hiring decisions get made. You don’t need to cut your technical prep dramatically. Just reallocate. If you’re planning to spend 80 hours preparing, take 10 of those hours and move them to non-technical preparation. That reallocation will do more for your odds than 10 more hours working on practice problems.

Learning #2: How you deliver the story matters as much as the story itself

You can have the most impressive accomplishment of your career ready for your interview and completely waste it with bad delivery. The most common version of this is what I call the “ramble and stumble.”

The candidate starts talking, and you genuinely can’t tell if they’re figuring out the story as they go or if they’ve simply never said these words out loud before. Or they might give you five minutes of context and then still backtrack to add details they forgot. By the time they reach the outcome, you’ve lost track of how you got there.

Here’s something that’s always struck me as odd. If you had a big presentation at work, you’d spend hours preparing for it, right?. You’d think about the structure, the flow, the key points. You’d rehearse it. You might even do a couple of dry runs with a colleague. Nobody wants to walk into a presentation and wing it.

But in a job interview, where the stakes are arguably higher than any single presentation you’ll ever give? People wing those constantly. They walk in having never practiced their stories out loud. They might have thought about them, but they’ve never spoken the words, heard how they sound, or timed how long they take. Then they’re surprised when the words come out as a mess.

Think about any other high-stakes skill. You wouldn’t expect to be good at golf without practicing at the driving range. You wouldn’t expect to give a great keynote the first time you stepped on stage. Nobody calls a musician fake for rehearsing before a concert.

But for some reason, many people feel that preparing interview stories is inauthentic. As if it’s cheating somehow. As if the “real” version of you is the one that stumbles through an unrehearsed answer under pressure.

It’s not. The real you communicates clearly what you’ve done and what you’re capable of.

What to do: Good delivery doesn’t require a lot of charisma or natural presentation skills, but it does require practice. Start with the two questions that come up in virtually every interview: “Tell me about yourself” and “Why do you want to work here?” Write down your answers. Then record yourself delivering them. Watch the recording and take notes. Where did you ramble? Where did you fill space with filler words? Did you look nervous? Then do it again. And again. Keep going until you watch the recording back and think “That sounds like someone I’d like to work with.”

Once those two are solid, pick stories from your career and do the same thing. This process will be uncomfortable at first. Most people hate watching themselves on camera. Do it anyway. Thirty minutes of this will up-level your interview performance much more than 20 hours of coding exercises could ever do.

Learning #3: The interview is an audition for what it’s like to work with you

Most candidates think the interview is an exam. If you get the right answers, then you’ll pass the test and get the job. That’s simply not how it works. Yes, you are being evaluated, and what you say matters. But there is no answer key. The interviewer doesn’t have a rubric with the “correct” responses to which they compare your answers. They’re forming an impression of you as a person, and that impression is far more nuanced than “right” or “wrong.”

By the time you’re sitting across from the interviewer, you’ve already jumped through some technical hoops. The company already has evidence from your resume that you can code or design systems at the level they need. That bar has been cleared. The final round goes deeper on the technical side, but it’s also trying to answer a completely different question: Would we want this person on the team? Would we trust their judgment in a crisis? Would they make our team’s software better or worse?

As a Bar Raiser, my specific job was to determine whether a candidate would raise the bar, meaning that they would be better than at least 50% of the people already at the company in that role.

The thing most people don’t realize is that the type of coding we asked about in interviews wasn’t what we did on the job. Nobody was writing algorithms on a whiteboard during their workday. The questions we asked tested problem-solving ability in an artificial environment.

But the behavioral questions, the soft questions, those tested situations we dealt with every single day. Navigating disagreements, handling projects that were going sideways, influencing without authority, making tradeoffs with incomplete information. These weren’t hypothetical scenarios pulled out of a textbook. They were just another Tuesday.

So when I asked a candidate to tell me about a time they had to push back on a stakeholder, I wasn’t waiting to hear the right answer; I was picturing them in our next planning and prioritization meeting. When they described how they handled a conflict on their team, I was asking myself whether I’d want to be in that room with them. Every answer was a preview of what it would be like to work alongside that person day to day.

The candidates who treated it like a test tried to figure out what I wanted to hear and then gave me that answer. That’s exactly the wrong approach. They gave polished, rehearsed answers with no rough edges and perfect endings where everything worked out and every decision was the right one. I’d walk out thinking “I have no idea what it would actually be like to work with this person.” And when that uncertainty showed up across multiple interviews in the debrief, it almost always turned into a “No.”

What to do: For each story you’re preparing, stop thinking about what the interviewer wants to hear. Instead, think about what you’d want to hear from someone interviewing to join your team. You’d want to hear how they actually think. You’d want the real version of what happened, including the parts that were hard and the calls that were close. You’d want to walk away feeling like you understood what it would be like to work with them on a tough problem. Give your interviewer that same thing. Be honest and let them see how you think. That’s worth more than any polished answer.

What ~1,000 interviews taught me

After all those interviews, the lesson I keep coming back to is simple.

The people who get hired are the ones who can walk into a room and tell a clear story. This story is about their work and their capabilities, and makes the interviewer think, “I want to work with that person.”

Being able to tell this story is a skill. And like any skill, it gets better with practice. Most people never practice it because they don’t think of it as something you can prepare for, but you can. And a little preparation here goes further than almost anything else you can do for your career.

2. What companies are looking for during behavioral interviews

The below are excerpts from Chapter 2 from Technical Behavioral Interview: An Insider’s Guide. Some sections have been cut out and lightly edited for this article. Copyright © 2026 Steve Huynh. Used with permission.

Technical skills alone don’t determine your offer. Otherwise, those who can solve the coding and system design problems would get the same result. Instead, companies use behavioral interviews to answer two critical questions: Do you fit with both the role and the company? And if you do fit, at what level will you be most effective?

Get both right, and you will receive an offer at the appropriate level. Get the fit wrong, and you’ll be rejected regardless of your skills. Get the level wrong, and you’ll be either down-leveled or rejected for being underqualified.

This chapter explains how companies make their assessments of fit and level by analyzing the signals in your stories. Once you understand these dimensions, you’ll pick better stories and signal the right level.

Understanding Fit: Role and Company

The primary consideration for any tech role is whether you have the technical skills to do the job. Companies will assess this mostly through the technical parts of the interview, for example, coding challenges, system design, or whatever technical evaluation matches your role. If you can’t demonstrate the core technical capability, nothing else matters.

But technical skills alone don’t predict success. Companies learned this the hard way by hiring smart people who couldn’t work effectively in their environment. That’s why behavioral interviews focus on two additional types of fit:

Role Fit: Can you handle the specific challenges and working conditions of this position? A backend role at a fast-growing startup requires different capabilities than a backend role at an established enterprise. The technical skills might be similar, but the role demands will be different.

Company Fit: Will you thrive in the environment in which this organization operates? This goes beyond surface-level culture. They are assessing whether your working style, decision-making approach, and values match with how the company gets things done.

How Companies Detect Fit Through Signals

Companies can’t directly ask the question, “Would you fit here?” What candidate would torpedo their chance of success by answering with a “No”? Instead, companies look for signals in your stories that indicate alignment or misalignment.

Role Fit Signals emerge from how you describe handling situations similar to what the role requires:

• If the role requires working with ambiguous requirements, do your stories show comfort with uncertainty?

• If the position involves cross-team coordination, do you show an ability to cope with organizational complexity?

• If the job needs rapid iteration, do your examples show shipping quickly and adjusting based on feedback?

Company Fit Signals come from the choices you made and how you describe them:

• A company that values “bias for action” looks for stories that show you moving quickly despite incomplete information.

• An organization that prizes “customer obsession” wants to hear examples of you going deep to understand user needs.

• A place that emphasizes “radical transparency” seeks stories that show you sharing information openly, even when you’re uncomfortable.

The same story can send different signals to different companies. You spending three weeks perfecting a solution might demonstrate attention to quality at one company but analysis paralysis at another. Moving fast and fixing issues later demonstrates good judgment at a growth startup but recklessness at an established healthcare company.

Common “Mis-Fits”

Even a talented candidate will get rejected sometimes if they are not a good fit. The same behaviors that are positive at one company can signal poor fit at another.

Independence vs. Collaboration: This covers both how you work and how you make decisions. Some companies need people who pick up a problem, run with it, and come back with a solution. Others expect you to bring the team along at every step. These often go together: companies that want you to work solo also tend to want you to make calls on your own, and companies that want collaborative work also want group buy-in on decisions.

If every story you tell involves going off and building something alone, consensus-driven companies will worry you’ll steamroll people or make choices that won’t stick. Flip it around: if every story involves checking with the group before you act, companies that prize individual ownership will wonder whether you can make a decision without a meeting.

Speed vs. Thoroughness: Startups often need rapid experimentation, where you ship MVPs and iterate based on feedback, while companies in healthcare or finance require careful validation before any release. This tension also shows up in how teams think about code quality: some organizations will happily spend extra weeks on clean architecture, while others want a working solution on deadline even if the code needs cleanup later. Whereas stories about methodical testing might bore a startup, your “ship it and fix it” examples could terrify a medical device company.

Excellence vs. Pragmatism: Some organizations value technical excellence and clean architecture above all else. Others need pragmatic solutions that ship on deadline even if imperfect. Focusing on perfect code fails at deadline-driven companies, just as accepting technical debt everywhere fails at companies maintaining critical infrastructure.

Innovation vs. Stability: Some roles require creating new solutions and challenging existing approaches, while others need you to maintain and optimize proven systems. If you say that you’re constantly reinventing established processes, teams that value stability will not consider you a good fit. Conversely, stories that show you only follow existing patterns will disappoint teams that are looking for creative problem-solving

Direct vs. Diplomatic: Some cultures prize radical candor and want you to say exactly what you think. Others value maintaining harmony and face-saving communication. If you are too blunt, you will not fit in well at a relationship-focused company. If you are not direct enough, you will not like working at a company that values “disagree and commit.”

Data vs. Intuition: Some companies require data to justify every decision (”data-driven” cultures), while others trust experienced judgment and move on gut feel. Showing that you make decisions based on instinct does not impress analytical companies, and telling a company that values experienced judgment that you conduct three A/B tests to choose a button color will get you struck off their list.

Specialist vs. Generalist: Large companies often want deep experts who master one domain, while smaller companies need people who are comfortable wearing multiple hats. Know which sort of company you are walking into.

Once you understand fit, you can pick stories that match the company and the role.

The Four Dimensions That Determine Your Level

Companies assess your level through four dimensions that appear in every story you tell. Each dimension reveals different aspects of your capability. Together, they show the company where you operate most effectively.

Scope (Dimension #1)

Scope provides a measure of the number of people on your team and, extending outward as you advance, whose work was affected by your actions. The greater the number affected, the higher your level for this dimension.

Entry Level: Your work affects your own productivity and starts to help other team members. For example, you might improve how you handle assigned tasks or fix issues that were slowing down a few teammates.

Mid Level: Your work affects aspects of the team and shapes how it operates. You might redesign a process that changes a significant part of how your team works or solve problems that affect most of the team’s effectiveness.

Senior Level: Your work directly impacts your entire team and is beginning to influence at least one other team. Perhaps you create solutions that change how your whole team operates and affect workflows in adjacent teams, or you solve problems that require coordination with other groups. You may also start collaborating more closely with product or design partners on your immediate team’s work.

Staff Level: Your work directly impacts at least two teams and is beginning to have an influence on the broader division or organization. Examples of this include developing technical strategies that change how multiple teams make decisions and solving problems that require buy-in across several parts of engineering. Your influence extends beyond engineering into product, design, and program management as you shape solutions that affect how cross-functional partners work.

Principal Level: Your work affects many teams or changes how large parts of the organization operate. Perhaps you have created technical strategies that have influenced how dozens of teams make decisions. Or you have solved problems that cut across a large engineering organization. At this level, your influence regularly extends into business strategy, shaping decisions alongside product, design, program, and business leadership.

Contribution (Dimension #2)

Contribution captures what you did, not what happened around you. It is important to be precise about the line between “I” and “we.” Companies will expect to see evidence of increasing leadership and ownership as you advance in your career.

Entry Level: You execute assigned work and are beginning to take ownership of small pieces. Examples: implementing solutions designed by others; fixing bugs in existing systems; taking full responsibility for well-defined features within larger projects.

Mid Level: You own complete solutions from problem to implementation while also guiding others. Perhaps you have identified issues, designed the approaches, implemented them, and you have verified that they work, and you have helped your teammates understand the reasons for your decisions.

Senior Level: You lead initiatives requiring coordination. You’re expected to make progress even when the requirements are unclear or the path forward is uncertain. Examples of this include driving technical decisions for your team; mentoring others through complex problems; architecting solutions to be implemented by others; and ensuring quality work outcomes for many people.

Staff Level: You lead cross-team initiatives and establish technical direction, often in situations where the right approach isn’t obvious and stakeholders have competing priorities. This could look like defining technical approaches that are adopted by multiple teams, creating systems that enable other teams to solve problems on their own, or driving agreement on complex technical decisions across several teams.

Principal Level: You create organizational capabilities and establish new ways of working. At this level, you’re frequently operating in highly ambiguous environments where you must define the problem before you can solve it. You might define technical standards that guide dozens of teams, build systems that enable others to solve entire classes of problems, or transform how the organization approaches its hardest challenges.

Impact (Dimension #3)

Impact shows what changed for the better as a result of your work. Companies want to see that your work produced results worth the investment. Strong stories put numbers on the impact and connect technical wins to business or user outcomes.

Entry Level: You improve your personal productivity and are starting to help the team work better. Examples include reducing the time you spend on repetitive tasks, fixing issues that were slowing down teammates, or improving the quality of code in the areas you touch. Even simple measures matter at this level: time saved or bugs prevented.

Mid Level: You improve team effectiveness in specific areas and influence team-wide practices. Perhaps you reduced deployment times for specific workflows, eliminated categories of bugs in your domain, or you created tools that have made the team more productive in particular areas. You can quantify these improvements and connect them to broader outcomes like feature velocity or reliability.

Senior Level: You transform how your entire team works and are starting to have an impact beyond your team. For example, you might have introduced new workflows that changed your team’s capabilities. Or perhaps you eliminated major sources of operational problems, or the improvements that you have created have been adopted by adjacent teams. Your impact extends beyond just engineering metrics to product outcomes, user experience, or operational costs.

Staff Level: You improve how multiple teams operate and drive organizational improvements. These sorts of impact come from achievements such as establishing practices that several teams adopt, solving infrastructure problems that were impeding multiple teams, or creating new capabilities that open up new types of work across teams. Your measurable impact can be tied to business metrics like revenue, customer retention, or time-to-market.

Principal Level: You create organizational capabilities and drive strategic changes. Impact at this level could come from establishing technical foundations that dozens of teams use to build upon, solving problems that were blocking major business initiatives, or creating leverage that compounds benefits across the company. Your impact is measured in business outcomes and strategic capability, not just technical improvements.

Difficulty (Dimension #4)

Difficulty reflects the complexity of problems you’ve tackled, the constraints you have faced, and the trade-offs you have managed. Under this category, solving easy problems with big impacts is less impressive than hard problems solved well.

Entry Level: You work on straightforward problems within established patterns. For example, you might face challenges learning new technologies or debugging unfamiliar code, but the path forward becomes clearer once you understand the problem or ask for help.

Mid Level: You work through challenges and obstacles in your work. The problems you tackle have more moving parts and less obvious solutions. These could be competing requirements or having to work through technical complexity you haven’t seen before. Or perhaps you have had to manage dependencies within your team that affected your timeline or figure out solutions when the approach wasn’t immediately obvious.

Senior Level: You manage constraints and make technical decisions with team-level architectural implications. The problems you solve involve multiple interacting systems and competing concerns. You might have to balance needs across multiple stakeholders with different priorities. Maybe you make architectural decisions that affect how your whole team works, or you have to work around technical limitations that require creative solutions, or solve problems that require you to address both technical and business factors.

Staff Level: You manage competing trade-offs across multiple teams while handling problems with significant technical and organizational complexity. Examples of difficulty at staff level include:

• Balancing different technical approaches when teams have genuinely conflicting needs.

• Creating solutions that affect how several teams work together.

• Making architectural decisions that have to work across diverse contexts.

• Getting teams to agree when the technically optimal solution differs for each team.

Principal Level: You handle fundamental trade-offs between competing organizational needs or solve problems where no clear solution exists. The complexity at this level often involves novel problems that lack established patterns or precedents. You might balance technical excellence against delivery speed at organizational scale; work within organizational constraints while maintaining technical integrity; create approaches for entire classes of problems the company hasn’t solved before; or make decisions that affect company strategy and require executive buy-in.

What Each Level Looks Like

Here’s how the same types of accomplishments look across each level. These aren’t templates. They’re meant to help you develop a sense for the difference between a mid-level story and a senior one. Compare adjacent levels and notice what actually changes as you move up and down.

Researching What Companies Really Value

You’ll never have perfect information about what a specific company values, but a little focused research will often reveal surprising insights that most other candidates will miss. The difference between having even partial intelligence and going in blind can be whether or not you emphasize the right things in your stories.

Start With Your Recruiter

Most candidates treat recruiters as gatekeepers to avoid, but if you do this, you will waste your best source of insider information. Recruiters want you to succeed, because their performance is based on the number of accepted offers received by the candidates they put forward. They have prep materials, they know the interviewers’ focus areas, and they understand what they are looking for.

Ask your recruiter directly: “What should I know about this company’s current challenges?” Or “What competencies matter most for this role?” Or “Can you share any interview prep materials?” Many recruiters have documents about interview format, team priorities, or even the specific behavioral competencies they evaluate. The questions that are used as examples in the prep materials have a high likelihood of being asked in the interviews.

Mine Publicly Available Information

When companies repeat certain words when describing job opportunities, they’re telling you what matters. For example, a job posting that mentions “fast-paced” several times signals something different than one emphasizing compliance. Those words are there for a reason.

Where to dig:

• Engineering blogs: How do they describe their wins? What problems do they celebrate solving?

• Tech talks and conferences: What topics do their engineers present? Speed of delivery? Scale? Innovation?

• Open source contributions: What they choose to open source reveals their priorities. If they open source developer tools, this suggests they value community. If they are happy to make internal tools public, this shows transparency.

• Technical documentation: The existence of public API docs or technical guides (and the quality thereof) shows how they support both users and their own teams.

• Status pages and postmortems: Companies that publish detailed postmortems demonstrate that they value learning from failure. A company that shares their incident response processes likely has a strong operational culture.

Even companies without engineering blogs will leave traces. Product release patterns tell you about their development pace. Technology choices show their priorities: newer frameworks suggest a focus on innovation, whereas relying on proven technologies indicates they prefer stability.

Look for Patterns in Discussions

Glassdoor, Blind, and Reddit contain gold buried amongst rubble. Ignore the rubble (e.g., individual rants). Instead, look for patterns across multiple posts. If five different people mention “lots of process” or “no work-life balance” or “amazing learning culture,” that’s a pattern you will want to know about.

Pay attention to what people complain about and what they praise. Complaints about “too many meetings” may suggest the company has a collaborative, consensus-driven culture, or, alternatively, that productivity within the company is inhibited by an excessive number of meetings. Praise for “autonomy” indicates they trust their people to make decisions without checking in. Both types of comments reveal what behaviors the companies will reward.

Talk to Current Employees

If you know someone at the company, ask them directly what behaviors get rewarded and, conversely, what behaviors will cause people to struggle. Skip surface-level queries about culture, and ask specific questions:

• “When someone gets promoted here, what do they do to earn it?”

• “What behaviors get negative feedback?”

• “How does the team make decisions when there’s disagreement?”

• “What surprised you most about working here?”

Current employees will tell you truths the company website never would. Perhaps they’ll tell you that at their company, “customer obsession” really means checking usage data before writing code, or that “ownership” means being available to resolve production issues at two o’clock in the morning.

What You’re Really Looking For

All this research serves one purpose: understanding what stories will resonate at your interview. Think of it as finding the real intersection between your experience and what they care about.

If research reveals they prize speed over perfection, then emphasize stories that tell how you shipped quickly and iterated. If they value technical depth, highlight examples of diving deep to understand root causes. If they care about collaboration, make sure your story focuses on cross-team work rather than solo accomplishments.

The research will also help you decide whether this company is the right place for you. If everything you learn suggests they value the kinds of behaviors you don’t naturally demonstrate or don’t want to develop, then perhaps you don’t need to pursue that particular role.

Putting It All Together

Companies aren’t just evaluating whether you can do the job. They’re also assessing whether you’ll thrive in their specific environment and at what level you’ll be most effective. These two dimensions determine not just whether you will get an offer, but also whether that offer will position you for success.

Understanding fit helps you know which of your experiences will connect most with what the company values. This small company needs someone who ships fast and figures things out alone. That enterprise needs someone who navigates processes and builds consensus. Neither is inherently better than the other. They’re simply different environments that reward different approaches.

Understanding levels helps you position your stories appropriately. The same project can demonstrate entry-level execution, mid-level ownership, or senior-level leadership depending on your actual contribution and how you frame it. Get this wrong and you will either get rejected for overreaching or down-leveled for not properly communicating your capabilities.

The payoff is immediate. You’ll pick better stories, focus on the right details, and make it easier for interviewers to see what you can do. You’ll make better decisions about which roles actually match who you are and what you want to do. The goal isn’t to get any offer. The goal is to get the right offer at the right level at the right company to ensure your success.

Takeaways

Gergely, again. Thanks to Steve for both sharing his learnings, as an interviewer, and for sharing nearly a full chapter from his whole book. The book goes a lot deeper than the above sample chapter. A few of the ones I found helpful:

• High-signal storytelling (Chapter 3): a framework for explaining your work in a way that “sticks” with the interviewer

• 9 competencies with many examples and stories throughout the book: ones like “delivery” (Chapter 6), “earning trust and dealing with conflict” (Chapter 8) and “Strategic leadership and thinking big (Chapter 13)”

• Examples of what interviewers typically see as key signals, yellow flags and red flags

If you would like to have a fresh resource to prepare for behavioural interviews at tech companies, the full book offers far more explanations, tactics and exercises to do so:

Get the full book on Amazon

Steve also writes a newsletter titled A Life Engineered: you can sign up to it here.

It’s helpful to understand how and why companies hire, and what they look for. To us engineers, hiring processes often look illogical from the outside. We’ll ask things like:

• “Why does the interview process not resemble day-to-day work?”

• “I already have open source code I wrote: why does the company need to do a coding interview to confirm what is clear: that I need to code?”

• “Why did I get a rejection, even though I did well on all of the interviews?”

It feels to me that there are similarities between hiring and dating: both parties show up with goals and expectations in their head, which are often not communicated. Sometimes there’s a match; sometimes there is not. This phase of a relationship is often about “selling:” as a candidate on the job market, it’s about selling yourself, and convincing the company that you would be a fit for what they are looking for.

Doing your research on the company is underrated, and not all that many candidates do so, in my observation. When I was a hiring manager at Uber, roughly half of the people who got on the call with me did not do any research about the company, and perhaps 1 out of 10 candidates did any research on the team they interviewed for – when we had public blog posts about our work, on the company blog! So those showing up prepared helped them stand out in the “motivation” dimension, from the get go.

It all starts with being able to pass the “technical” interviews – but it’s a mistake to sleep on the “behavioural parts.” To state the obvious: candidates who do not do well on the technical interview rounds will not get offers. But I’ve personally had to say to several candidates who did great on the technical side of things, but turned out to be misaligned with what we were looking for, as confirmed on the behavioral rounds.

And I do believe you can uplevel in doing better on these behavioral rounds: starting with researching what the company’s culture is like, practicing how to present yourself better, and putting yourself in the shoes of the interviewers, understanding what they are looking for.

I know plenty of software engineers who refuse to do any preparation for interviews, staying “if the company doesn’t want me as I am, they don’t deserve me anyway.” This is a valid strategy, and can work for highly in-demand professionals, the same way as showing up to a first date in sweatpants and slippers can still work out for highly attractive and desirable people. For the rest of us not as incredibly in-demand for a position we’re applying for: it’s probably worth putting in additional effort, in hopes for better outcomes during interviews.

For today’s issue, we’ve dug into your 900+ responses to look for trends in AI tool usage among software engineers and engineering leaders. This article surfaces insights that are less about specific tools, and more about the effect these tools have on tech professionals. We cover:

1. Costs. Unsurprisingly, companies pay for most tool usage, and those responsible for budgets are increasingly nervous that AI-related costs are headed only one way: up.

2. Usage limits. Around 30% of respondents say they have hit limits. Switching tools, upgrading plans, or moving over to API pricing are common responses.

3. Impact on “Builders.” Folks who make larger code changes and do “quality-of-life” work are builders, and they’re also dealing with more AI slop. Some also grapple with a loss of professional identity.

4. AI tools speed up “Shippers.” Engineers who focus more on getting things done are the most positive about AI tools. But they also add tech debt faster and might build the wrong things.

5. “Coasters:” learning faster while generating AI slop. Less adept engineers can uplevel faster with AI, but they generate a lot of “AI slop” while doing so, which frustrates builders.

6. Changing software engineer & engineering manager (EM) roles. Engineers have to orchestrate and context switch more often, while engineering managers can be more hands on. It’s interesting to see the engineer and manager roles becoming more similar.

7. Other impacts on the craft. We’re going from “how” to build to “what” to build, solo devs are seeing improved results, workloads are increasing with AI tools, and more.

We previously published a detailed summary of the survey which focused on AI tooling for software engineers, covering the most-used AI tools, trends, AI agent usage, company size and usage, and tools that engineers love.

1. Costs

Concern about the cost of AI tools is a trend throughout the survey, with around 15% of respondents mentioning it in some way.

Tech companies foot the bill for the majority of spending on AI tools. More respondents say their employers pay for AI coding tools than those who say they pay themselves, and predictably, employers fund more expensive packages than what individuals buy personally.

Companies commonly pay for “max” plans with the likes of Claude Code, Cursor, and Codex (around $100-200/month per engineer), although some companies’ budgets only stretch to $20/month per engineer – around the price point of GitHub Copilot, and the cheapest Claude or ChatGPT subscriptions.

The most-mentioned AI tool spending patterns:

• When companies pay: ~$200/month plans. Many have enterprise subscriptions, sometimes with subsidies and vendor lock-in. Some companies allow usage-based coverage on top of monthly plans.

• When personally paying for tools: ~$20/month or free tiers. This can stack up across different tools. Around 5% of respondents have separate work and personal subscriptions, and free tier usage is widespread for personal use.

For now, companies seem to be in the experimentation phase with AI tools, and several respondents say that they believe their companies have unsustainable AI-tooling budgets. This is likely because businesses are figuring out the best way of leveraging the tools, and the message to engineers at such places is to not worry about price and usage while that unfolds. A CTO at a small, US-based company shares:

“Right now, we’re not sweating the costs because we’re trying to evolve best practices for the tools, but that has resulted in some devs really blowing through budget, so we may start instituting caps on spending.”

Breaking the budget

At small and mid-sized companies, leadership teams seem more comfortable about going over budget, than engineers running out of budget. There are more accounts from C-level folks and founders about racking up large bills than there are from engineers. A CPTO (Chief Product and Technology Officer) at a mid-sized company:

“I ran up several monthly bills of $600 with Cursor. We have the dev team subscribed to ~$100/month plans. We’re now in the process of moving the rest of the team to Claude Code, as we can get more resources for around $100/month in cost.”

Top spenders can be allocated higher budgets. A number of tech businesses have separate, larger budgets for their heaviest AI users. A senior C++ engineer working in the video game industry says:

“I’ve become my team’s AI champion. In theory, my limits are higher than normal, but I keep myself limited to what others can use, so I can show them useful things they can do.”

UK and EU companies worry more about budgets than US-based ones. Most responses which mention finance teams pushing back against spending even $30-50/month per engineer on AI tools are based in the UK and EU. One amusing example is a 10-person, seed-stage startup where the CEO questioned why they were paying as much as £25/month per engineer for one of the cheapest AI tools around.

In general, it feels like European companies want to see clear value-add in order to justify an increase in tooling spend, whereas US companies are more comfortable with investing first and measuring impact later. At present, the impact of these tools is hard to quantify.

A niche approach is that of AI teams educating devs to use cheaper models. Some European companies go as far as offering education to new joiners on using cheaper models. From an AI Enablement Lead at a 1,000+ person, digital transformation company:

“Within our organisation, we’ve had incidents where our Claude users have overshot their limits. We’re now attempting to educate devs in knowing the difference between different models (knowing when to use Claude Sonnet versus Claude Opus).”

Cost trajectory worries

The cost trajectory of AI tools is generally considered unsustainable in our survey. Devs using the tools heavily tend to hit usage limits, and their employers then have to pay more. At places with API-based pricing, usage is increasing. Those in leadership positions who are responsible for budgets are generally concerned about the direction of costs.

Subsidies are keeping things at bay – for now. A common enough pattern in our survey is of heavily-subsidized enterprise plans that come with vendor lock-in. Several responses raise concerns about what will happen when the subsidies run dry. Experienced engineering leaders recall that cloud providers also played the same game of subsidizing for a few years, then raising prices when a customer was fully “locked in.”

The AI hype cycle is dampening awkward conversations about budgets at some places. A principal engineer at a Fintech tells us:

“The AI hype has created a special, generous budget for AI tools, and there’s no effective budget – yet!”

But some finance teams are getting grumpy. A CTO at a sports-tech company says:

“It’s hard to keep our CFO supportive about investing in these tools because the productivity benefits have proven difficult to conclusively prove. The point that resonated the most was the loss of value when people hit daily limits: having to stop work immediately! Surprisingly, our CFO is still pushing back, despite having experience of getting a lot of value through their own AI usage with their spreadsheets.”

Most survey respondents think the price of AI tools will have to rise. If that happens, it would cause problems at several companies – particularly those in Europe:

“I cannot see how the spend on AI tools is fiscally sustainable in its current form; Max 100 with Claude Code is $100 a month. A single small task powered by Kimi K2.5 using OpenCode is $5, mostly in input cost. If we assume that the third party inference providers are doing so at a sustainable price, the much more expensive Opus model cannot be sustainable, never mind profitable at these plan costs.” — Founder at a seed-stage company, Europe.

“From the economic perspective, at some point, these companies will need more funding or profit, I’m curious how much it costs them to have a proper agent, and still become profitable. It feels slow when you run out of credits when working on repetitive tasks.” — Principal Software Engineer at a seed-stage company, Europe

2. Usage limits

Another major trend in our survey results is the topic of usage limits:

• Hitting limits: ~30% of respondents. Running out of tokens or hitting reset limits is frustrating and disruptive, especially when you’re working on a task or are in a flow state. The majority of respondents who complain about hitting limits are on cheaper plans (typically $20/month.) But this issue is also mentioned about higher subscription levels.

• Under the limit: ~20% of respondents. Avoiding usage caps generally correlates with being on more expensive plans with higher limits, or in roles with enough non-coding work for it not to matter, or when devs do enough work “manually” for AI usage to not be an issue.

Why users of AI tools hit limits

Common reasons cited in the survey:

Being a new AI user or a power user. These are two distinct groups, but an engineering manager at a mid-sized company in Canada says that each one similarly blows through token limits for different reasons:

“We’re mindful of trying to manage costs by setting AI spend limits across the org. We have two subsets of users at odds with each other:

1. Individuals who are still learning and blow through their credits at an inordinate rate, forcing us to keep limits low.

2. Power users who hit the limit through regular use and apply pressure to raise the limit.

It’s a tough balance.”

Using Opus for all work. A few engineers mention being careful about how they use Opus because it previously ate up their token budgets. Here’s a software engineer at a mid-sized company in Europe:

“I made the mistake of using Opus in the past and burning through budgets quickly. Now, my routine is to start in ‘plan’ mode with Opus. I then paste the acceptance criteria and description of the issue and let the plan mode figure it out. I then switch to Composer or Sonnet and have the agent take over from there.”

Mistakes that eat up tokens are easy to make. These include starting on work or a problem from the wrong end, using AI directly for a task rather than opting for a simple script, trying some new tool or technique that ends up consuming tokens (OpenClaw and Ralph Loops are cited), and others.

What happens when the limit is hit?

Hitting the limit with an AI tool is inconvenient and happens to many developers, who take a variety of next steps:

Switch the model or tool. Around a quarter of respondents who hit limits mentioned switching. From a software engineer at working at Atlassian:

“In my company, for Cursor and Windsurf we have monthly limits. Our internal coding tool (called codelassian) also has daily prompt and hourly token limits. When I hit a limit in one tool, I switch to the other.”

Upgrade to a pricier plan. When it’s an option, this is a no-brainer at most places, especially as the alternative would be devs twiddling their thumbs waiting for the limit to be reset. A senior engineering manager at a mid-sized company says:

“In my team, we are regularly hitting session limits with Claude. We upgraded some teammates to the Max 20x plan – and on this plan we have not been hitting limits, so far.”

Adopt API-based pricing. This is the easiest way to keep working without abandoning a task you’re knees-deep in. A senior engineer at a large company says:

“The company provides both the Claude and Copilot corporate offerings. When the limits are reached, I tend to use API keys that my teammates give me.”

3. Impact on “builders”

We identified three different types of professional in the survey:

• Builders: those who care about quality, good architecture, following good coding practices, and who talk about the craft of software engineering, etc.

• Shippers: those who primarily focus on outcomes for a product, features, testing, and experimenting with users. A fair number of leaders, managers, and engineers who were more hands-off with coding before AI tools are in this category, as are product engineers.

• Coasters: engineers who are not considered particularly good or great engineers, but they get the work done. They often do this without much taste or concern for quality, and seem to be mostly coasting along and doing what they’re told.

The overall consensus in our survey results is that AI will amplify and multiply tendencies and patterns that existed before, and the impact of the tools varies accordingly among users. Let’s start with the impact we’ve observed upon builders in the responses:

Builders say they get value from AI tools in the following areas:

Larger code changes. Builders generally find AI helpful for work like:

• Refactoring

• Migrations

• Improving test coverage

• Carrying out large codebase changes

All these are changes that are laborious, but not very challenging technically. They also require experience in knowing what you want to do and how to do it.

Accomplishing “quality of life” tasks. Builders mention that with AI tools, they get to fix and improve things like nagging bugs that otherwise wouldn’t be “worth it” in time invested, but the barrier to entry is lower with AI.

A good example of this is in last week’s podcast with David Heinemeier Hansson (DHH), the creator of Ruby on Rails, in which he revealed how one of their engineers optimized P1 – the fastest 1% of web requests:

“One of our most agent accelerated people asked: “What about P1? What about the floor? Can we fix the floor?” He found that the floor [of request speed] was 4 milliseconds.

Well, 4 milliseconds can add up if you have a bunch of fast requests. So, he just said: “We’re going to optimize P1. The fastest 1% of requests, we’re going to make them even faster.” He took it from 4 milliseconds to less than half a millisecond. He did this P1 project over a couple of days, like a side project.

He had an intuition that there was something here. He let agents run with it. The work ended up being 12 pull requests, and about 2,500 lines of code changed.

This is exactly why the explosion of the pie suddenly lets us look at problems we would never have contemplated looking at before.”

Typing is no longer a bottleneck. Some builders report falling even more in love with coding with the help of AI and agents, since physically typing out code is no longer a bottleneck for them. They enjoy being able to prompt. From one “builder”:

“For someone who loves to build – but also values code quality, performance, reliability, and security – I ship a lot more quality code faster, if for no other reason than because the AI can read and write 100x faster than me. I get to stay at the conceptual level of shipping a product, and I can dive into debugging with the agent as needed. But if the agent has a good handle on the situation I can give it as much of the tedious parts as I wish.” – Staff Engineer, at a large tech company, US

The negative sides of AI tools, as experienced by builders:

• More AI slop. Builders seem to be the most overwhelmed and derailed by reviewing a lot more AI-generated code. They can get frustrated with low-quality code shipped by colleagues which could be categorized as “AI slop.”

• More debugging. AI-generated code introduces bugs and issues, and builders tend to spend the most time debugging and fixing those issues.

• Identity loss. Some builder-types report a sense of identity loss and even some grief. Much of this relates to no longer doing hands-on coding because they cannot justify it, since AI agents generate pretty decent code faster than someone can type it.

4. AI tools speed up “Shippers”

The “shipper” archetype thrives when they get things to production quickly. This group is by far the most enthusiastic about AI tools in survey responses. They are also the ones who praise – or hype up – the tools because of their personal experiences of shipping much faster with them.

The biggest upsides mentioned by shippers:

Read more

Martin and Kent go back decades, and Martin jokes that his career is “mostly about writing down Kent Beck’s ideas.” They first collaborated in the 1990s, and each has published influential books – ‘Extreme Programming Explained’ and ‘Test-Driven Development’ by Kent, and ‘Refactoring’ and ‘Patterns of Enterprise Application Architecture’ by Martin.

At the Pragmatic Summit, they each shared a wealth of hard-earned learnings and decades-worth of perspective, along with a healthy dose of skepticism. Needless to say, the conversation did not disappoint, and this article summarizes what we discussed in their own words. You can also check out the full recording.

We cover:

1. Technology shifts similar to AI. The arrival of the microprocessor, introduction of object-oriented languages, the Internet, and agile software development principles were all major changes – but one big difference was that it took time for these technologies to be adopted. Not so with AI.

2. Agile and AI similarities. With Agile, company incentives were often misaligned, “snake oil” vendors were everywhere, and a “mid pack” of developers who resisted the change saw their career prospects hit. These trends look likely to repeat with AI.

3. What’s happening inside companies. There’s some confusion – and even panic – at large companies, while AI tools don’t work nearly as well on large and complex codebases as on greenfield projects. Also, a “re-soloing” of software development is inbound.

4. Avoiding burnout with AI agents. Set and maintain boundaries, and pay attention. Martin suggests to catch when you start producing “negative value”: that’s when to take a break.

5. Unhealthy performance metrics. Companies are starting to measure things like frequency of pull requests – when they should be looking to quantify outcomes and results.

6. Lower quality on purpose? It seems every business is optimizing for speed with AI, but quality can get dropped. Also: building features is more obvious with AI, than investing in “futures.”

7. Test-Driven Development (TDD): tests no longer optional? Kent pioneered TDD, and today it’s more relevant than ever for working with AI.

8. Thriving in an AI-native industry. Focus on working with agents to express your craft, try to get more enjoyment in understanding your domain, and take on more ambitious work.

Before we start, a programming note: this week, there will be no The Pulse on Thursday — I’ll be attending AI Engineer Europe in London on Thursday and Friday, including doing a fireside chat, and hosting one with Linear CTO Tuomas Artman.

1. Technology shifts similar to AI

Do you recall a tech change as similarly promising and unpredictable as AI?

Martin: “Nothing has hit with the magnitude of AI. This is a whole size different from anything we’ve faced before. On a smaller scale, we were very much involved in the growth of object oriented languages, which scared a lot of people. It didn’t scare us so much because we were part of it.

Looking back, the internet had a huge impact on us all, and of course, Agile software development, too. Agile had a very big impact on a lot of organizations: you could tell by how hard they resisted it. We had to persuade people of the importance of these technological changes; yes, even the internet! It may sound surprising but there were people who didn’t think it was important.

The thing about AI is that today there is no argument about how important it is.”

Kent: “The other analogy I have is the introduction of the microprocessor. Before that, computers were big boxes; you couldn’t move them around. If you wanted another computer, you’d mortgage your house for it. Having a computer was a big deal.

I was a kid in Silicon Valley with my dad as a programmer when the Intel 4004 hit the market [in 1971]. We went: “Wait a minute, that chip is a computer? Oh my goodness!” The possibilities of computing suddenly expanded thanks to it. If you could figure out how to write software on this chip and figure out how to design hardware around this thing, you could suddenly do things you hadn’t even imagined.

And so I think part of AI is this expansion of imagination. I’m writing projects that are ridiculously ambitious: I’m working on a persistent Smalltalk. I’m writing library-quality code for Rust.”

Balancing skepticism and curiosity

What was the feeling in the industry during those revolutions, and the differences between professionals who thrived back then and those who didn’t?

Martin: “There was a mix of people chasing the hype and those saying, “this new thing is nothing special.” I think you’ve always got to have that balance of skepticism and curiosity, and to be selective about it. I mean, I have been completely skeptical about some big changes: Blockchain was one I was extremely skeptical about.

My skepticism is well-rooted because I’ve seen so much “snake oil” over the years. In fact, my skepticism has to be absolute and total, which means I have to be skeptical about my skepticism! To be that skeptical also requires curiosity: you’ve got to be curious enough to say “how do I probe in order to detect signs of something useful?”

You also need to be aware that your early interactions may not actually be a true signal. When I started playing around with AI, it was with GitHub Copilot a year and a half ago. I was pretty unimpressed; it would give you something wonderful, but most of the time it gave you such garbage that you would just delete it right away. If that had been my only impression of AI, I would’ve immediately flipped the “bozo bit” on it, like I did with blockchain.”

Kent: “Here’s the thing, the capabilities of AI can change week to week. I’ll try something with Gemini one week and it fails miserably. Then Claude Code works pretty well, and then it doesn’t. And then I try Gemini for the same thing and it works, when it hadn’t worked last week!

People want an answer, but the answer’s always changing. In this environment, you can’t possibly have the answer. That’s the bad news, but the good news is that nobody else has the answer either. So, you’re just as smart as everybody else because we’re all equally ignorant.”

2. Agile and AI similarities

In 2001, the ‘Agile Manifesto’ came out, of which you were both co-authors. I think many companies are expecting the same thing with AI as Agile promised: better, faster, cheaper software. But how did Agile adoption really play out?

Kent: “It turns out people don’t want faster, cheaper, better! Inside some companies, the incentives are misaligned with actually achieving that. And so as geeks trying to achieve these improvements and saying: “it’s 40% better, 12% cheaper and less fattening,” people will punish you if that doesn’t align with their incentives inside organizations.

In the ideal organization, everybody would care about the same things, but that’s just not the way it works! So, if AI is coming along to promise the same things, we’re going to see the same reaction as before.”

Martin: “An obvious difference is the sheer magnitude and speed there is with AI. Also, I think there will be a big difference between people who use it well and people who use it badly. The trick is figuring out how to use it well and putting the effort in to learn. There will be a big distinction between those two groups.

But I suspect there will still be some similarities with Agile. The core notions behind Agile and extreme programming are solid and good, but a huge snake-oil industry appeared around it – the “Agile industrial complex”, as I refer to it. This is also happening with AI right now, and it’s often hard to see the difference between snake oil and the real stuff.”

AI as an amplifier

Kent: “AI is an amplifier. If you’re young and learning quickly, AI can amplify your learning. I personally think this is the golden age of the junior programmer. I get people coming to me all the time saying things like “my son started his second year in CS and wants to go into something more commercial like art history.” And I’d say, “this is like if you’re a carpenter and they just introduced the circular saw and you think, ‘oh, well, carpentry is over. Anybody can build a house now.’ Well, no! Now, you have more powerful tools. You have less of the crummy work to do.

I think that young people are going to learn faster, and experienced folks who are working effectively are going to work quicker and more effectively.”

Developers stuck in the middle

Kent: My concern is that there’s a “middle” of people who got into programming as a way to make money. If we look back at the Dotcom crash, there was a “mid pack” of such people who ended up going into real estate, more or less. But today, I don’t know where that “middle” will go, and it’s also much bigger now than 25 years ago.”

Martin: “But that middle has also been “flushed out” to some degree by retrenchment in the software industry at the end of the zero interest rate period. So, that’s an interesting difference because we’ve had these things occurring at once: the AI boom, and the economic headwinds of the past 2-3 years.

This is an interesting mix that wasn’t present in the ‘90s with the Dotcom Boom. Back then, it was pretty much all a solid boom.”

Return of “let’s get rid of programmers!”

Kent: “Another interesting confluence of factors is the periodic, “we can get rid of all the programmers, woo-hoo” trend, which started with Cobol in the 1970s. With Cobol, business analysts were supposedly going to be able to write the programs, and the logic was that we wouldn’t need programmers anymore. That comes back repeatedly.

Agile, however, was definitely not a “let’s get rid of programmers” trend. With Agile, we wanted programmers to be more effective in their jobs. And since we started it, and were programmers, we were able to push that agenda pretty effectively.

However, today the “get rid of programmers” trend is repeating. As programmers, it behooves us to think about why they keep wanting to get rid of us. Some of that’s about us as programmers, and some of it not. Still, we should think about why people periodically want to axe us. In the end, this trend amps up the fear factor that everybody’s experiencing.”

“Re-soloing” of programming

Kent: “A big trend is the “re-soloing” [reduced in-person collaboration] of programming.

A big part of extreme programming (XP) was creating a safe social environment for basically antisocial people. On an XP team, people are talking to each other for hours a day, and are happy to do so because it’s set up to be a positive experience.

Now, I see programmers saying, “I’ve got six agents, so really I’m managing a team.” No, you’re not: you’re using six tools at once, which is fine, but it’s very different from having a conversation with somebody who sees things slightly differently, or has a different energy level from you on the day.

We used to have programmers in individual offices with doors, and you’d shut the door and slide the pizza underneath. That was easy to manage, but then along came this messy, social, complicated, chaotic process of software development, which just happened to produce really good results.

But now, instead of 50 people on my team, I can have five and they don’t have to talk to each other, and each can have 10 agents. Is that the same? No, it’s not.”

More effective two-pizza teams & the future of pairing

Martin: “Are we seeing two-pizza teams [of 5-10 people] becoming one-pizza teams because agents don’t eat pizza, or do we see two-pizza teams staying and becoming much more effective and capable? My bet is on more effective two-pizza teams.

We’re beginning to see some interesting feedback in terms of pair programming. With pair programming, is it one human and the genie (AI) programming, or is it two humans and one genie? If it’s two of us, perhaps we can control the genie a bit better, and we also have interaction.

I’ll be very interested in reports of people trying to control genies in pairs, possibly even beyond pairs. There’s also the whole ‘mob programming’ thing, and how that will go with genies. I don’t necessarily think that one person and many genies is the right answer.”

Kent: “My experience of pairing with two humans, plus one or more genies, has been very positive. And the fact the AI is slow is really nice. Every time models come out and are faster, I’m like, “Oh, there’s less time to talk.” When the AI goes away for three minutes, we can talk about our philosophy of naming, or how we express conditionals, or about what we should be doing next. But if it pops back in 15 seconds, you don’t have time for that conversation.”

4. Avoiding burnout with AI agents

Do you find yourself getting close to burnout, especially when spinning up multiple threads? Do you have strategies for managing the mental impact?

Read more

And now, inference engineering is becoming more widespread, too, as open LLM models grow more capable. This is because with closed models, inference engineering is done only by the AI engineers who build the model, whose number might add up to a few thousand globally. In contrast, with the open models which tech companies are adopting, it’s possible to tweak them to perform better at inference. For example, Cursor built its new Composer 2.0 model on top of the open Kimi 2.5 model, and successfully used plenty of inference engineering approaches to make it even faster.

So, based on this industry-wide prevalence and the related need for superior technical performance, it’s worth understanding a bit about what inference engineering actually is, and some interesting approaches worth knowing about, as a software engineer.

For some answers, I turned to Philip Kiely, a software engineer who has been working for four years at the inference startup, Baseten. With his hard-earned experience, Philip has written an excellent, in-depth book about precisely this topic, “Inference Engineering.”

In today’s issue, we cover:

1. Setting the stage: why is inference so important? More capable, widespread, open models are driving demand for inference engineering.

2. What is inference? As the phase that comes after training a model, the inference layer introduces new engineering challenges like batching, caching, and quantization.

3. When is inference engineering needed? Investing in this area is typically worth it when your product and usage scales up, and there are product requirements which a current, off-the-shelf solution lacks.

4. What hardware does inference use? Datacenter GPUs are the most common, while on-premises, air-gapped GPUs are also employed.

5. What software does inference use? Commonly-used software includes NVIDIA’s CUDA and Dynamo, as well as hardware-agnostic projects like PyTorch, vLLM, and others, which are growing in popularity.

6. What infrastructure does inference need? Autoscaling is a baseline requirement. Kubernetes is a popular choice for autoscaling inside a cluster, while multi-cloud inference might be necessary for high-scale use cases.

7. Five approaches to make inference faster. Quantization (reducing the numerical precision of a model’s weights), speculative decoding (taking advantage of spare compute to generate “draft tokens”), caching, parallelism (tensor parallelism and expert parallelism) and disaggregation (separating the prefill and decode phases to run on separate workers, not the same GPU).

This deepdive uses a few abbreviations and concepts that are everyday lingo for inference engineers, but maybe are not for those less versed in the domain:

• CUDA: Compute Unified Device Architecture. NVIDIA’s proprietary API to program NVIDIA GPUs for high-performance computing, including LLM-related use cases.

• TTFT: time to first token. Think of this as the “time to process the prompt.” This metric determines the perceived responsiveness of models and GenAI systems.

• TPS: tokens per second. Akin to a model’s “typing speed.”

• ITL: intertoken latency. The time between generating one token and the next.

• KV cache: key-value cache. The cached results of the attention algorithm, reused between requests to speed up inference. We cover more on KV cache in the Scaling ChatGPT deepdive.

• Prefill / decode: the two phases of inference. Prefill is when the model takes the full input and processes tokens, outputting the KV cache. Decode is the phase in which the model generates one token at a time.

• MoE: Mixture of Experts. An architecture that enables models to be pretrained with far less compute. More details on this approach.

Below is an introduction to inference adapted from Philip’s book, “Inference Engineering,” which is free to download as an e-book. Physical copies are currently sold out, but Philip is printing more as fast as possible.

My usual disclaimer: as with all my recommendations, I was not paid to mention this book, and no links in this article are affiliates. See my ethics statement for more.

With that, it’s over to Philip:

1. Setting the stage: why is inference so important?

Inference is the most valuable category in the AI industry, but inference engineering, on the other hand, is still in its infancy. In their work, inference engineers work across the stack from CUDA to Kubernetes in pursuit of faster, less expensive, and more reliable serving of generative AI models in production.

When ChatGPT launched in late 2022, there were perhaps a few hundred inference engineers in the world, and they didn’t call themselves that. These specialists mostly worked at frontier labs like OpenAI, Midjourney, and Anthropic, or at big tech companies like Google and NVIDIA.

Back then, it looked like this might be the way of the AI industry: that training generative AI models would be so hard and expensive that only a handful of companies would develop closed models and thereby require inference engineering for production serving. In that alternate future, the rest of the world would be mere consumers of AI via APIs, renting intelligence a token at a time.

Three years later, it turns out that training generative AI models is indeed both hard and expensive – but it’s not so hard and expensive to be limited to a handful of players. Instead, a proliferation of open models – more than two million and counting on Hugging Face (the “GitHub for AI”) – means that today every engineer can now deploy their own intelligence to power AI products.

Research labs around the world, from OpenAI and NVIDIA Nemotron in America, to Mistral AI and Black Forest Labs in Europe, to Alibaba Qwen, DeepSeek AI, Z AI, and Moonshot AI in China, regularly release open models of all modalities.

Despite closed models getting smarter and cheaper, the movement into open models is accelerating, which differ by the availability of their weights:

• Closed model: A proprietary model whose weights are unavailable to the public, like GPT-5 and Claude Sonnet.

• Open model: A model whose weights are publicly available, like Llama or DeepSeek, and which is usually released under the MIT license, or a similar permissive license (some models restrict commercial use, so always double-check license terms).

Before December 2024, there was a meaningful gap in intelligence between closed and open models, but when DeepSeek V3 and R1 were released, that gap disappeared. Note from Gergely: we previously covered how DeepSeek’s release rocked the AI industry.

Today, new closed models are matched by open models within months if not weeks, and occasionally, open models like Kimi K2 Thinking even exceed closed models’ capabilities for brief periods.

Despite the fact that open models are constantly chasing closed models on benchmarks, they nonetheless change the equation for AI product builders. And as both types improve, closed and open models cross capability thresholds.

In 2022, it was impossible to build the kinds of AI-native products that define the industry today. But over time, closed models got smarter and new categories like customer service voice agents and AI-powered IDEs became possible. The early models were slow, expensive, and unreliable, but the capabilities were there and AI engineers began building companies around them.

As open models crossed the same capability thresholds, these folks began using them to replace closed models. Many also began fine-tuning open models to cross capability thresholds faster, and even exceed closed model quality in their specific product and domain.

Switching to open models means the opportunity to use inference engineering to make the models powering AI products better in new ways:

• Latency: Closed model APIs are built for throughput, but open models can be optimized for real-time applications.

• Availability: While APIs for GPT and Claude are stuck at two nines of uptime, it’s possible to achieve four nines or better with dedicated deployments of open models.

• Cost: Open models are often at least 80 percent less expensive at scale.

So, whereas three years ago it looked like inference engineering was a niche field, the fact is that today, every company aiming to build truly differentiated and competitive AI products needs an inference strategy.

AI-native startups like Cursor, Clay, Gamma, and Mercor are redefining hypergrowth by building products that rely on open and in-house models. Leading digital native companies like Notion and Superhuman succeed by deeply integrating AI capabilities into their category-defining products.

Elsewhere, a new generation of blended research and engineering teams – World Labs, Writer, Mirage, and dozens more – are building businesses by training and productizing their own foundation models.

Adoption is even strong in enterprise and regulated industries, which historically were slow to adopt new technologies. Companies like OpenEvidence, Abridge, and Ambience are making generative AI ubiquitous in healthcare, while at the world’s largest companies, AI initiatives are moving past the pilot stage into massive user adoption. Market-wide demand for inference means that everyone from developers to executives has the opportunity to learn inference engineering and use it to advance their career and business.

I’ve been incredibly fortunate to have a front-row seat in the fastest-moving market in history over the last four years at Baseten, where we power mission-critical inference for the best AI products, including every company listed in the previous paragraphs.

The good news is that you are early. There are still relatively few professionals working on inference, and newcomers can become experts quickly. Also, the potential and impact of inference is becoming ever clearer, but the domain is still in its infancy. That means there are enormous opportunities to solve novel, interesting, and deeply technical problems at all levels of the stack.

2. What is inference?

Inference is the second phase of a generative AI model’s lifecycle:

• Training: The process of learning model weights from data.

• Inference: Serving generative AI models in production.

During the past decade’s machine learning (ML) boom, hundreds of thousands of data scientists and ML engineers became familiar with the full lifecycle of training and inference for ML models.

Inference for classic ML models is relatively straightforward. In the early days of Baseten, we ran inference for models built with tools like XGBoost on lightweight CPUs with a simple software stack.

In contrast, inference for generative AI models is complex. You can’t simply take model weights, get some GPUs, and expect inference to be fast and reliable enough for large-scale production use. Doing inference well requires three layers:

• Runtime: Optimizing the performance of a single model on a single GPU-backed instance.

• Infrastructure: Scaling across clusters, regions, and clouds without creating silos, while maintaining excellent uptime.

• Tooling: Providing engineers working on inference with the right level of abstraction to balance control with productivity.

These three layers must work together to create a system that can handle mission-critical inference at scale.

The runtime layer is responsible for ensuring an individual model running on a GPU (or across several GPUs in a single instance) runs as performantly and efficiently as possible. This layer depends on a sophisticated software stack, from CUDA, to PyTorch, to inference engines like vLLM, SGLang, and TensorRT-LLM. Low-level optimization is important, with kernels like FlashAttention delivering significant performance gains.

The runtime layer relies on a number of model performance techniques that apply new research to the challenges of inference on generative AI models:

• Batching: Run incoming requests in parallel, weaving them together on a token-by-token basis to increase throughput.

• Caching: Reuse the KV cache – the cached results of the attention algorithm – between requests that share prefixes.

• Quantization: Lower the precision of select pieces of the model to access more compute and reduce memory burden.

• Speculation: Generate and validate draft tokens to produce more than one token per forward pass during decode.

• Parallelism: Efficiently leverage more than one GPU to accelerate large models without introducing new bottlenecks.

• Disaggregation: Separate the two phases of LLM inference, prefill and decode, onto independently scaling workers.

These model performance techniques are used for all modalities and not just LLMs, such as vision language models, embedding models, automatic speech recognition, speech synthesis, image generation, and video generation, which extend the capabilities of AI systems and require their own inference optimizations. But these runtime optimizations are not enough: no matter how performant a single instance of a model server is, it will eventually receive more traffic than it can handle. This is not a CUDA problem or a PyTorch problem, it’s a systems problem that needs to be solved at the infrastructure layer.

The nature of infrastructure problems changes at each level of scale. At first, the problems are around autoscaling: knowing when to add and remove replicas, and figuring out how to do so quickly.

Past a certain scale – generally a few hundred GPUs – infrastructure problems are defined by capacity. To get access to enough GPUs, inference engineers begin spreading workloads across multiple regions and cloud providers. This quickly leads to silos, where models in one cluster may be starved for resources while other clusters have unused capacity. The final level of scale in infrastructure is a global system that treats all available resources as a single unified pool of compute.

Thoughtful multi-cloud infrastructure also improves reliability, protecting against downtime in any individual region or cloud provider. And for global applications, running inference near to end users improves end-to-end latency.

Once these runtime and infrastructure capabilities are built, they need to be presented at the appropriate level of abstraction. Inference providers like Baseten and internal teams building inference need to consider what tooling and developer experience to provide as the critical third layer in a complete inference platform.

Of course, developer experience is subjective. For inference, one extreme is the black box: give a platform model weights, and get back an API. At the other extreme is providing only basic constructs for compute, network, disk, and so forth.

The right developer experience is somewhere in the middle, where inference engineers have enough control to run mission-critical inference confidently, and enough abstraction to work productively.

This article – which is an excerpt of Inference Engineering – presents an overview of the technologies and techniques that power inference across all three layers of runtime, infrastructure, and tooling.

3. When is inference engineering needed?

Inference engineering adds speed and scale to AI products by optimizing production serving of generative models. Optimization means identifying the best solution from a range of options.

Before optimizing model performance and building robust infrastructure, you need to know what “best” means for your product; many performance improvements come from making tradeoffs in latency, throughput, and quality. In practice, optimization is often about finding the right balance, rather than maximizing a single factor.

For example, NFL players are big, fast, and strong. But they’re not as big as sumo wrestlers, fast as Olympic sprinters, or strong as champion powerlifters. Their bodies and skills are optimized to fulfill the specific demands of their position over the course of a full season.

Similarly, your inference system must be optimized to fulfill the specific demands of your model, product, and traffic. The more constraints you can introduce, the better the outcomes that can be achieved.

You should know:

• Model requirements: Which model(s) do you need to run inference on?

• Application interface: How will inputs be delivered to the model, and how is the output expected to be formatted?

• Latency budget: How fast does your product need to respond to a user action, end-to-end?

• Unit economics: How much sense does it make to spend on a per-request, per-user, or per-month basis?

• Usage patterns: How many concurrent users are you serving, and is there any pattern to their usage (e.g., more activity during business hours)?

Early in building an AI product, the answers to these questions may not be clear. At this point, it’s often better to use off-the-shelf APIs whenever possible, rather than investing in dedicated inference. But as a product scales, the requirements become clear and inference engineering becomes a worthwhile pursuit.

4. What hardware does inference use?

Inference engineering relies on accelerators: powerful hardware designed to load terabytes of data and perform trillions of operations per second.

The most common type of accelerator for inference is the GPU, and the market leader in GPUs for inference is NVIDIA. My book focuses on inference engineering for NVIDIA GPUs in the datacenter, and also covers other vendors of datacenter accelerators and local inference.

Across vendors, there are three types of GPUs on the market:

• Datacenter GPUs: Racked servers with interconnected high-performance GPUs. Example: NVIDIA B200.

• Workstation GPUs: Individual desktop GPUs for professional workflows. Example: NVIDIA RTX Pro 6000.

• Personal computing GPUs: Individual desktop GPUs for everyday use. Example: NVIDIA GeForce RTX 5090.

Inference at scale uses datacenter GPUs mounted on racks: refrigerator-sized chassis with standardized power, networking, and cooling.

Datacenter GPUs like the NVIDIA B200 offer the highest individual performance, and more importantly, include high-bandwidth GPU-to-GPU interconnects, are installed in highly standardized configurations, and are available by the millions in datacenters worldwide.

I doubt there is a B200 GPU running under your desk, but if there is, please send me a picture! Instead, inference on datacenter GPUs runs in one of three modes:

• Cloud: GPUs are rented in someone else’s datacenter, usually hyperscalers like AWS and GCP, or neoclouds like CoreWeave and Nebius.

• On-premises: GPUs are purchased and installed in a datacenter that you control directly.

• Air-gapped: GPUs are installed on-prem and you need to physically access the GPUs to run inference.

Most inference engineers use cloud GPUs. Large enterprises and governments run on-prem and air-gapped deployments, but cloud-based GPUs offer the flexibility and access that fast-growing AI products need to scale.

Even with constraints, navigating the hardware landscape is complex. From variations between cloud providers, to NVIDIA’s personal naming conventions, there are many nuances in selecting the right accelerator.

5. What software does inference use?

NVIDIA’s market dominance in the inference space is in no small part due to the robust, mature software ecosystem around its hardware. Hardware iteration cycles are slow. Best-in-class hardware companies like Apple and NVIDIA release new architectures and generations at most annually, with two-year release cycles being more common. But software iteration is fast. Often, to run a newly released open model on day zero, you need to install a nightly build or other pre-release version of each software dependency just to get support for the new model.

Software’s fast iteration cycle and lower barrier to entry dramatically expands the landscape of inference engineering. There are countless companies building software at various levels of the inference stack, in contrast to hardware, which centers on NVIDIA and a few competitors.

For inference engineers, these are some of the key software players:

• NVIDIA: Invests heavily in its own sometimes-proprietary software ecosystem, from CUDA up to Dynamo.

• Hugging Face: Maintains a model registry for all open models plus transformers (models built on the transformer architecture), and diffusers (models built on the diffusion-based generative ones).

• The Linux Foundation: Maintains hardware-agnostic projects like PyTorch and vLLM.

• LMSYS Org: Develops essential tools for inference and evaluation, most notably SGLang.

There are thousands more companies, universities, and research institutions making essential open-source contributions to inference. Over time, technologies have been built at increasing levels of abstraction:

• CUDA: Direct communication to the GPU for explicit control over computations and memory.

• Deep learning frameworks: Abstractions over CUDA for training, exporting, and running neural networks in Python.

• Inference engines: Highly configurable PyTorch-backed inference for common architectures.

• NVIDIA Dynamo: Sits on top of inference engines to power large-scale deployments.

Most inference engineering today happens at the higher levels of abstraction, configuring and deploying inference engines and orchestrating inference across multiple GPUs. No matter which level of the stack you work at, it’s essential to have a strong mental model for the adjacent levels of abstraction to guide your work.

6. What infrastructure does inference need?

When you scale production traffic, your assumptions are rigorously tested. Everything from input and output sequence lengths, to traffic patterns, to what topic a user decides to chat about; they all impact your observed performance in production. And maintaining secure, robust infrastructure is an entirely different skillset from optimizing model inference on the GPU.

No matter how fast and efficiently a single instance can serve a model, the service will be overwhelmed if traffic gets high enough. It’s an infrastructure problem, not with PyTorch or CUDA, and it requires a different mindset and different technologies.

Scaling in production introduces new complexities about where and how to get GPUs, balance traffic across them, and prevent downtime. The goal of autoscaling is to ensure you always have enough resources to serve all incoming requests, while maintaining latency SLAs and without wasting money on idle GPUs.

Autoscaling systems use Kubernetes, an open-source container orchestration system, along with a cluster-level system for provisioning and deallocating compute. Kubernetes can run one or more replicas of a model container, each on its own instance. An instance includes the GPUs and other hardware resources that the container requires.

Unless your traffic is unusually consistent, there probably isn’t a specific number of replicas that perfectly matches your needs.

Autoscaling is the practice of dynamically adjusting the number of replicas allocated to a given model within a cluster. There are two ways to make autoscaling decisions:

• Utilization: Scale up or down based on GPU utilization signals like memory usage or compute usage.

• Traffic: Scale up and down based on the number of requests being processed in the system.

Utilization and traffic don’t always match. For example, in LLM prefill, a few requests with hundreds of thousands of uncached input tokens could cause much higher utilization than many small requests with high cache hit rates.

Traffic-based scaling decisions can be made proactively, while utilization is a lagging indicator. Use both in combination to keep system resources matched with demand.

When designing a traffic-based autoscaling system, you want to configure five factors:

• Min replicas: What is the minimum number of replicas that stay running, regardless of traffic?

• Max replicas: What is the maximum number of replicas you can allocate when traffic is high?

• Autoscaling window: How long is the sliding timeframe used to measure traffic and make autoscaling decisions?

• Scale down delay: For how long after a scale-down is suggested do you wait, in case there’s another traffic spike?

• Concurrency target: How many requests can each replica handle at once?

The exact configuration determines how well the autoscaling system achieves its goals of maintaining latency SLAs without wasting resources. For example, increasing the scale-down delay prevents premature scaledowns for spikey traffic, but could result in unnecessary spend after traffic has properly cooled down.

Autoscaling within a single cluster works up to a certain point, but high-volume deployments serving a global user base need thousands of GPUs distributed around the world.

It’s straightforward to build multi-cloud inference as a collection of siloed compute across different cloud providers. But in these setups, there’s no way to use inter-cloud compute fluidly, and moving workloads across clouds is a tedious, error-prone process.

True multi-cloud inference requires building a multi-region, multi-provider bin packing tool, which treats distinct pools of compute as fungible with each other. Like Kubernetes within a single cluster, multi-cloud capacity management must take a global view, enabling global scheduling.

Running true multi-cloud inference unlocks:

• Capacity: Pool capacity from multiple providers for greater, more flexible GPU access.

• Redundancy: Split inference across providers for resiliency against outages.

• Latency: Run inference close to end users to reduce network latency overhead.

• Compliance: Run inference in compliance with data sovereignty and other regulatory requirements.

Scaling from one cluster in one cloud to many clusters in many clouds requires a new coordination layer. A multi-cloud architecture contains:

• Control plane: Handles model deployment and global scaling decisions, receives real-time event streams.

• Workload planes: Handles direct inference traffic and in-cluster scaling decisions, reports utilization and demand.

This separation of responsibilities ensures that individual workload planes can serve traffic independently. If something happens to the control plane or any given workload plane, other workloads should be unaffected.

7. Five approaches to make inference faster

One of the coolest things about working in inference engineering is that, unlike many industries where new academic research takes years or decades to be adopted, techniques from new papers are live in production within months or even weeks.

But there is a gap to bridge between research and production, and some of the most visible inference engineering work of all comes from doing so.

Real-world traffic defies constraints. But with volume, you can adapt systems over time to match the changing nature of usage. Tuning the parameters of inference engines, speculation algorithms, and model servers isn’t a one-time task. Instead, either through iterative deployments or dynamic runtime adjustments, you can continuously improve the performance of an inference system.

Finding the right combination of techniques and configurations takes patient experimentation. I remember an internal hackathon during which one of Baseten’s inference engineers worked on an autocomplete model for code, and ended up trying 77 different configurations via a handwritten script before finding a non-obvious solution that doubled TPS (tokens per second) for a customer’s model.

Sometimes, techniques are symbiotic or incompatible, which makes inference optimization even more complex. For example, quantizing the KV cache alleviates a bottleneck in disaggregation, but increasing batch sizing reduces the compute available for speculation. An inference engineer’s challenge is always to create a balanced set of optimizations that delivers more than the sum of its parts.

Let’s look into the key categories of applied research for inference acceleration: quantization, speculation, caching, parallelism, and disaggregation.

Approach #1: Quantization

Quantization means reducing the numerical precision of a model’s weights. It improves latency (both TTFT [time to first token] and TPS, increases system throughput, and opens up headroom for other optimizations like disaggregation, speculation, and prefix caching to be even more effective. But when it goes wrong, quantization can materially reduce a model’s output quality.

Models are trained with weights, activations, and other components represented in a certain native number format. Usually, this is BF16 or FP16, although 8-bit and 4-bit native precisions are becoming more popular for training.

Post-training quantization works by changing those model weights and other values from their native number format, to a lower-precision format. Cutting precision in half improves performance in both phases of inference:

• Prefill: Compute-bound prefill now runs on lower-precision Tensor Cores with twice the FLOPS.

• Decode: Memory-bound decode now loads half as much data per value, effectively doubling memory bandwidth.

Working with quantized data introduces overheads, so it’s not linearly twice as fast to go from 16 to 8 bits. In practice, quantization down a single level of precision generally offers 30%-50% better performance for LLMs. The catch with quantization is that it runs the risk of reducing a model’s output quality, and has the potential to introduce precision errors throughout the calculations that power inference.

Precision errors compound over time. Consider what happens when you square and cube different precisions of Pi:

Most of the work in quantization is in preventing precision errors and minimizing their impact on the final model output.

Sixteen-bit, 8-bit, and 4-bit precisions are the primary formats for inference. Number formats contain:

• Precision: The number of bits used to express a single value in the format. For example, FP16 uses 16 bits.

• Type: Whether these bits are interpreted to represent an integer (non decimal) or a floating-point number (decimal).

• Scale factor: A multiplier used to map values from a low-precision format back to the higher-precision format.

Combined, these attributes determine the two factors behind how well a number format represents the values used in inference:

• Dynamic range: The difference between the lowest and highest value that can be represented in the format.

• Granularity: The number of parameters or other values that are quantized along a single scale factor.

Dynamic range is essential to low-precision inference without quality loss. Sixteen bits can represent 65,536 distinct values, while 8 bits can only represent 256 different values. The dynamic range is the distribution of these values – the difference between the smallest and largest available value.

Dynamic range explains why floating-point formats are better than integer formats for inference. Floating-point formats have three properties:

• Sign: A single bit that represents whether a number is positive or negative.

• Exponent: A set of bits that, taken together, represent an exponent factor.

• Mantissa: A set of bits that together represent the base value multiplied by two to the exponent.

An FP8 number in an E4M3 data format means it has a 4-bit exponent and a 3-bit mantissa, with the remaining bit for the sign. Integer formats only have sign and value bits.

The exponent in floating-point numbers gives it a higher dynamic range, meaning it can better express very large and very small numbers. This is important because outlier values are significant in inference, and floating-point number formats better represent outliers after quantization.

Within floating-point formats, there are multiple options at each precision, like FP4, MXFP4, and NVFP4. These formats differ in granularity, or in the number of values quantized by a single scale factor.

Quantization can be applied at three levels:

• Tensor level: Calculate a single scale factor for the entire QKV tensor.

• Channel level: Calculate a different scale factor for each feature vector within the tensor.

• Block level: Within each feature vector, divide the vector into blocks of N values and calculate a scale factor for each block.

More granular quantization has a lower chance of smoothing over outliers, which preserves quality. However, more granularity also introduces extra overhead for storing and applying scale factors.

The components of a model have varying sensitivities to quantization. Reducing the precision of more sensitive components runs a higher risk of quality degradation. From the least to most sensitive components:

1. Weights: the linear layers are least sensitive to quantization.

2. Activations: The intermediate output of activation functions are only somewhat sensitive to quantization. They are rarely quantized as they are such a tiny fraction of the model’s weights.

3. KV cache: The cached values from the attention calculation are moderately sensitive to quantization.

4. Attention: The attention layers of a model are highly sensitive to quantization, especially equations like softmax.

Within each component, you can get more selective about quantization.

Even in linear layers and activations – generally the least sensitive to quantization due to their size – early and late layers, like the input and output layers of the neural network, may be left in their original precision as these layers are more sensitive.

While quantizing weights and activations helps performance, KV cache quantization gives an additional boost to techniques like prefix caching and disaggregation. The KV cache is a valuable resource and quantizing it allows inference engines to store more of it in memory and read it more quickly.

However, the KV cache for each token is used by each subsequent token. This means precision errors introduced by quantization can compound from token to token. Compounding errors are exactly why attention layers are the riskiest to quantize: not only is attention very sensitive to dynamic range, but each attention calculation relies on the results of each previous attention calculation. Therefore, over a sequence of thousands of tokens, errors accumulate quickly.

All but the most aggressive quantization schemes run functions like softmax in their original precision.

A moderate approach to low-precision inference uses a format like FP8 with high dynamic range – if possible, a microscaling format like MXFP8 – to carefully quantize select linear layers, activations, and often KV cache values. Even with these high dynamic range formats, components of the attention layer are rarely quantized.

Approach #2: Speculative decoding

The decode phase of LLM inference is an autoregressive process in which tokens are generated one at a time. The bottleneck on decode is memory bandwidth, with compute sitting idle at low-to-moderate batch sizes as weights are read from memory.

Speculative decoding takes advantage of that spare compute to try and generate multiple tokens per forward pass through the target model. If an inference engine could generate two, three, or even more tokens for each round-trip of weights through memory, it would generate far more tokens per second. Note, speculative decoding only improves TPS / ITL (inter-token latency), not TTFT (time to first token.)

There are multiple algorithms for speculative decoding and they share a common mechanism:

1. The speculator generates one or more draft tokens.

2. The target model – or the underlying model that you’re trying to accelerate – performs validation on these tokens to check if they match what the model would generate.

3. The target model accepts any valid draft tokens and generates an additional token itself, completing the forward pass.

This generates N+1 tokens per forward pass, or iteration through the decode loop, where N is the number of accepted draft tokens.

Generating draft tokens is not free, it takes both compute and memory. However, it is much faster for a target model to validate a draft token than to generate an original token. If you imagine a sudoku puzzle, solving it is hard, but checking if the solution is correct is very easy. For the target model, generating a token is like solving a sudoku, while validating a draft token is like checking a finished sudoku.

The performance uplift from any speculative decoding strategy depends on three factors:

1. Draft token cost: Time taken to generate a draft token.

2. Draft sequence length: The number of draft tokens generated per forward pass.

3. Token acceptance rate: The percentage of draft tokens accepted by the target model.

Token acceptance rate is high early in the draft sequence, but draft tokens get less reliable deeper in the sequence.

Aim for short, high-percentage sequences because while generating and validating tokens is inexpensive relative to generating tokens in the original model, it still comes with meaningful overhead. Additionally, once a single draft token is rejected as wrong, all subsequent tokens in the sequence are also rejected.

Working with speculation is interesting because so many factors affect token acceptance rate. The big one is the temperature parameter – higher temperatures yield token distributions that are harder to predict, reducing the effectiveness of speculative decoding. But even factors as simple as subject matter can make a difference on acceptance rate if the draft model or additional decoder head used for speculation is better versed in, say, math than history.

Another limitation on speculative decoding is that it’s most useful at low batch sizes where there are spare compute cycles. At higher batch sizes, speculative decoding must be dynamically disabled as compute is too saturated to afford verification.

Each speculation algorithm navigates these tradeoffs differently, and careful implementation of the right algorithm for the situation can lead to major improvements in TPS.

Approach #3: Caching

During prefill, the inference engine builds a KV cache (a store of keys and values for each token) on the input sequence. It then updates the KV cache for each token during decode. As inference is autoregressive, the value for each new token depends on the value of every previous token in the sequence.

Every inference engine uses KV caching by default on a request-by-request basis. Without KV caching, LLM inference would be unbearably slow since each previous value in the entire sequence would need to be recalculated for each subsequent token.

However, engineers can get more utility from the KV cache by reusing it between requests rather than solely within each inference sequence.

Consider the following two prompts, each with four tokens on most tokenizers:

By default, the inference engine has to run prefill on all four tokens of each prompt. But the first tokens of each prompt – “Weather in” – form a shared prefix between the pair.

With prefix caching, you can reuse the KV cache from the first request to improve TTFT on the second request by skipping prefill on the first two tokens and reading in the existing KV cache instead.

When you see pay-per-token APIs charge less for “cache hit” input tokens than “cache miss” tokens, this is why: reusing cached tokens takes very little compute power and time. As an inference engineer, you can apply the same principle to reduce latency, improve throughput and therefore save money on your own deployments.

Saving two tokens won’t make a big impact on TTFT, but prefix caching can skip prefill on thousands of tokens in certain domains:

• Complex system prompts: Agents, customer-facing chatbots, RAG scaffolds, and tool calls often feature long, complex system prompts on every call.

• Code completion: Code completion, code generation, and other coding functions require passing the same thousands of lines of code as shared context.

• Documents and retrieval: Document summarization, question answering, and retrieval all add repeated context ahead of user prompts.

• Multi-turn conversations: Ordinary conversations repeat back every message in a chat template, increasing the savings from prefix caching with every turn.

Prefix caching works from the start of the input sequence until the first non-repeated token. The fourth token in the weather example, a question mark, is shared between the two input sequences. However, the prefix ends at the first non-repeated token, so the fourth token isn’t read from cache.

Since prefixes end at the first unique token, your context engineering determines TTFT savings. Consider a different approach to the same prompt:

Here, there is no savings from prefix caching, as the very first token differs between the two sequences, even though every subsequent token is the same. To take advantage of prefix caching, ensure that novel tokens are as late in your prompt as possible.

Approach #4: Parallelism

Tensor Parallelism (TP) should be your default strategy for multi-GPU model inference. It supports dense models like Llama 405B, and the MoE (mixture of experts) models that currently dominate the open model landscape.

TP works by splitting apart each layer of the model (as opposed to Pipeline Parallelism, which keeps layers intact) and distributing the layer fragments across the allocated GPUs. For each layer, the expense of reading from weights’ memory and executing matrix multiplication is shared across the GPUs.

However, the results of each layer need to be communicated in an all-reduce fashion (across all eight GPUs) into a single output before the next layer can be computed. In nodes with high-bandwidth intra-node NVLink and NVSwitch, this communication overhead is minimized.

Increasing Tensor Parallelism improves TPS on a per-user basis, assuming the model is large enough and the sequences are long enough that the communication overhead doesn’t outweigh the faster forward pass – which is the case for most frontier models.

Expert Parallelism (EP) neatly divides experts across GPUs, so that in a model with 128 experts served in EP8 across eight GPUs, each GPU hosts 16 full experts.

EP improves total system throughput, making inference more scalable and less expensive. With individual experts processing tokens separately, each token takes just as long, but the system as a whole can handle more simultaneous tokens.

Many deployments use a mix of TP and EP to achieve both benefits.

EP requires less inter-GPU communication than Tensor Parallelism. The Expert Router, which determines which experts each token activates, is replicated onto each GPU as it is a relatively small component of the model. Inter-GPU communication is necessary for passing tokens from expert to expert, but unlike TP, it is not required to collect the results of each layer.

Thanks to this lower communication overhead, EP scales well to multi-node deployments and systems with limited interconnect bandwidth.

Approach #5: Disaggregation

Disaggregation combines three important ideas in inference engineering:

1. Prefill is a compute-bound process that determines the time to first token (TTFT), while decode is a memory-bound process that determines TPS.

2. Specialization improves performance in everything from kernel selection to inference engine parameter tuning.

3. You can effectively parallelize model serving over multiple GPUs, or even multiple nodes, if you can avoid bottlenecks from lower-bandwidth interconnects.

When prefill and decode run on the same node under heavy traffic, they have a higher chance of interfering with one another. Ideally, prefill uses more compute resources, while decode uses more memory, and the two can co-exist efficiently. However, with larger batches and more compute-intensive optimizations, prefill and decode start competing for resources.

Disaggregation, or disaggregated serving, is the idea of separating prefill and decode into separate engines on separate GPUs or nodes.

Disaggregation turns LLM inference into a three-step process:

1. The prefill engine takes the input sequence and generates a KV cache while computing the first token.

2. The prefill engine sends the KV cache over the hardware interconnect to the decode engine.

3. The decode engine computes all subsequent tokens.

In conditional disaggregation, the request is first sent to the decode engine, which checks if the input sequence is already cached, or is short enough to handle locally:

1. If so, the decode engine handles prefill locally, skipping disaggregation.

2. If not, the decode engine transfers the request to the prefill engine for disaggregated serving.

Conditional disaggregation is better for real-world traffic.

Another benefit of disaggregation is that with separate prefill and decode engines, you can optimize each engine individually and the system as a whole. For example, the compute-bound prefill engine requires a lower TP than the memory-bound decode engine.

Takeaways

This is Gergely again. Thanks to Philip for this deepdive into inference engineering, which is around 10% of the contents of his new book, ”Inference Engineering.” If you’d like to go deeper into this topic, you can download the full book for free:

Get the full e-book, for free

This title will also be available in physical, printed form: sign up to the waitlist to be notified when it’s available.

It’s encouraging that inference engineering is no longer a “monopoly” belonging to a few leading AI labs. Top AI model makers like OpenAI and Anthropic control all aspects of their AI models – from training to inference – so there’s no inference engineering to be done with them.

However, thanks to increasingly capable open models, engineering teams have the opportunity to tweak how they use models, and this is where the theory and practice of inference engineering becomes invaluable.

Even so, the discipline of inference engineering still seems to only make sense for a subset of tech companies. To justify investment in inference engineering, you need to be spending big money on inference from vendors. This is the point at which it can make sense to invest time and money to see if you can set up your own inference stack on top of open models, and swap out some existing usage.

I wonder if inference engineering is the AI version of the “build vs buy” dilemma. For software-as-a-service (SaaS), the question for every company is whether to build it in-house, or buy from a vendor. For example, should you build a project management software (it’s possible!), or just buy an existing one? And what about feature flagging, not to mention observability?

Experienced engineers all understand the pros and cons of building it yourself (time and maintenance, which is a constant drag.) Tuning and operating your own LLM stack is a much newer field, and inference engineering is at the heart of building better inference stacks than what comes “out of the box” with open models.

Picking up the basics of inference engineering feels like a valuable skill – and it’s also new and interesting. If you become well-versed in inference engineering, you could create optionality for your own team and company in LLM usage. Running your own inference stack on top of an open model gives control of what you’re running and of pricing. Inference engineering helps create options for achieving better performance from an open model by using the approaches covered in the extract above from Philip’s book.

After I left Uber, we stayed in touch, and a few months ago I heard he was exploring the next opportunity, and found out from him that Sam’s job search looked nothing like most people’s: he didn’t apply for a single role. Instead, there were reachouts from former colleagues desperate to hire them.

When we talked for this article, Sam had three warm leads which wanted to interview him ASAP. One startup was not even hiring, but the founder was ready to create a new position just for him.

I posted a message on LinkedIn about Sam:

“I hate the term “10x engineer” but this engineer is a role model for what a standout engineer is - in fact, some of my writing of standout engineers reference my interactions with folks like them (e.g. my article on the product-minded engineer, this one: https://lnkd.in/et7nWBgW)

And still, from the outside, this engineer is nearly completely invisible.

No social media footprint. The LinkedIn profile lists his companies worked at, and nothing else: no technologies, no projects, nothing. Their GitHub is empty for the last 5 years, and has perhaps a dozen commits throughout the last 10 years.”

This is Sam’s GitHub contributions for the last several years: absolutely nothing.

One of the most upvoted comments on my post was by cloud technologist Olivier Frolovs, who requested an article on Sam for others to learn how he operates. Now, Sam has generously agreed to an interview and has asked to remain anonymous, hence the nom de plume (pseudonym).

He doesn’t seek public attention and has a strong professional reputation. During our chat, he offered pointers for engineers looking to excel, and also as proof that an empty GitHub profile and zero social media presence don’t mean you can’t be a truly standout developer.

I also interviewed one of his former managers from Uber for that perspective. Today, we cover:

1. Getting things done. High-level task breakdowns, combined with communicating delays as tradeoffs to stakeholders.

2. Setting boundaries. Saying “no,” prioritizing family or work – and being clear about it, and treating prioritization as a daily practice.

3. Office politics. Participate selectively and cautiously, build relationships with influential colleagues, pre-sell ideas, direct communication.

4. Negotiation and conflict. Approach engineers before their managers, build bottom-up consensus, and start with relationship-building.

5. Promotions, keeping up-to-date, and finding the next job. A personal take on The Big Tech promotion processes, keeping up with the industry, and relying on referrals more.

6. Becoming a manager. Ownership and independence separate “good” engineers from “great” ones.

7. Feedback from Sam’s ex-manager. A former manager reveals what made Sam stand out to them – and shares some potential growth opportunities.

Background

In this article, my questions are in italic and Sam’s voice is in normal text.

Sam, how did you get into tech?

I was always intrigued by computers, software, and – as it came around – the internet. My dad had a black-and-white screen laptop for work with Windows 3.1 and some games. We got our first personal computer with Windows 95 and I remember it vividly.

I started developing websites for my primary school and the company my mom worked at when I was 12. I got paid a small sum, so that was my first-ever paid programming project! Around then, I taught myself Visual Basic 6 and started building 2D and 3D mini-games. I found the website DirectXVB (which is still live today), emailed the website’s owner with issues I ran into, and they helped me with pointers. Later, I taught myself PHP and built more dynamic websites.

From the age of 14, I stopped all coding – I just got tired of it! – and focused on my studies, and getting into college. I chose a non-computer science major for university, but picked up coding on the side, and I rediscovered that spark, so, in the first year of my Master’s, I decided to switch and do a Computer Science Bachelor’s. It was during college that I started to build apps and websites, and it’s when I got truly hooked on software development.

From agency, to large company, then Uber

I joined an agency as their first hire, building apps for local companies. It was a small team, we learned with trial-and-error, and getting done at 2-3 o’clock in the morning was common enough. I stayed 18 months and learned a lot about ownership, the importance of an eye for detail, and collaborating with others.

My favorite part of the job was the few times I worked directly with a designer: our agency employed freelance designers who were not involved in most of the projects because the company was trying to save money by having them work less, and not be involved in planning and rollouts. But during the implementation phase, I’d find myself talking with the designer and bouncing implementation and design ideas around.

I then joined a small startup where we built our own product. A highlight was having two designers on the team fulltime, whom I could work with and learn from. Engineering also felt like a level up: everyone cared about software quality and UX details.

Our startup got acquired by a larger company and most of us moved to the Bay Area. We stayed together as a team and were told we would maintain a “startup culture.” The founders tried their best to stay true to their word, but they couldn’t shield us from the reality of working in a corporation.

I learned a lot about corporate processes, and it was more interesting than I’d expected. As I was getting closer to the senior engineer level, I had to understand how internal politics worked, how to “massage” peer teams to help support our proposals, and how to talk with engineering leaders like senior managers and directors. Our company was also hugely focused on the annual company event: it was eye-opening for me to see just how much effort went into preparation. It consisted of several rehearsals and dedicated engineering work to showcase our stuff in a way that was near-flawless on the day.

After a few years, I felt ready for a change and joined Uber. I took a “title cut,” something akin to the “seniority rollercoaster.” At Uber, I worked in a new area and got promoted several times. After Uber, I worked at another Big Tech, and now – very recently – I’ve begun at a startup.

1. Getting things done

Feedback at Uber about you during performance calibrations was that you’re excellent at getting things done. What’s your process?

When I started out as a junior dev, I pulled long hours so I could deliver on time – regardless of how much effort it took. I don’t know what it was, but I always felt that failing to deliver on time was never an option.

I still vividly remember one project where I worked incredibly hard but still failed to deliver with the quality I expected from myself. As embarrassing as it is, I was so exhausted that I almost started crying on the spot. One of my coworkers comforted me and told me:

“Man, you’re crying about the wrong thing. No one died, no one got hurt, and no one will even care that we’re a few days late, save for the project manager. But even he’s used to everything being late. Go home, have some sleep, come back tomorrow and take it easy.”

They were right, of course. Still, I’m pretty sure this inner pressure to be unsatisfied with “good enough” explains a lot about how I work.

Having a high-level breakdown of the work, and communicating to stakeholders has been my “secret”, later in my career. After a few years as a dev, my estimation skills got better and I had to pull fewer late nights. I also found a hack that greatly helped was doing a high-level breakdown as early as possible, in all cases. As soon as I understand what the work is, I break it all down, ideally on a whiteboard or paper.

Importance of communication

You were also seen as a strong communicator, whether it was with engineers, engineering managers, or product managers. How do you get your point across?

Communicating delays as “tradeoffs” works extremely well. As soon as I start a project which I’m the lead on, I establish communication channels with key stakeholders: product managers, my engineering leadership, and business stakeholders, via email or Slack, I keep them in the loop at least weekly, and about anything that could be a roadblock.

In my experience, delays are not an issue as long as they are communicated upfront with an explanation and potential alternatives. When we hit a roadblock that slows down our work, I would never communicate that we’re “behind”. I would offer alternatives like:

• We can still ship on time, but we’d need to cut X and Y features for this release

• If we are not comfortable cutting X and Y features, then we will need to push out the target date by 2 weeks. If we are comfortable, we can push it out by 1 week

The trick, I’ve found, is to make it clear to stakeholders that we have a choice: choose more features to ship, or choose a lower-priority feature to drop.

I learned most of my hacks from people who are good at getting things done, and they have a few attributes:

• Task breakdown: early in my career, there was a senior engineer who was methodical about breaking down tasks and making estimates, even for seemingly trivial projects – and it worked!

• Communication tools: I observed the few really organized product managers, engineering managers, and tech lead, and made their communication styles into a “package” that worked for me; things like email updates, facilitating kickoff meetings, launch announcements (including how to communicate a failed/sunset project as a successful launch), and more.

Being good at communication means you have a solid foundation, and then develop a feel for how to best utilize the tools you have. There’s no “one-size-fits-all” approach: people react better or worse to different things. Try to get to know folks around you and put yourself in their shoes.

Doing great work

What does “standout” work look like to you?

I think about the quality of my work similarly to the quality of work I do at home. I have moved houses and renovated them several times and I greatly care about the quality of that work. And I’ve seen plenty of contractors come to my place, perform their work, and then leave without actually caring about the quality. They just want to “get s*** done” and be out of there. I never understood how someone can keep doing their job without feeling a lot of love for it!

I need to get energy from everything I do, not just in my job. Whether it’s playing games with my kids, helping my wife with her website, or building a new website feature for the company I work for: I approach it all with the same attitude.

Equally, if I no longer get energy from the work I do, then I basically stop enjoying it and this can be a nudge to start to look for something else. If it continues for a long time, this urge can become more persistent, and that’s the point when I have switched companies or teams. I can go on for some time without getting energy from my work, but it drains me. I try to catch myself before it gets too bad, and I’ve managed to do so, up to now. This is why I quit my last job without having anything lined up: I stopped getting energy from it for many months and talked with my management chain about it, but they were unable and unwilling to change anything. I needed a change, so it was me who made it.

Stepping outside of domain expertise

You frequently went outside of your domain, working with engineering teams on other platforms and contributing to codebases you’re not expert in. You seemed to have a great relationship with most engineers, in contrast to some devs. How did you do this?

I am pretty curious and prefer to talk directly with engineers. So, when I’d work on a project with engineers on a different stack, I would ask them to explain their high-level architecture approaches, and roll up my sleeves to make small code changes in a stack I was unfamiliar with.

Once you understand the high-level structure of a different codebase, and you also know how to make a few small changes, suddenly, it’s so much easier to figure things out on your own!

An approach that consistently worked for me is approaching problems from the customer’s perspective, and being genuinely curious. For example, I might ping an engineer working on a different system and ask:

“I noticed a customer has this problem, and to fix it, we probably need to touch the system you own. I don’t know much about this system: can you explain how it works, and what we could perhaps do to solve this issue that causes frustration for the customer?”

By making it clear that my goal is to solve a customer problem, I’m not coming across as just digging around for nothing. And by making it clear that I’d like to learn from them, it avoids being seen as someone trying to confirm what they are doing, which could come across as arrogant – especially when the other engineer is the expert on their own system. I’ve found fellow engineers are happy to explain their understanding and decisions.

2. Setting boundaries

At Uber, I recall you were very good at setting boundaries and saying “no.” How do you do that?

Honestly, I find it tough to say no – but I learned that it’s worse when I don’t. I found that saying ‘yes’ to everything usually results in an unmanageable, unbalanced pile of work. Prioritizing is key: I always remind myself to focus on what matters most. For me, the “most important” thing for any given topic could be:

• A shipping deadline that needs to be hit and is non-negotiable

• Family needs

• An urgent task that needs to be done on the same day

Family is very important to me. When I worked at Uber, I had a reasonably long commute to the office. I blocked out my calendar so I could leave on time in order to be home for dinner with my family. This did not mean I stopped work immediately; I would sometimes work during my commute and, when necessary, I logged back on to continue working after my kids were in bed.

When we had important deadline agreements at work, I made an agreement with my partner that I stayed longer in the office because I knew it was important to put in extra effort and deliver standout work then.

It goes back to prioritizing and focusing on the most important thing. Looking back, I’d say most of the time, the most important thing for me was family, and that work overrode this every now and then.

My approach to prioritizing keeps changing, though. Demands at home keep changing and expectations at work also change; after Uber, other jobs increasingly focused on async and remote work. This meant more flexibility to accommodate family time – but work could spill over into evening hours if I did not finish everything.

If I can give one piece of advice, it’s to understand what is important for you. Know your number one, number two, number three priorities: and arrange your workday so you do your top priorities. Don’t compromise on the most important one!

3. Office politics

At work, how plugged in were you to office politics?

I was aware of politics and tried to build relationships with “influential” people. I try to stay away from “cocky” types, and to find what I want to achieve through different folks.

The importance of politics is something I really started to understand when working at Uber. Initially, I was ignorant, but the more experience I got in Big Tech, the more it became obvious. It took a while before I was able to participate in it. I never liked it; I tend to be direct and transparent, but that does not work in every situation.

Did you take part in it to get stuff done?

Yes, sometimes by being direct and transparent, and communicating the right amount of information, you can get a lot done. Occasionally, it required me to “massage” an idea on multiple people before going to the person who called the shots.

What is your view of engineers who are seen as“political”?

It’s part of the game and sometimes it’s useful to have a good relationship with those people, as you can use that for your own benefit, as well. I personally would never invest much time in understanding and practising politics, as I prefer to focus on building and product.

4. Negotiation & conflict

Negotiating with teams

You were perceived as being good with other teams, and at removing roadblocks for your own. How did you approach this?

Read more

Last week, we took an inside look into how Uber is adopting AI, internally. The rideshare giant has built close to a dozen internal systems to deal with code generated by AI agents. However, when quantifying the impact of AI, the focus was on how much output has increased, and how devs who use more AI also generate more pull requests; these are the “power user” devs who generate 52% more PRs than devs who use AI less. There was no mention of product quality – at all!

And there are signs that product quality is dropping overall. Today, we dig into this under-discussed topic, covering:

1. Anthropic: degraded flagship website. An annoying UX issue irritated paying Claude customers – and no one at Anthropic noticed. The company moves very fast, generates 80%+ of production code with Claude, but quality and user experience seem to be taking a backseat.

2. Amazon: AI-agent reliance triggers SEVs. Amazon’s retail org has a leap in outages caused by its own AI agents. Now, senior sign off is needed for junior engineers’ AI-assisted changes.

3. Big Tech: “use AI or you’re unproductive.” Companies like Meta and Uber are tracking AI token usage in performance reviews, putting pressure on engineers to use it heavily — irrespective of the tools’ quality impact.

4. OpenCode: more time spent cleaning up. Dax Raad, OpenCode’s creator, warns that AI agents are lowering the bar for what ships, discouraging refactoring, and don’t speed teams up.

5. 5. Startups: founders see LLMs slowing down long-term velocity. Sentry’s CTO and others observe that while AI removes the barrier to getting started, it also produces bloated, hard-to-maintain code that slows long-term development.

6. Research: AI agents underperform claims. Some studies show AI coding tools produce short-lived velocity gains followed by significant tech debt increases.

7. How do we solve it? Engineers with strong architectural sense become more critical than ever, proposed solutions include formal validation methods, and perhaps reviving some old school QA ideas.

1. Anthropic: degraded flagship website

This article’s genesis was last week, when I’d finally had enough of a persistent UX bug on Claude’s flagship website: the prompt I typed in regularly got lost. Below is a video of me typing “How can I…” – and “losing” the first two words when the page loaded:

It’s pretty straightforward:

1. The page starts to render and the textbox is displayed

2. The user starts to type their prompt, but the page has not finished loading subscription data

3. The subscription information loads around a second later

4. The textbox is reset and the typing is lost

This is a pretty basic bug you might expect in a prototype, except that this is the landing page of Claude.ai, and it’s a bug that impacts every paying customer – easily millions – every day. Even worse, the bug happens every time you visit the site.

Somehow, nobody at Anthropic tested the site to catch a plainly obvious bug which impacted 100% of paying customers. At the same time, no company uses AI coding tools more than Anthropic: around 80% of the company’s code is now generated by Claude Code, so we can assume a good part of the website is also created that way.

My complaint about Anthropic’s website being broken went a bit viral and got the attention of the developer team:

To their credit, three days later the bug was gone. There’s no longer a “double load” of the textbox: it takes a bit longer to load but only does so once.

Still, it makes me wonder how much longer this issue would’ve continued had nobody complained. Also, how many more bugs are present on the Claude website that nobody highlighted on social media? How many more features could be shipped in a state that is subpar for production-grade software with millions of paying customers?

Anthropic seems to be prioritizing moving very fast over doing so with high quality. There is no denying that the company is moving at incredible speed and running laps around competitors. A good example is how they built Claude Cowork in just 10 days. Claude Cowork handled work with Microsoft Word and Excel documents surprisingly well, to the point that it set off a “code red” inside Microsoft’s Office division, I understand.

Microsoft responded as fast as possible, but it still took 2-3 months to launch their (cloned) response, called Copilot Cowork earlier this month, with full access still to follow soon.

In the case of Anthropic, moving fast with okay quality seems to make good business sense: they build a better product than what already exists, so no matter if it’s a bit rough around the edges; they can fix quality issues post-launch and still be months ahead of the competition.

2. Amazon: reliance on AI agents causes SEVs

Anthropic can afford to move fast while it’s growing at an extremely high rate and expanding its market share rapidly. At the same time, established players like Amazon have extreme focus on reliability: AWS has become the top cloud provider not least by being extremely reliable (as well as aggressive on pricing).

Well, reliability at the online retailer seems to be getting worse, too, and the company’s AI agent, Kiro, could be causing SEVs (Amazon’s phrase for “outage”), according to The Financial Times (emphasis mine):

Amazon’s ecommerce business has summoned a large group of engineers to a meeting on Tuesday for a “deep dive” into a spate of outages, including incidents tied to the use of AI coding tools.

The online retail giant said there had been a “trend of incidents” in recent months, characterised by a “high blast radius” and “Gen-AI assisted changes” among other factors, according to a briefing note for the meeting seen by the FT.

Under “contributing factors” the note included “novel GenAI usage for which best practices and safeguards are not yet fully established”.

“Folks, as you likely know, the availability of the site and related infrastructure has not been good recently,” Dave Treadwell, a senior vice-president at the group, told employees in an email, also seen by the FT. (...)

He asked staff to attend the meeting, which is normally optional.

Junior and mid-level engineers require more senior engineers to sign off any AI-assisted changes, Treadwell added in the briefing note.”

This meeting was the regular “This Week in Stores Tech” operational one, but what was new was the note telling staff to attend this “optional” meeting, and the mandate for senior engineers to sign off code changes from juniors. The outages may have been caused by less experienced engineers over-trusting GenAI’s output. Also, there were incidents caused by AI changes, said the FT:

“Separately, the company’s cloud computing arm — Amazon Web Services — has suffered at least two incidents linked to the use of AI coding assistants, which the company has been actively rolling out to its staff.

AWS suffered a 13-hour interruption to a cost calculator used by customers in mid-December after engineers allowed the group’s Kiro AI coding tool to make certain changes, and the AI tool opted to “delete and recreate the environment”, the FT previously reported.”

Again, a tool causing an outage is not its own fault: it’s on the engineer who lets the tool run wild. If I delete two lines of code, then push it to production, and the server crashes, the fault is not with the text editor or the Git client, but with me who made the change. Similarly, if you prompt an AI agent to do something, and the AI agent goes off and does its stuff which causes an outage, then responsibility lies with the engineer who didn’t set up guardrails for the agent.

However, there is the issue that AI agents can wreak havoc in ways devs don’t quite understand or expect, until learning the hard way. This was what took down a lesser-used AWS service, according to the report:

“Amazon Web Services experienced a 13-hour interruption to one system used by its customers in mid-December [2025] after engineers allowed its Kiro AI coding tool to make certain changes, according to four people familiar with the matter.

The people said the agentic tool, which can take autonomous actions on behalf of users, determined that the best course of action was to “delete and recreate the environment”.

It sounds like an engineer gave overly broad permissions to the coding agent which then used its scope to delete a service. As mentioned, the engineer is responsible, but there is also a learning curve with these AI agents to consider: it’s not like this type of outage happened in the past. Plus, companies like Amazon are heavily incentivizing using AI agents for as much work as possible, which naturally leads to overuse.

3. Big Tech: “use AI or you’re unproductive”

Something happens at places that measure devs’ AI usage: pressure builds for all devs to use more AI or else be seen unproductive and at risk of poor performance reviews, potentially leading to a PIP or worse.

Meta is taking token usage into account during perf reviews. A current engineering manager at the social media giant told me that the token usage of each engineer is now a data point — one of many! — for performance calibrations. By itself, it is not a positive or negative signal, but someone perceived as having low impact and with low token usage is now seen as a blatant low performer. For high performers with outstanding impact, very high token usage is seen as a good thing as it conveys to the manager group that they’re personally invested in AI and are improving their workflow – as proved by results.

We previously covered how performance calibrations work at places like Meta.

Big Tech CEOs are starting to see AI “power user devs” as superior to their coworkers. Uber is a good example: the Dev Platform team started to analyze the output of engineers by whether or not they’re in the “power user” category, meaning they use AI agents at least 20 days per month. They found more PR output by engineers who are power users. So far, that’s useful data, but it’s just one piece of information, and doesn’t reveal the quality of the PRs, the impact of the engineer, or any other business outcome.

By the time this data reaches CEO level, it has turned into something else. Here’s Uber CEO, Dara Khosrowshahi, interpreting the same data points on the Diary of a CEO podcast (emphasis mine:)

“While 90% of our engineers are using AI tools of some sort, there’s about 30% of them that are using them at a completely accelerated pace. And it [using AI tools heavily] really is changing their productivity in a way that I’ve never ever seen before.”

There’s a step from observing more PRs per engineer, to judging power users as being more productive for that reason. Dara continues:

“I can imagine maybe 5 years from now, as the engineers get more and more productive, that I may not decide to add engineering headcount because at that point instead of adding an engineer, I should add agents and buy some more GPUs from Nvidia. That may be the investment in the future.”

Unsaid in the above is that by that time, only engineers “using AI at a completely accelerated pace” would be employed. Would it also mean that engineers not on the bandwagon are on the way out? I appreciate Dara speaking his mind and shedding light on the thought process of a Big Tech CEO.

Inside large tech companies, it’s becoming a career risk to not use AI at an accelerated pace, regardless of output quality. These large companies are the ones likely to be mulling layoffs, like Meta reportedly preparing to cut up to 20% of staff. And when it comes to identifying redundancies, it’s a fair assumption that things like “AI usage” and “pull requests per engineer” will be taken into account, especially as one theme of such layoffs will almost certainly be that the employer wants to focus more on AI.

So, it’s common sense (and self-preservation) to use more AI, if only not to be seen as unproductive. Their perceived output will rise and engineering leadership will share more reports about productivity being up, and interpreting more code generated and more pull requests as the proof.

4. OpenCode: “more time spent cleaning up”

Dax Raad is founder and CEO of OpenCode, an open source AI coding agent, into which you can plug in models like Claude, ChatGPT, Gemini, and others. It’s an increasingly popular alternative to the likes of Claude Code and Codex. In our recent AI tooling survey, it came up as a tool used nearly as much as Google’s Gemini CLI and Antigravity. A small team works on this increasingly influential tool, and is seeing problems with AI overuse. Dax wrote this note to the OpenCode team (emphasis mine):

Read more

Update on 11 March: the Uber team shared updated numbers as of March 2026:

• 84% of devs at Uber are agentic coding users (either using CLI-based agents or making more agentic requests than tab-completion in their IDE)

• 65-72% of code is AI-generated inside IDE-based tools. This number is, naturally, 100% for AI command line tools like Claude Code.

• Claude Code usage nearly doubled in 3 months — from 32% in Dec to 63% in Feb, while IDE-based tools (Cursor, IntelliJ) have plateaued.

The article is updated with these new and correct numbers.

I spent four years working at Uber until 2020 and experienced firsthand the company’s standout engineering culture. Uber is a company that did the speed run of going from a small startup, through hypergrowth, to being a large company facing major risk during the pandemic, when the rideshare business briefly collapsed. Today, it’s maturing as a publicly traded, profitable company, and employs almost 3,000 people in the tech function.

At the recent Pragmatic Summit in San Francisco, one of the most interesting behind-the-scenes sessions came from the ridesharing company’s principal engineer, Ty Smith, and director of engineering Anshu Chada, who pulled back the curtain on what Uber has been doing with AI tools, internally. They were candid about the amount of work it took to build up Uber’s internal “AI stack,” why all that work was necessary, and also discussed the drawbacks as well as benefits of this rapidly spreading technology. You can watch their presentation at The Pragmatic Summit here.

In today’s issue, we cover:

1. Agentic layers & systems. Four layers spanning an internal AI platform, context sources, industry tools, and specialized agents for testing and code review.

2. Internal tooling: MCP Gateway, Uber Agent Builder, and the AIFX CLI. Uber built several internal tools to make it easier for devs to use AI tools, and to make internal AI agents more effective.

3. How AI changes developer workflows. A move away from single-threaded coding in an IDE, to orchestrating multiple parallel agents. Engineers naturally gravitate toward kicking off new agents, which starts to create resource and cost challenges.

4. Minion: running background agents at scale. Uber built Minion, an internal background agent platform with monorepo access and optimized defaults. It’s a clever abstraction layer that works well in practice.

5. New internal dev tools. More AI-generated code means more code reviews and more noise, so Uber built Code Inbox for smart PR routing, uReview for high-signal AI code review comments, Autocover for generating 5,000+ unit tests per month, and Shepherd for managing large-scale migrations end to end.

6. Challenges. AI adoption is slower than expected, even at a forward-thinking company like Uber. Top-down mandates are less efficient than engineers sharing their wins with peers.

7. Impact in numbers. 92% of Uber devs use agents monthly, 65-72% of code is AI-generated inside IDEs, and 11% of pull requests opened by agents. At the same time, AI-related costs are up 6x since 2024, and token cost optimization is a growing priority.

Longtime readers might recall we’ve covered Uber’s engineering culture over time:

• Developer Experience at Uber – with Uber’s founding engineer on Developer Platform, Gautam Korlam (2025)

• How Uber built its observability platform (2023)

• Inside Uber’s move to the Cloud (2023)

• How Uber is measuring engineering productivity (2022)

• The Program and Platform split at Uber (2021)

Let’s get into it:

AI is not new at Uber, but rolling it out companywide is. The company has used machine learning and AI technologies in many systems, including its Marketplace platform, which are responsible for routing and matching drivers with riders, forecasting demand, etc. What is relatively new at nearly all tech companies is the process of integrating AI across engineering and beyond.

The official strategy at the ridesharing giant is to become a “GenAI-powered” company:

I appreciate Uber sharing this approach openly because while most companies say that they want to be “AI-powered” – however cliche that claim might be – not all provide as much transparency.

It’s worthwhile engineers internalizing how leadership views AI. These folks, in general, see a tool that can bring efficiency everywhere. My take is that in some ways, AI is seen similarly to the cloud, which has been perceived as a means to reduce costs and improve the flexibility and elasticity of hardware resources. Today, AI is seen as the way to increase efficiency and lower costs, such as customer support, software development, the finance function – or any function.

Uber is focusing not on automating everything possible in engineering. Instead, it wants to:

1. Eliminate toil: helping AI do “boring” work like upgrades, migrations, trivial bug fixes, etc.

2. Free up engineers to focus more on creative work.

As Anshu Chada, Engineering Director on Uber’s Dev Platform, puts it:

“What we found is when we push some of the boring stuff to AI – upgrades, migrations, bug fixes – not only does it result in much higher satisfaction from our engineers, but they’re able to push our product and create features for end users in ways that we didn’t even think were possible.”

1. Agentic layers & systems

Uber’s “agentic system” for software engineering is actually made up of several systems:

Categories of systems:

• Internal AI platform. Built on top of Michelangelo, Uber’s ML/AI platform. This layer provides things like a model gateway to proxy to frontier models or internally hosted models.

• Internal Uber context: accessing Uber’s source code, engineering documentation, Slack information, JIRA tickets, etc. These all serve as “memory” for agents to use.

• Industry agents: Uber’s approach is to enable the “latest & greatest” AI agents for engineers, so they support several tools like Claude Code, GitHub Copilot, Codex, and other clients.

• Specialized agents: Uber’s background agent platform, the test generation platform, code review agents, and more.

• Engineering enablement: measuring the efficiency of agents, controlling costs, and educating engineers about which tools to use.

2. Internal tooling: MCP Gateway, Uber Agent Builder, and the AIFX CLI

MCP – the Model Context Protocol – has quickly become the standard way to connect agents and data sources with one another. A frequent analogy is that MCP is like the “USB-C interface for AI agents.” We published a deepdive on the MCP protocol and covered real-world MCP server use cases.

Uber put together a “tiger team” (a temporary unit that gets things done fast) to design the MCP strategy and build the central MCP gateway, which looks like this:

This MCP gateway allows:

• Proxy internal endpoints to MCPs: any internal Thrift, Protobuffer, or HTTP endpoint can be exposed as an MCP server with a simple configuration change. Uber uses the Apache Thrift protocol and Protobuffer protocols extensively for backend service communications

• First-party MCPs: these are exposed as a single, consistent interface

• Third-party MCPs: external MCP servers are also exposed via the gateway, which handles all authentication and authorization tasks.

• Platform concerns: the gateway takes care of authorization, telemetry, and logging in one central place. Plus, it conveniently offers a unified interface to interact with any MCP.

The MCP gateway also provides:

• A registry: to look up MCP servers, and for devs to be able to register their own.

• A sandbox: for devs to experiment with MCP servers without long-winded setups.

Uber Agent Builder

Uber’s Agent Builder product is a no-code solution to build agents that can access Uber’s internal data sources (both MCP servers and Uber data sets), and hand off work to other agents:

The platform includes a tool called Agent Studio, where multi-agent workflows can be visualized, debugged, traced, versioned, and evaluated. This is how it looks:

The agents built in Agent Builder become discoverable through a registry:

Accessing agents via the AIFX CLI

Uber’s Developer Experience platform team had a few issues with deploying AI agent tooling at scale:

• How does the company update all clients when a new version comes out? For example, if a new version of Cursor is released, how can they ensure all devs use the latest one?

• How can the clients be configured with helpful defaults? Uber’s Dev Platform team might have found better defaults for tools, like more helpful models. How are these rolled out to all devs?

• How can devs easily discover MCP servers and configure them for agents?

• How can agents connect to Uber’s background task infrastructure?

The Dev Experience team built the AIFX CLI, which is the AI tooling command line used by all engineers there. Here’s what it looks like:

This tool supports:

• Provisioning AI agents (client tools like Claude Code, Codex, Cursor, and others)

• Finding and using MCP servers

• Running background agent tasks

• Updating AI agents and clients to the latest versions

3. How AI changes developer workflows

The traditional way of building software:

Devs spent some time planning, most time writing code (code authorship), and then some time in code review.

The first AI agent-based workflows were single-threaded: devs worked with a single agent in the command line or inside their IDE:

At Uber, the latest workflows which many software engineers follow are pretty different, involving parallel agents, each kicked off with their own tasks:

As Ty explained, running multiple agents comes naturally to most devs:

“[Once you start using agents] as an engineer, you’re giving a prompt and waiting for something. While it’s running and you’re waiting you’re thinking: ‘What am I going to do? Have a coffee or browse Reddit? Might as well kick off another background agent.’

And so, engineers get into this mode of running several agents at once, right? Both us [at Uber] and a lot of the industry is trying to push towards this.”

4. Minion: running background agents at scale

Read more

This article provides a high-level overview of those findings from our latest AI tooling survey. Thank you to everyone who participated!

There are plenty of interesting details, most notably, validation of just how much Anthropic and Claude Code have risen to dominate tooling usage. Claude Code is today nearly as widespread as GitHub Copilot was in our survey three years ago (in the spring of 2023) – which shows how fast the AI market moves.

In today’s issue, we cover:

1. Interesting findings: Claude Code has rocketed to #1 in just eight months, AI is fully mainstream with 95% weekly usage among respondents, and more.

2. Most-used AI tools: Claude Code leads the pack, followed by chatbots and GitHub Copilot. Cursor’s rising fast, and newcomers like Codex and Antigravity are gaining traction. Most engineers juggle two to four tools at once.

3. Popular models: Anthropic’s Opus and Sonnet models dominate coding tasks by a wide margin, with more mentions than all others combined.

4. AI trends: mainstream adoption achieved. 95% of respondents report using AI tools at least weekly, 75% use AI for half or more of their work, and 56% report doing 70%+ of their engineering work with AI.

5. AI agent usage rising: 55% of respondents now regularly use AI agents, with staff+ engineers leading adoption on 63.5% usage in the survey results. Agent users are twice as excited about AI as non-users are.

6. Company size and tool usage: smaller places overwhelmingly favor Claude Code (75% at the tiniest businesses), while large enterprises default to GitHub Copilot. This popularity is probably down to enterprise procurement preferences – and Microsoft’s enterprise marketing efforts.

7. Tools engineers love: Claude Code is the most loved tool at 46%, far ahead of Cursor on 19% and GitHub Copilot at 9%. Senior leaders are especially enthusiastic about Claude Code.

8. Demographics: Overview of who took part in the survey.

Full subscribers also have access to a longer, 35-page report with additional details - linked at the end of this article.

1. Interesting findings

Here are my ten personal, most-interesting findings from this survey:

1. Claude Code has gone from zero to be the #1 tool in only eight months. Released in May 2025, it’s already the most-used AI coding tool, overtaking GitHub Copilot and Cursor.

2. AI is now mainstream. 95% of respondents use AI tools at least weekly, or more often, and 75% use AI for at least half their software engineering work. Among readers of The Pragmatic Engineer, it seems the question is no longer whether to use AI in day-to-day work, but which tools to use.

3. Cursor is catching up fast on GitHub Copilot. As much as we hear about companies dropping Cursor for Claude Code, Cursor is doing more than fine, growing in mentions 35% since our previous survey nine months ago.

4. Most engineers juggle multiple AI tools. 70% use between two and four tools simultaneously, while 15% use five or more.

5. Staff+ engineers are the heaviest agent users. 63.5% use agents regularly; more than regular engineers (49.7%), engineering managers (46.1%), and directors/VPs (51.9%).

6. Codex is seeing explosive early growth. Despite not existing during the last survey, OpenAI’s Codex already has 60% of Cursor’s usage (!!)

7. Using agents correlates strongly with being positive about AI. People using agents are nearly twice as likely to feel excited about AI while non-users are twice as likely to be skeptical.

8. Company size influences tool choice more than preference. Huge companies (10K+) more likely to use Copilot (56%), tiny startups mostly go with Claude Code (75%) and Cursor (42%). It seems like enterprise procurement, not individual preference, is behind this divergence.

9. A tight chatbot race. ChatGPT, Gemini, and Claude as standalone chatbots have nearly equal numbers of mentions, suggesting there’s no clear winner outside of coding-specific tools among software engineers.

10. Directors and senior leaders are especially into Claude Code. The survey finds this tool is twice as popular with these folks as it is at less senior levels, while Cursor gets less love as seniority increases.

Let’s jump into some of the data:

2. Most-used AI tools

Just eight months after its release, Claude Code is already the most-used tool, overtaking both GitHub Copilot and Cursor:

Tools mentioned, in order of popularity:

1. Claude Code: a terminal-first coding agent from Anthropic. We cover its history in the deepdive, How Claude Code is built

2. Chatbots: ChatGPT, Claude, Gemini, and others

3. GitHub Copilot: launched in 2021, it’s the “oldest” AI coding tool on this list

4. Cursor: agent-powered coding IDE. Learn more about it in our deepdive, Real-world engineering challenges: building Cursor.

5. Codex: OpenAI’s AI coding agent. We recently covered how Codex is built

6. Gemini CLI: Google’s command line agent

7. OpenCode: the most popular open source coding agent, where you can swap out the model being used, sidestepping vendor lock-in

8. Antigravity: Google hired the original team behind Windsurf and launched its own agentic IDE

9. JetBrains Junie: the AI coding agent by JetBrains

10. Zed: a fast editor with agentic workflows

11. Windsurf: when Google acquihired the team, it did not buy the product. An AI IDE, now owned and operated by Cognition

12. Amp: a model-agnostic coding agent. A free version is supported by ads

13. Augment Code: an enterprise-focused coding agent

14. Factory: agent-native software development, which calls agents “droids.”

It’s interesting to compare how people answered the same question just nine months ago, last May:

Notable trends:

• Claude Code has massive momentum and is already the market leader. Since being released in May 2025, it has become the most-used AI coding tool among survey respondents. Anecdotally, I’ve heard several teams that have been Cursor power users are introducing Claude Code with great success (while still keeping Cursor in-place.)

• Cursor has grown circa 35% in nine months and now threatens GitHub’s popularity. In this survey, 35% more respondents mention using Cursor than in our previous research, last May. Such growth is impressive: at this rate, Cursor will have more users than GitHub in 6-9 months!

• Chatbot usage remains high. Combined mentions for ChatGPT, Claude, Gemini, Perplexity, and others used outside of coding apps are still higher than for any other tool except Claude Code. For context, the most-mentioned chatbot (ChatGPT, 107 times) only has as many as Gemini CLI (also 107).

• GitHub Copilot usage stable. Nine months ago, 46% of respondents said they used GitHub Copilot. Since then, it’s barely risen.

• Explosive growth for Codex. OpenAI’s Codex wasn’t available during our last survey. But it now already has 60% of Cursor’s usage(!), and could be growing even faster since this survey.

• Up-and-coming tools: OpenCode, Gemini CLI, Antigravity. None of these had launched nine months ago, but today they’re used by around 10% of respondents – no small feat!

• Tools growing more than before: The likes of Zed, Windsurf, Amp, Augment Code, and Factory appear less often in these results, but all are mentioned more by respondents than they were nine months ago.

Chatbot usage: ChatGPT leads – just about

Here’s how mentions of chatbot usage line up in our survey:

Tool usage: multi-tool

Most tech professionals use between two and four AI tools. An interesting detail is how many different ones are mentioned by respondents:

Noteworthy details:

• 70% of survey participants mention using between two and four tools

• 15% use a single tool

• 15% are using 5 or more

3. Popular models

Anthropic’s Opus and Sonnet dominate the ranking of models used for coding.

This is not even a contest: Opus 4.5 and Sonnet 4.5 (latest models at the start of our survey) come up more often than all other models, combined. Anthropic has become the go-to model developer for coding-related work – for now, that is. When this survey launched, Opus 4.6, Sonnet 4.6, and GPT-5.3 were not yet out.

Around 1 in 8 respondents say they just use whatever model is the default at their company. This is interesting to note: these are likely folks who might not bother changing default settings, and just go with whatever’s available. If the default model is powerful enough, that’s fine, but if the company’s default is a cheaper, less capable model, then these people could face a more frustrating experience than those who get to choose what they use.

In the “other” category of models, some other mentions include:

• Cursor’s custom Composer/Composer-1 model, and its “Cursor Auto” auto-select model

• Kimi/Kimi K2.5 — Moonshot

• DeepSeek model variants like R1, V3.2, Coder

• Alibaba’s Qwen/Qwen3

• xAI’s Grok

• Various Mistral models

4. AI trends: Mainstream adoption achieved

How often do people use AI tools? Very often, as it turns out; 95% of respondents are using them weekly, at a minimum:

It’s worth reflecting on this data in relation to insights presented by Laura Tacho at the recent Pragmatic Summit in San Francisco:

It seems that AI is now mainstream in software engineering. Anecdotally, this has been my sense since the beginning of the year: everyone whom I talk with is using AI tooling on a roughly daily basis. Now, there’s data to prove it.

How much software engineering work gets done with AI?

This year, we asked readers to estimate the percentage of their software engineering work that’s done using AI. The results:

The data show that AI is embedded in the workflows of participants in our survey:

• Only 25% of respondents use AI for less than 40% of their work

• 56% do 70% or more of their work using AI

• 75% use AI for at least half of their software engineering work

5. AI agent usage rising

Eighteen months ago, AI usage was mostly for code generation and tab completion. There were one or two respondents who experimented with early AI agents in March 2024, as something equivalent to a junior software engineer.

This year, 55% say they regularly use AI agents. This is 507 people, a massive jump!

The most common use cases for agents:

• Code review and code validation

• Automating manual / annoying tasks

• Bug fixing / investigating bugs

• Code investigation

• Debugging

• “Crafting” code, or “weaving” it together with the agent

Below is a typical-enough comment from one software engineer who uses agents at a smaller company:

“I use agents for pretty much all coding work, mostly prompting with Cursor Chat. I use it for code investigation, bug investigation, creating commits and pull requests. It’s my tool for reviewing code, I am always still in the loop. Almost all of my AI-written code is still reviewed and ‘crafted’. When using it for code review, I find it helpful to chat with and gain understanding, rather than letting it loose on code review. So, I use it for everything, but I am still very much in the loop.”

A common arrangement in the survey is the split-screen setup: a terminal with Claude Code open to drive work, and an IDE also open to review changes made by the agent.

The tools mentioned by those who regularly use AI agents split almost identically to the broader survey results:

Staff+ engineers are the heaviest users of agents. Here’s the data on agent usage by experience level:

This data point is slightly amusing because it shows there’s not much difference between the lowest and highest levels: 46% of leads and engineering managers say they use AI agents regularly, while for Staff+ engineers it’s 63%. Does it suggest the most experienced engineers are also the most curious?

The more someone uses AI, the more they also use AI agents. We segmented the data by the percentage of software engineering work that respondents do with AI:

• Heavy users have AI for 80% or more tasks, or use AI on an hourly basis

• Moderate users have AI for 30-80% of tasks, or use AI on a daily basis

• Light users employ AI for 30% of tasks or less, or use AI on a weekly or monthly basis

The more positive someone is about AI, the more likely they are using agents on a regular basis. In contrast, those negative about AI barely use agents:

One question is whether this finding indicates correlation or causation: that is, does starting to use AI agents more, actually cause people to feel more positive about AI?

A couple of details:

• Using agents seems to make people nearly twice as enthusiastic about AI (61%) as those who do not use them (36%)

• Respondents who don’t use agents are twice as likely to be skeptical (22%) about AI than those using agents regularly (11%)

It looks like that if you don’t use AI agents on a regular basis, you may have a negative opinion about the tools, in general, which could come at the cost of not experiencing what the technology has to offer.

6. Company size and tool usage

In our results, company size and tooling choice correlate for some tools; for example, the smaller a team or company is, the more likely it is to use Claude Code or Codex:

Claude Code is used by a whopping 75% of the smallest companies and teams. This is a big number, far ahead of any other tools. At the smallest places, the most-used tools are, in order:

1. Claude Code: 75%

2. Chatbots: 55%

3. Cursor: 42%

4. GitHub Copilot: 35%

5. Codex: 26%

6. Gemini CLI: 14%

7. OpenCode: 13%

GitHub Copilot overtakes Claude Code at large companies. This confirms what was known: Microsoft is very good at enterprise sales, and at bundling GitHub Copilot in its suite of products:

Cursor and OpenCode usage drops at massive companies with similar usage patterns. Across the board, usage is roughly the same, regardless of company size. We only see a drop at the very largest of companies with 10,000+ employees:

One theory for such a drop at massive companies could be that large companies often build their own internal coding agents that engineers use; e.g., at fintech, Block, the agent is called Goose, Meta has its own agent, as does Google with Jetski (its version of Antigravity) and Cider.

Google’s tools are evenly used across the spectrum of company size. Google’s Gemini CLI and Antigravity are the only tools in this survey whose usage is notably stable, regardless of company size. Both tools hover at around 10% from the smallest to largest workplaces:

The feeling among survey respondents that they can experiment at work, correlates with Claude Code being available. We asked if readers experiment frequently with tools. Below are the “yes” responses by company size:

When these responses are mapped to the percentage of people using Claude Code, there’s a very similar distribution. My theory is that Claude Code is new enough at 9 months old to have not yet been approved at companies with bureaucratic processes for approving new tools, and this is partly why some respondents feel their chance to experiment with the range of tooling is being thwarted.

Engineers at places with lots of red tape for tooling are less empowered to experiment with new AI tools; not just Claude Code, but any new, interesting tool.

7. Tools engineers love

We asked respondents: “Which AI tools do you love using the most, and why?” Below are the tools which respondents enjoy most, in descending order of number of mentions:

Claude and Cursor stand out in terms of how much they are loved. A whopping 57% of respondents mention either Claude Code (46%) or Claude models (11%) as the tools they are attached to. Cursor was at a respectable 19% – double GitHub on 9%.

Other notable tools with two or more mentions:

• Warp: a terminal for building agents

• Zed: a fast editor with agentic workflows

• Amp: a model-agnostic coding agent. A free version is supported by ads

• Cline: an open source coding agent

• RooCode: an open source, AI-powered coding assistant running in VS Code

• Continue.dev: AI checks on every pull request

Claude Code is especially loved by Director-and-above folks. Segmenting the “most loved” responses by level (engineers up to the senior levels of staff+ engineers, leads/eng managers, and Director+ folks):

Both Claude Code and Cursor become less loved – or used! – as seniority goes up, but it’s notable that folks in senior engineering leadership positions are obsessed with Claude Code, but not Cursor.

GitHub Copilot is equally loved by engineering managers as Cursor is — and this is surprising:

Both OpenCode and GitHub Copilot are surprisingly loved by Staff+ engineers. Another unexpected detail is that, despite OpenCode being used about a quarter as much as GitHub Copilot, it rivals the “loved” mentions. For Staff+ engineers, it matches GitHub on those terms:

Finally, when segmenting based on company size, we see the familiar pattern of Claude Code vs GitHub Copilot. Claude Code is less frequently mentioned as a “loved” tool as company size increases.

GitHub Copilot sees the opposite trajectory: it’s more loved within larger companies where it’s likely to be harder to experiment with alternatives.

8. Demographics

In closing, below are details about who took the survey, and how the 906 responses came together.

Engineers comprise 55% of respondents, and engineering leadership another 34%:

Respondents are experienced professionals. The median respondent has 11-15 years of experience:

Company size is also a fairly even split across this group:

Region-wise, most respondents are in Europe or the US:

Takeaways and the full report

We have compiled additional findings from this survey which did not fit in this article: it’s a 35-page article:

Read more

It was well-timed, considering that right now seems like a period of change for tech that’s unfolding faster than before. That was a consensus opinion at both events, also held by veterans like Martin Fowler and Kent Beck, who said things haven’t shifted so rapidly during their 50+ years in the industry.

At the very start of this year, I predicted AI will write almost all code, going forward, and several others have said the same. But at the Pragmatic Summit, I met an embedded engineer writing Assembly and C code who is still writing more of his code by hand than with AI agents – and was the only person I spoke with in San Fran who was not yet “giving in” to AI agents.

Even so, today, this engineer has between a third and a half of their own low-level code being generated by AI agents since the launch of Opus 4.5, and this share keeps on rising. Their view was an interesting counterpoint to the prevailing trend.

This article shares some thought-provoking ideas and conversations from both events, covering:

1. Data vs hype: how orgs actually win with AI. Laura Tacho’s keynote at The Pragmatic Summit. Exclusive data reveals 92% of devs use AI coding tools monthly (!!), “unhealthy” orgs see 2x more incidents, healthy ones have 50% fewer, and other new data.

2. Building world-class engineering orgs in the AI era. The closing session of the summit dug into what AI-native teams look like, in a fireside chat with GitHub’s former CEO and the CTO of Atlassian.

3. The Future of Software Development with Martin Fowler. Laura Tacho, distinguished engineer Annie Vella, Martin Fowler, and myself, look back on the event in Utah. Nobody has the AI shift all worked out, which is reassuring!

4. Mid-level engineers’ quiet crisis. Something I heard that engineering leaders talk about behind closed doors a lot is that mid-career engineers are being left behind by the AI wave. New grads are more productive with the tools, while seniors have more of that all-important experience. Advice on how to catch up fast.

5. Future of refactoring in an AI-native world. Refactoring legacy code has long been an important tool for keeping software maintainable. With AI generating most code now, is there any point? One answer is “hell, yeah!”

6. Future of Agile. It was only fitting to consider the future of Agile on its 25th anniversary, in the company of two of the writers of the Agile Manifesto. Amusingly, one major trend is the return of Extreme Programming (XP) practices which predate Agile.

1. Data vs hype: how orgs actually win with AI

At the debut Pragmatic Summit in SF, Laura Tacho delivered the keynote as a well-known expert on developer productivity, and former CTO of DX. She presented new and exclusive data on AI usage, and what currently works with AI adoption. Her session was one of the most popular at the conference – and you can watch the whole thing here. Full subscribers also get access to Laura’s slides.

New data points on AI adoption shared at The Pragmatic Summit, provided by DX:

Usage:

Time management:

Onboarding impact:

There’s no typical experience with AI, said Laura:

“AI is extremely different in every company because each place has its own problems and its own culture.

Organizational performance is multidimensional and these organizations are just going off to different extremes, based on what they were doing before. AI is an accelerator, it’s a multiplier, and it is moving organizations in different directions.

Some organizations are facing twice as many customer-facing incidents. At the same time, some companies are also experiencing 50% fewer incidents. So, for some companies that have used AI and have a really healthy system, it has amplified that system. They are seeing fewer incidents, moving faster, accelerating with higher quality, higher code maintainability, and higher change competence.

On the other side, we have organizations which were dysfunctional already and are now more dysfunctional: they’re dysfunctional faster. Similarly, organizations are seeing really uneven results economically from using AI.

There are a lot of steep drop-offs when it comes to using AI in a pilot context with production, and then trying to tie it to profit. This is from an MIT study in July 2025, The Gen AI Divide”.

Good analogies for the age of AI might be the space race of the previous century and the age of exploration in the 15th-17th centuries, said Laura.

• Wonder of space: space exploration includes immense hope and grounded scepticism.

• Defining moments: even though the space race was incredibly expensive – or utterly wasteful – landing on the moon was a big moment for humanity.

• Delivering but not on the grandest ambitions: in the 1960s, going to the moon carried the expectation of human habitation on the lunar surface. As yet, that hasn’t happened but landing on the moon created progress back on Earth:

There’s a rare opportunity to channel the energy around AI into practical improvements, said Laura in closing remarks:

“I want to urge you to find that balance between a sense of wonder, aiming for Mars, and a moon colony, but also understanding we need to solve problems here on Earth and we have to live in this reality.

So, please stay grounded, stay skeptical, stay human. Most of all, stay pragmatic”.

Watch Laura's keynote

Access Laura's slides

2. Building world-class engineering orgs in the age of AI

Also at the Pragmatic Summit, Thomas Dohmke (founder & CEO of Entire, and former CEO of GitHub) and Rajeev Rajan (CTO of Atlassian) took part in a fireside chat to explore what a modern, world-class engineering team looks like today. Thomas is building a new, AI-native startup with Entire – an “AI-native GitHub” – while Atlassian has more data than anywhere else on developer productivity after acquiring DX.

Watch the closing session

A number of timely, thought-provoking topics were covered by the pair on stage:

Topic #1: What does an AI-native team look like?

Rajeev (Atlassian):

“It starts with the mindset. You have to really believe in doing AI-native work. Some teams at Atlassian have engineers basically writing zero lines of code: it’s all agents, or orchestration of agents. As a result, teams are not necessarily getting smaller, but they’re producing a lot more, sometimes 2-5x more, and creativity is up.

Efficiency framing is missing the point, it’s more about what you can create now with AI which you could not before”.

Thomas (Entire.io):

“AI native is the new ‘cloud native’ and what we are calling AI native now is going to be very different from AI native in a few years’ time.

There’s a lot of BS out there about how all day-to-day tasks are now ‘AI native’, and using agents for everything. I’m a startup founder: most of the time, I’m still dealing with things like old school HR school systems”.

Topic #2: AI makes being distributed an advantage again

Thomas:

“A remote team is kind of lonely, right? But agents basically gave us these sparring partners, these experts that I can just ask “bring some help in for this topic, I need to write an ADR or solve a problem.

This help has taken it to a new level for remote work. I now have my code review agent, my coding agent, my brainstorming agent, my research agents. In some ways, agents have given an advantage again for being remote-first.”

Rajeev:

“Agents have changed things a lot. I remember the days when I was in the office at 2:00AM writing code and would get stuck. All you can do is read the code, then read the code some more until you figure it out.

Well, now you can ask an agent to explain it, or even better, do the code for you. So, with agents it’s gotten even better and easier for us to be able to be distributed.”

Topic #3: Why restrictive corporate IT gives startups a massive advantage

One hilarious exchange between Rajeev and Thomas was about how large companies are slow with AI due to restrictive IT:

Rajeev: “Over the holidays, I bought a laptop because sometimes you can’t install things because of IT. So, I bought my own personal laptop at the Apple store, installed Claude Code, built an app, a bunch of Python scripts, and things like that. As someone who has grown up writing code to my personal bar, it’s super high – as it is for all my engineers. Now, I can write code much faster with agents and things like that.”

Thomas: “To all startup founders: when an investor asks how you’re preventing the incumbent from doing the same thing you’re doing: just tell them the CTO of Atlassian had to buy a laptop on his own money to start coding!

That’s the best answer I can give because it shows you how even relatively agile companies [like Atlassian, a promoter of Agile] that actually develop agile tools, can’t really do what you can do in five minutes!”

Topic #4: Why CTOs are rolling out agents with top-down mandates

Thomas discussed how engineering leaders introduce agentic tools after playing with them as side projects.

“What happened in the last two years through coding agents like Copilot, Cursor, and Devin, is that many CTOs and CIOs, even in the largest banks, realized they can go back to coding. It no longer takes hours to install all the packages and figure out all the problems.

I actually have friends in these spaces who tell me stories of how every night they’d give the agent a task and say: “build me this.” Then, in the morning they would check in with the agent and instruct it to keep going. And then they have their normal day-to-day as CTO of a big bank.

You do that for two weeks and you realize everything is going to change – and that it has to change in my organization. And in contrast to the Twilio founders – in their case, they had developers decide, bottom up, what tools to use – these CTOs and CIOs say: “I don’t want to hear any excuses. We’re going to roll out agents.”

Still, many banks are on very archaic systems and have developers log into Citrix, and jump through all sorts of hoops to use agents.”

Claude Code was introduced in a similar way at Canadian fintech, Wealthsimple, as we covered two weeks ago.

The fireside chat touched on more subjects, including the changing role of software engineers and engineering leaders, and the biggest changes Thomas and Rajeev expect this year.

Watch the closing session

You can now also watch the other keynote session: How AI is reshaping the craft of building software with OpenAI and the Codex team. Full subscribers can already check out all sessions at The Pragmatic Summit, Q&A included. The other sessions without Q&As will be released next week for everyone.

The remainder of this article covers an event hosted by Martin Fowler.

3. ‘Future of Software Development Summit’ with Martin Fowler

Back in February of 2001, a group of 17 software developers gathered at a ski resort in Snowbird, Utah, where they drafted and signed the famous Agile Manifesto.

Almost exactly 25 years later, Martin Fowler and Thoughtworks organized a retreat at almost the same place in Deer Valley, Utah. He invited around 50 tech leaders from large enterprises and nimble startups, researchers, authors, and experienced, hands-on software engineers. Held earlier this month, it was “The Future of Software Development,” and I was delighted to attend the two-day gathering featuring Annie Vella, Kent Beck, Steve Yegge, Gene Kim, Laura Tacho, – and of course, Martin himself.

This was no attempt to draft a second landmark manifesto; rather, it was an opportunity to share notes and consider where the tech industry might be headed. It was an intense day of back-and-forth about what we’re seeing and what to make of it.

One declaration was drafted. Kent Beck, Laura Tacho, and Steve Yegge, came up with an apposite statement on inflated expectations upon AI to magically improve everything in workplaces:

“Organizations are constrained by human and systems-level problems. We remain skeptical of the promise of any technology to improve organizational performance without first addressing human and systems-level constraints.

We remain skeptical and we remain human”. – Kent Beck, Laura Tacho, and Steve Yegge.

Personal notes

I have my own thoughts on the topics covered:

Everyone, everywhere is adopting AI rapidly, which in some ways is very surprising. Looking back on tech innovation over the last 20 years, such as mobile, cloud, or even crypto – adoption was gradual. For example, native mobile progressed like this:

1. Indie devs: in the first few years, startups, experimental individuals, and teams onboarded to the new technology. For native mobile apps, 2008 was when iOS and Android launched their app stores. Between 2008 and 2010, it was mostly indie developers, small teams, and companies that built their apps.

2. Startups and forward-thinking companies: a few years later, more startups saw the opportunity to jump in. For example, Uber was founded in 2010 and Snap in 2011. Some forward-thinking places also launched in this period, like Starbucks with an iOS app in 2009.

3. Larger tech companies: Years on, some forward-looking, larger tech companies adopted the technology. For example, Bank of America launched an app in 2011 (3 years into native mobile), Yahoo started to go all-in on native mobile apps from late 2012.

4. “Traditional” tech companies finally catch up: Eventually, slow-moving companies also move over when it becomes a baseline expectation to use a new technology. For example, Ryanair launched its native iOS app in 2014, six years after native mobile apps emerged.

Contrast that to today, when it seems like everyone is moving all at once. I talked with folks from companies like John Deere (agriculture), 3M (industrial products), Cisco (networking hardware), Wealthsimple (finance), AWS (cloud), and startups, and none could be called “behind.” Every company is rolling out agentic AI tools; for example, Wealthsimple did a global Claude Code rollout just a week earlier.

Based on past experience, you might expect at least some traditional companies with no exposure to AI to wait and see what happens. But I found none doing so.

Traditional, “old school” companies don’t seem to be lagging behind the pace. A traditional company with 10,000+ developers has more than 50% of software development outsourced. This place founded an AI platform team with a mission to eliminate outsourcing activity via a mix of efficiencies from AI tools and targeted hiring.

They have already built an AI debugging agent popular across the whole business that’s connected to all internal systems like monitoring, logging, and internal data stores. It helps a dev pinpoint any and all errors and bugs. This platform team is building more internal tools and hiring aggressively.

It’s a similar story at other trad places: most are already using AI agents, some are experimenting with AI used for large-scale refactors, and others are seeking use cases for agent swarms.

Smaller engineering teams are a certainty. Around 20 engineering leaders – many at large enterprises, some at tiny startups – said that engineering teams are shrinking. One Head of Engineering at a 200-year-old company in the agriculture sector put it like this:

“We are already seeing the end of two-pizza teams (6-10 people) thanks to AI. Our teams are slowly but surely becoming one-pizza teams (3-4 people) across the business”.

To emphasize, this is one of the most traditional companies which sells physical goods and hardware, and whose business has no disruption to fear from AI. But they’ve also adapted!

Observations from Laura Tacho

Laura was kind enough to share her notes on this event for this article:

Read more

Personally, I’ve been warming up to Codex since doing an interview for The Pragmatic Engineer podcast with Peter Steinberger, the creator of OpenClaw, in which he revealed that he writes all of OpenClaw with Codex, preferring longer-running agentic loops. Update: on Monday, Peter announced he is joining OpenAI to work on building next-generation agents. It’s a major win for OpenAI and the Codex team, while OpenClaw remains independent and open source. Check out my podcast with Peter in his first in-depth interview, around when OpenClaw (back then: Clawd) was getting massive momentum.

To find out how Codex was built, how teams at OpenAI use it, and what effect it’s having on software engineering practices at the ChatGPT maker, I spoke with three people at OpenAI:

• Thibault Sottiaux (Tibo), head of Codex.

• Shao-Qian (SQ) Mah, researcher on the Codex team who trains the models that power it.

• Emma Tang, head of data infrastructure, who isn’t on the Codex team but whose team uses Codex heavily.

This deep dive covers:

1. How it started. From an internal experiment in late 2024, to a product used by more than a million devs.

2. Technology and architecture choices. Why Rust and open source? In-depth on how the agent loop works.

3. How Codex builds itself. Codex itself writes more than 90% of the app’s code, the team estimates. Also: interesting engineering practices like tiered code review, Codex self-testing itself, and onboarding new engineers to the team via pairing.

4. Research. Training the next Codex model with the current one has parallels with software engineering. Running evals, A/B testing and internal dogfooding.

5. Codex usage at OpenAI. Emma’s data team built an internal “data agent” in two months — that would have probably taken over a year, before. The number of PRs is so large that the traditional PR flow is starting to crack, though.

6. How software engineering is changing at OpenAI. The “30/70 rule:,” some engineers returning to tab-complete, and the importance of “taste.”

7. Next steps. GPT-5.2 step change, the capability overhang, and where Codex goes next.

Last week’s debut Pragmatic Summit, in San Francisco, featured a fireside chat with Tibo, myself, and the audience, and featured new details about how Codex is built. Paid subscribers can watch this recording now. Free subscribers will get access to all videos from the Pragmatic Summit in a couple of weeks.

Longtime readers might recall a deepdive entitled How Claude Code is built, based on interviews with the founding engineers at Claude Code. Some comparisons with today’s topic are obvious: Codex and Claude Code have each made bets that seem to be paying off. I was initially skeptical when I talked with the Codex team last October because the cloud-first, long-running task approach didn’t click with me. But I’ve now changed my mind.

1. How it started

In 2024, OpenAI was experimenting with various approaches for building a software agent. That fall, the company declared that building an aSWE (Autonomous Software Engineer) was to be a top-line goal for 2025. This vision came from the top: Greg Brockman and Sam Altman believed they should have an autonomous software engineer working alongside teams. Tibo describes the thinking:

“Greg and Sam had the strong conviction: ‘eventually, we should have an autonomous software engineer working alongside us and with the capabilities seen from o1-preview, the time is now to have a group absolutely dedicated to making this a reality’”.

A number of folks who’d worked on earlier prototypes were pulled into the effort, which featured:

• Michael Bolin: tech lead for the Codex open source repository.

• Gabriel Peal: who subsequently built the VS Code extension, mostly solo, and built the foundations of the Codex desktop app.

• Fouad Matin: who led the initial release of the Codex CLI, and is responsible for Codex’s safety and security approach.

OpenAI had two teams tackle different segments of the problem space: Codex Web would focus on an async, cloud-based solution, while Codex CLI targeted iterative, local development. Both products would launch in the spring, with Codex CLI being announced in April 2025, and Codex in ChatGPT introduced in May.

2. Technology and architecture choices

An obvious difference between Codex and Claude Code is the programming language. Claude Code is written in TypeScript, “on distribution”, which plays to the underlying model’s strengths. Meanwhile, the Codex CLI is written in Rust. Tibo explains why:

“We debated TypeScript, Go, and Rust. All three seemed like solid contenders for different time horizons. In the end, our reasoning came down to a few layers:

• Performance: We want to eventually run this agent at a massive scale where every millisecond matters. Performance is also important when running locally in a sandboxed environment.

• Correctness: We wanted to choose a language that helps eliminate a class of errors with things like strong typing and memory management.

• Engineering culture and engineering quality: There’s this interesting thing that language choice does: it gets you to think about the engineering bar you set. We decided to pick Rust because it’s extremely important for our core agent implementation to be extremely high quality”.

There was also a practical concern about dependencies. Choosing TypeScript means using the npm package manager. Using npm often means building on top of packages that may not be fully understood – which could clearly be problematic. By going with Rust, the team has very few dependencies and can thoroughly look through the few dependencies there are.

They also want to eventually run the Codex agent in all sorts of environments – not just laptops and data centers – and even places like embedded systems. Rust makes this more achievable from a performance perspective than TypeScript or Go.

Tibo tells me that while Codex’s early performance was less standout with Rust than with TypeScript, they expected the model to catch up. Plus, choosing Rust gave them one more engineering challenge to work with. The Codex team also hired the maintainer of Ratatui – the Rust library for building terminal user interfaces (TUIs). He’s now full-time on the Codex team, doing open source work.

The core agent and CLI are fully open source on GitHub.

How Codex works

The core loop is a state machine, and the agent loop is the core logic in the Codex CLI. This loop orchestrates the interaction between the user, the model, and the tools the model uses. This “agent loop” is something every AI agent uses, not just Codex, and below is how Codex implements it, at a high level:

1. Prompt assembly: the agent takes user input and prepares the prompt to pass to the model. On top of user input, the prompt includes system instructions (coding standards, rules), a list of available tools (including MCP servers), and the actual input: text, images, files, AGENTS.md contents, and local environment info.

2. Inference: the prompt is converted to tokens and fed to the model, which streams back output events: reasoning steps, tool calls, or a response.

3. Response:

   1. Stream the response to the user by showing it on the terminal.

   2. If the model decides to use a tool, make this tool call: e.g. read a file, run a bash command, write code. If a command fails, the error message goes back to the model, the model attempts to diagnose the issue, and may decide to retry.

4. Tool response (optional): if a tool was invoked, return the response to the model. Repeat steps 3 and 4 for as long as more tool calls are needed.

5. Assistant message: the “final message” intended for the user which closes one step in the loop. The loop then starts again with a new user message.

Compaction is an important technique for efficiently running agents. As conversations grow lengthy, the context window fills up. Codex uses a compaction strategy: once the conversation exceeds a certain token count, it calls a special Responses API endpoint, which generates a smaller representation of the conversation history. This smaller version replaces the old input and avoids quadratic inference costs. We covered how self-attention scales quadratically in our 2024 ChatGPT deepdive.

Safety is an important consideration because LLMs are nondeterministic. Codex runs in a sandbox environment that restricts network access and filesystem access by default. Tibo reflects on this choice:

“We take a stance with the sandboxing that hurts us in terms of general adoption. However, we do not want to promote something that could be unsafe by default. As a dev, you can always go into your configuration and disable these settings if you want.

We made this default setting because many of our users are not that technical. We don’t want to give them something that could have unintended consequences”.

There are several releases per week. Internally, the team ships a new version of Codex up to three or four times a day. Externally, new releases are cut every few days and are distributed via package managers, Homebrew, and npm.

Michael Bolin’s recent blog post, “Unrolling the Codex Agent Loop,” lays out the internals of how the agent loop works.

3. How Codex builds itself

More than ninety percent of the Codex app’s code was generated by Codex itself, the team estimates, which happens to be roughly in line with what Anthropic has reported for Claude Code, according to what its creator Boris Cherny told me. Both AI labs share the meta-circularity of using the coding tools to write their own code.

Tibo tells me that a typical engineer on the Codex team runs between four and eight parallel agents, which do any one of a number of tasks:

• Feature implementation

• Code review

• Security review

• Codebase understanding

• Going through plans and summarizing

• Going through what team members have done and summarizing changes

• Bugfixes

• … and more.

Codex engineers are now “agent managers” and no longer just write code. Tibo says it’s common for an engineer to walk into the office with several tabs open on their laptop: a code review running in one, a feature being implemented in another, a security audit in a third, and a codebase summary being generated in a different tab. He says:

“Codex is really built for multitasking. There’s this understanding that most tasks will just get done to completion.

People on our team have figured out what Codex is and isn’t capable of. There is a tricky thing in all of this, though: we have to relearn these capabilities with every model”.

Frequently-used “skills”

“Agent Skills” are ways to extend Codex with task-specific capabilities, which is pretty much the same concept as Claude Code’s skills. Internally, the Codex team built 100+ Skills to share and choose from. Three interesting examples:

• Security best-practices skill: a comprehensive write-up of all security practices adopted by the team. When invoked, Codex goes through each practice, checks the code, and generates patches for anything missing.

• “Yeet” skill: takes any code change, writes up the PR title and description based on the original plan, and creates a draft PR in one step.

• Datadog integration skill: Codex connects to Datadog, reviews alerts and issues, finds problems, and tries to generate a fix for them.

The way Tibo thinks about skills is that they help steer the model to more specific behaviors, and they can also be combined. Skills are continuously published internally and team members copy from each other.

Tiered code review

The team set up AI code review and it always runs. The team trained a bespoke model for code review, optimizing it for signal over noise. Around nine out of 10 comments point out valid issues, says Tibo, which is equal to or slightly better than human reviewers. AI reviews are run automatically whenever a pull request moves from “draft” state to “in-review” state, and are automated via a GitHub webhook.

Following an AI review, there are two possible next steps:

• Non-critical code: can be merged with no further human review. For non-critical parts of the code where speed matters more than perfection, the engineer running the agent can decide to merge the code following an AI review, when they decide it’s ready.

• Key parts of the codebase: mandatory human review. For key code parts such as the core agent and open source components, the team insists on careful human code reviews.

Other engineering practices on the team

Here are more practices that Tibo rates as helpful for the Codex team:

• AGENTS.md. Instructions stored inside a repository. On the Codex team, these files tell the agent how to navigate the codebase, which commands to run for testing, and how to follow the project’s standards. These are a bit like README files, but written for AI agents instead of humans. Agents.md has become a de facto standard across agents, and the only major agent not to use it is Claude Code. See agents.md for more details.

• Structure code for agents: have tests! The team has deliberately structured their codebase “to make it inevitable for the model to succeed”. Structuring means having tests in-place, clear module boundaries, and instructions on how the model should run validation (eg: tests, linting, etc). When the model implements something incorrectly, a test fails, the agent notices, and it tries to figure out what went wrong. Since the model is trained to be persistent, it keeps trying until it gets it right. Tasks can run for 20 to 30 minutes, or sometimes an hour.

• Codex running the test suite to test itself. In what is an interesting meta-testing capability, Codex can fully test itself, using a specific skill. This skill runs tests for all of Codex’s own features. Amusingly, Codex seems to invoke this skill more often than expected.

• Overnight, Codex runs to generate suggested fixes. The team set up nightly runs of Codex, instructed to look for issues across Codex. Every morning, engineers review issues that Codex identified with the codebase, with fixes waiting for review.

• Onboarding to the Codex team via pairing. New joiners are asked to keep an open mind about how the Codex team does development, and are warned that the “how” of building software is different than at most places. A new joiner is paired with an engineer on the team and shadows them for the first part of the day, observing how they develop with Codex. They are then assigned a task in the second half of the day, and are expected to ship it to production on the same day.

• Unlimited Codex usage. To put everything in this deepdive in context, OpenAI employees – just like at Anthropic – have no usage cap on LLMs. This makes sense given they are building a tool they want customers to use as much as possible. However, it’s in contrast to non-AI labs. Keep this in mind as we go through how the team operates.

Using Codex to debug Codex

The Codex team holds meetings to discuss Codex, during which it’s common for engineers to fire off a thread within Codex to see if it can come back with some information. This January, something interesting began happening, Tibo told me:

“When we asked questions related to why Codex runs in a certain way, Codex decided to debug its own systems. It connected to logs, SSH’d into research dev boxes, and analyzed ML instabilities.

It figured out a lot of useful information, and wrote a little report we could present on the screen. So, we ended up going through this doc in the meeting”.

This feels like another meta-circularity of Codex debugging itself – or at least systems that power it!

4. Research

Codex is built by researchers as well as software engineers, of whom one is SQ Mah. He managed to move into research from software engineering by competing in the Vesuvius Challenge, of reading millennia-old carbonized scrolls from the ancient Roman town of Pompeii which were buried by a cataclysmic volcanic eruption of Mount Vesuvius in AD 79. SQ finished second in the contest by renting GPUs on Google Cloud, training models, taking research ideas and turning them into useful algorithms.

So, what is “research” like at OpenAI? SQ’s take:

Read more

The next podcast episode with Kotlin’s creator, Andrey Breslav, will be out on Thursday instead of tomorrow (Wednesday). Regular scheduling, including The Pulse, resumes next week. We’ll release session recordings from the Pragmatic Summit to paid subscribers first, and later to everyone.

Please wish me luck with this event tomorrow, it’s the first ever in-person event by The Pragmatic Engineer, and there’s a full house of 500 attendees. If you’re there, come and say hi!

Steve Yegge has been a software engineer for over 40 years, with stints at GeoWorks, Amazon, Google, and Grab. He’s known for provocative, entertaining blog posts, of which the most famous might be Stevey’s Google platform rant.

Steve recently published the book “Vibe Coding”, has also built Gas Town, an open-source AI agent orchestrator, and created documentation site and community hub Gas Townhall around the project. A year ago, we did a podcast focusing on his career, including how he used AI for coding. Last week, Steve and I sat down again in Salt Lake City and explored what’s changed in terms of using LLMs for coding, and also the wider tech industry.

In this article, we cover:

1. LLMs, the end of coding by hand, and the S-curve. Initially skeptical of LLMs, Steve became a convert after trying out Claude Code. He now argues the industry is headed into a steep exponential curve that shows no signs of stopping.

2. “50% dial” and deep cuts. Could big companies cut up to 50% of their staff in order to make way for AI tools? Such a trend might dwarf pandemic-era layoffs.

3. Eight levels of AI adoption. A spectrum of AI adoption, from “no AI”, to “building your own orchestrator”. Steve fears engineers stuck at the lower levels will be left behind.

4. The Dracula effect: the drain of using AI. Vibe coding at full speed can be physically draining, and Steve argues that employers shouldn’t reasonably expect more than three hours of AI-augmented work from engineers, per day.

5. Big companies are doomed – but don’t know it. Innovation at large companies is effectively dead, and the real action will come from small, AI-augmented teams – like when cloud computing shifted the balance of power.

6. Redundant values in the AI era. Knowing how to code doesn’t make engineers special any more, but demand for software will keep growing.

7. Debugging and programming languages. Agents may not need good debugging tools, and language choices have never mattered less.

8. Grieving the demise of coding by hand. Steve endured a grieving process when he realized many skills are obsolete, BUT building software is now more fun than ever.

9. Predictions. Steve’s non-coder wife could be the top contributor to their video game, Wyvern.

Related The Pragmatic Engineer articles featuring Steve:

• Amazon, Google, and Vibe Coding with Steve Yegge (podcast episode in 2025)

• The full circle of developer productivity with Steve Yegge (deepdive in 2022)

Originally, I had audio quality issues with this recording. Fortunately, I managed to improve the audio and remove echos, and the episode is also available as a podcast now:

Listen to the podcast version

1. LLMs, the end of coding by hand, and the S-curve

One thing about you is that you’re pretty pragmatic. You were always into compilers, debugger tools, and worked on hard problems at Amazon and Google. When AI came out, I don’t remember you saying, “this is amazing!” What was your initial reaction to LLMs?

Steve: “I was pretty blown away that they could write fairly coherent Emacs and Lisp functions. The original ChatGPT in December 2022 could already write code in a weird language, right? Not very much of it, and it was janky; but for me, that was the beginning. I’ve had friends who were saying for 20 years, “any minute now, any day now” about AI. But 2022 was the first time that it was like, “okay, I see now.

I was still skeptical when the rumors emerged about Claude Code at the start of last year, that Anthropic had a tool internally that was writing code for them and it was a command line tool. Like many others, my reaction was “no, it’s not!” Then I used it and was like, “oh, I get it. We’re all doomed.” It was then that I wrote The Death of the Junior Developer.

So, was I a skeptic? Yes. But did I pay attention to the curves from the very beginning? Also yes. It was at that point that I thought we’re on a curve, this is a ride, and it’s not stopping, so let’s get on board and see where it goes.”

I dived in. I was feeling behind because I didn’t know AI, I didn’t know its fundamentals, or the lingo. I spent a year doing nothing but reading papers and catching up”.

Your book, ‘Vibe Coding’, says on its rear cover: “the days of coding by hand are over”. When did you decide this?

Steve: “It was a year ago [early 2025] and that wasn’t even my quote. It’s Dr. Erik Meijer, the inventor of many things in the programming world, and one of the most important compiler people in the world. He spent his life building technology for developers to be able to write code, and he’s saying developers aren’t going to write code anymore. That’s what caused Gene Kim (author of The Phoenix Project and former CTO of Tripwire) and I to both say that if someone this prolific in the tech world says ‘we’re done writing code’, then what does he see that we don’t?

He sees the curves; it’s that simple. It’s like exponential curves, they get real steep, real fast, and we’re heading into the steep part this year”.

One thing about being an engineer is that you can draw curves, but you never know when they end, or if they flatten. What makes you believe that this curve will continue? There’s also the question of how long they will scale for.

Steve: “The world is filled with unbelievers; people who think the S curve goes up and then flattens, and that we’re at the hump stage right now. They have thought that ever since GPT-3.5 came out. They’re like, ‘it’s not going to get any better’, but then GPT-4 comes out and they think that’s as good as it gets. Now Opus 4.5 is out and most people haven’t played with it, and don’t realize what’s there. But that thing is already two months old.

The half-life between models has gone from about four months at the beginning of last year to two months from Anthropic at the beginning of this year, and we’re going to see another model from them soon, which will be so much further up the curve that people will really freak out. All the bugs, all the mistakes they’re complaining about get fed right back in as training, so it doesn’t make them next time. This is what people aren’t understanding.

Also, time continues, and there’s this inevitable collision of curves: there will be societal upheaval. It’s already started and people are justifiably mad. And I’m mad too. I’m mad at Amazon for laying off 16,000 people and blaming AI without an AI strategy. Many of those people are not going to be able to find jobs, by and large, and they’re the first of many to come. Nobody has a plan for this”.

2. The “50% dial” and large cuts

Why do you think Amazon did those layoffs if they don’t have an AI strategy?

Steve: “People will hate me for saying this, and me saying it doesn’t make it true: it was true already. Every company has a dial [of the percentage of staff they can lay off] that they turn from zero to a hundred. It just has a default setting of what percentage of your engineers you need to get rid of in order to pay for the rest of them to have AI – because they’re all starting to spend their own salaries on tokens.

And so, at least for a while, if you want engineers to be as productive as possible, you’re going to have to get rid of half of them to make the other half maximally productive. And as it happens, half your engineers don’t want to prompt anyway and are ready to quit.

So what’s happening is that everybody on average is setting that dial to about 50%, and we’re going to lose around half the engineers from big companies, which is scary”.

That would be more than during the Covid-19 pandemic and afterwards.

Steve: “It’s going to be way bigger. It’s going to be awful. But at the same time, something else is happening, which is that AI is enabling non-programmers to write code. It’s also enabling engineers who have seen the light and believe the curves are going to continue to go up to actually get together in groups and start to do things that rival the output of big companies that are tripping over themselves.

So, we’ve got this mad rush of innovation coming up, bottom up. And we’ve got knowledge workers being laid off by big companies because clearly big businesses are not the right size anymore.

Even Andy Jassy [Amazon’s CEO] is saying they’re going to do the same thing with fewer people. So, does this mean we’re going to have a million times more companies? Is there going to be a massive explosion of software, or are people going to get out of software altogether and do other stuff? I’m very curious about where all this goes”.

3. Eight levels of AI adoption

Elsewhere, you’ve said something that might trigger many people, that if you use an IDE today, then you’re a bad engineer.

Steve: “Well, you’ve got to be a little provocative! I’m not going to say you’re a bad engineer because I know some very good engineers – better than me – who are still at level one or two in my chart. But I feel sorry for people who are good engineers – or who used to be – and they use Cursor, ask it questions sometimes, review its code really carefully, and then check it in. And I’m like: ‘dude, you’re going to get fired [because you are not keeping up with modern tools] and you’re one of the best engineers I know!’”

Tell us about your chart and these eight levels.

Steve: “I drew it on a board in Australia for a big group of people to show what happens, as I saw them all as being at different phases. Some had their IDEs open, some had a big, wide coding agent, and others had a really narrow coding agent. I put them all on a spectrum just to show what’s going on. Here’s the levels:

• Level 1: no AI

• Level 2: Coding agent in your IDE, permissions turned on

• Level 3: Coding agent in IDE, “YOLO mode.” Your trust is going up.

• Level 4: you’re starting to not look at the diffs anymore, but at what the agent is doing. You’re not reviewing as much, you’re letting more of it through, and you’re really focused on the conversation with the agent.

• Level 5: your approach is: “I just want the agent and I’ll look at the code in my IDE later, but I’m not coding with my IDE”.

• Level 6: several agents. You’re bored because your agent’s busy and you want to do something, so you fire up another agent, then another. And you find yourself just multiplexing between them, and you can’t “leave” [you start to get addicted to using more agents.]

• Level 7: 10+ agents, managed by hand. This is where you typically say “oh gosh, I’ve made a mess! I accidentally texted the wrong agent and didn’t realize. How do I coordinate all these agents? What if Claude Code could run Claude Code?”

• Level 8: you build your own orchestrator to coordinate more agents”.

4. The Dracula effect: the drain of using AI

One thing you’ve mentioned is that AI can really “drain” people.

Steve: “There is something happening that we need to start talking about as a community. There’s a vampiric effect with AI, where it gets you excited and you work really hard, and you’re capturing a ton of value.

I find myself napping during the day, and I’m talking to friends at startups and they’re finding themselves napping during the day. We’re starting to get tired and cranky.

Companies are set up to extract value from you, and then pay you for it. The way all companies have always been set up is that they will give you more work until you break. People have to learn the art of pushing back. Let’s say an engineer can be a hundred times more productive, who gets to capture all that value? Well, if the engineer goes to work and works eight hours a day and produces a hundred times as much, the company captures all that value, and that is not a fair exchange.

There’s another group of people who are capturing all the value for themselves. They go in and work for 10 minutes a day and get a hundred times as much done, they don’t tell anyone, and they’ve captured all the value. And that’s not really ideal either, right?

I think that the answer is that each and every one of us has to learn how to say “no” real fast, and get real good at it.

I seriously think founders and company leaders and engineering leaders at all levels, all the way down to line managers, have to be aware of this and realize that you might only get three productive hours out of a person who’s vibe coding at max speed. But they’re still a hundred times as productive as they would be without AI. So, do you let them work for three hours a day? The answer is yes, or your company’s going to break [because people get burnt out]”.

5. Big companies are doomed – but don’t know it

With non-technical people getting into building software, could there be jobs for engineers to take over maintenance? For example, with websites you don’t need an engineer to build one because there’s Wix, WordPress, etc. But there are frontend engineers at big businesses where the website actually matters, and you want all these advanced bells and whistles. Could this be a thing with AI?

Steve: “There are going to be a lot of engineers doing software engineering, and I just think we’re all going to be doing it with AI. I think it’ll be quite some time before companies are comfortable trusting their code to be written and deployed by AI without any human involvement at all. The important point that the naysayers and the skeptics are missing is that AI is not coming to replace your job. It’s not a replacement function. It’s an augmentation function. It’s here to make you better at your job. And that’s not a bad thing, actually”.

Assuming AI makes engineers and teams more efficient, why don’t we see more and better software being produced? Could it be that engineering teams are focusing on building internal tools around AI, not products?

Steve: “I’ll turn it around and ask whether what we’re seeing is innovation dying at large companies? What if we are only going to see innovation in small places? This is kind of what happened when the cloud came out.

Also, look at Meta. It feels like the biggest company in the world right now, but back in the day it was one dude. When a new enabling platform technology substrate appears, you’re going to see innovation at the fringes because of the innovator’s dilemma [when successful companies fail by ignoring innovation elsewhere]. Big companies can’t innovate. They’re all running into this problem.

Big companies may have hyper-productive engineers who are producing at a very, very high rate, but the company itself can’t absorb that work. Downstream, they’re just hitting bottlenecks and these engineers are getting shut down and they’re quitting. So, I think what’s happening is that we’re all looking at big companies going, “When are you going to give us something?” And the answer is we’re looking at big, dead companies. We just don’t know they’re dead yet”.

6. Redundant values in the AI era

What are some beliefs from your earlier career which no longer apply because of AI?

Steve: “‘Engineers are special’. There’s one. Sure, we learned how to do something by hand that computers can do now. Kind of cool, I guess”.

But what about the “engineer mindset”? It’s not just coding that we do, right?

Read more

But no more. Today, a plethora of tools outpace Copilot in various ways, like Cursor, Claude Code, Codex, and Gemini CLI, and there’s also AI code review tools like CodeRabbit, Graphite, and Greptile, not to mention all the MCP integrations which plug into agentic tools.

So, for this deepdive I asked 10 tech companies which tools their engineers use and, crucially, how they made their choices from among all the options. These businesses range from a 5-person seed-stage startup, to one that employs 1,500 people and is publicly listed. All are anonymous, except for Wealthsimple and WeTravel. WeTravel has also kindly shared the most detailed measurement framework I’ve yet seen.

We cover:

1. Speed, trust, & show-and-tell: how small teams select tools: At places with fewer than ~60 engineers, tooling decisions are fast and informal: developers try them for a couple of weeks and those which “stick” win.

2. How mid-to-large companies choose: bureaucracy, security, and vendor lock-ins. At companies with ~150 engineers, adoption is considerably slowed down by security reviews, compliance requirements, and executive-level budgetary considerations.

3. Measurement problem: metrics are needed but none work. Every workplace struggles to prove its AI tools work, and common metrics like lines-of-code-generated are distrusted by engineers.

4. How Wealthsimple measured and decided. The flagship Canadian consumer fintech ran a 2-month selection process to choose an AI code review tool. Rolling out Claude Code to all engineers was a decision made by the CTO, backed with a mix of personal conviction, validated by usage data from Jellyfish.

5. How one company accurately measures code review usefulness. WeTravel built a structured -3 to +3 scoring system across five dimensions, with five engineers evaluating ~100 comments. They found no AI code reviewer suitable for their codebase.

6. Comparative measurements at a large fintech. A team ran Copilot, Claude, and Cursor simultaneously across ~50 PRs, scoring ~450 comments. They found Cursor reviews the most precise, Claude the most balanced, and Copilot the most quality-focused.

7. Common patterns. Developer trust drives adoption more than mandates, the Copilot → Cursor → Claude Code migration path is well trodden, and nobody has cracked productivity measurement yet.

The goal of this article is to showcase what tech companies of different sizes are doing, and to offer a few pointers on measuring and comparing the tools. It’s hard to do, but not impossible, as two in-depth case studies illustrate, below.

Don’t forget, what matters is to find tools that work for your team. During this research, I found vendors which are beloved by one company and loathed in other workplaces. There’s no single vendor that’s universally rated by every team in all contexts.

As always, I have no affiliation with any vendor mentioned in this article, and was not paid to mention any of them. I used to be an investor in Graphite, but no longer am. For more details, see my ethics statement.

1. Speed, trust, & show-and-tell: how small teams select tools

Decisions are informal and made quickly at the smallest businesses in our survey, with the decisive factor being how people feel about the tools. Trial periods are short, at around two weeks, and individual developers have outsized influence on whether a tool is adopted or binned, with the decisions spreading organically. Below are two examples:

Seed-stage logistics startup (20 people, 5 engineers)

The head of engineering at this startup describes their approach as high-trust and developer-led:

“We agreed to try new tools for 2 weeks and see how everyone felt. We didn’t use any hard-and-fast measurement. TLDR: I trust our devs and their opinion is a big part of this”.

Developers there suggest which tools to try and decide whether to keep using them or to seek alternatives. For AI code reviews, the team first tried Korbit for around a week but the tool felt “off”, so they roadtested CodeRabbit which “stuck” within a few days:

“Within a few days of using CodeRabbit I could tell the devs just liked it and were embracing the suggestions, unlike with Korbit which they ignored when they’d lost trust [in it].”

And that was that: decision made. As a small team, it’s easy to switch to something better and it only takes a single engineer to suggest it.

The broader tooling stack of this startup has evolved quickly over the last year:

• Figma for designs, which works nicely with Linear. The company has 5 devs and one UX designer.

• Linear for ticketing and collaborating across UX and development. The UX person creates Linear tickets alongside her Figma designs.

• Claude Code and Cursor for development, connected to Linear via MCP.

• Claude Code writes tickets: a recent change which is working nicely with CodeRabbit, as more context is passed downstream for AI code review.

“Show and tells” – where team members show colleagues their tooling setups during weekly team meetings and demos – are used by this startup to identify which tools do or don’t work:

“Our show-and-tell process greatly helps. There are so many new tools, skills, IDEs, etc, that it can be overwhelming. We all learn from seeing what others in our team are doing.”

The team makes a clear distinction between company-wide tools like Claude and CodeRabbit that everyone is expected to use, and devs’ personal environments (IDE choice, terminal setup), over which individuals have full autonomy.

By now, almost everyone has migrated to Claude Code, but six months ago the team was evenly split between Cursor and Claude Code. The head of engineering said:

“We had a dev for a while who wouldn’t use Cursor or Claude. We didn’t force him to, but it became clear that everyone else seemed to ship more code, whereas his quality wasn’t there”.

Series A startup (30 people, 15 engineers)

A staff engineer at this company says the team is split on Cursor versus Claude Code, with the latter gaining momentum. He also says code reviews cause headaches:

“One of our main challenges has been code reviews, as the quantity of code produced goes up, and quality used to go down, pre-Opus 4.5.”

They evaluated three code review tools: Cursor’s Bugbot (okay but not great), Graphite (not good), and Greptile (good). They’re now trialing Greptile for PR approvals, taking advantage of its confidence-scoring feature.

What works really well for this team is maintaining extensive Agents.md and Claude.md files, which are very handy because they’re used by:

• Claude Code and Cursor for coding

• Greptile for code reviews

These two files help maintain a single source of truth for coding-style guidance across the toolchain.

There’s praise for Cursor’s integration with Linear and Slack from a staff engineer:

“An alternative flow I personally like a lot is Cursor cloud agents, as their integration with Linear/Slack is very good. For smaller changes, this means code is not even checked out, but goes directly from the Cursor agents, to GitHub, to being deployed.”

Series D observability company (150 people, 60 engineers)

The director of open source at this place summarizes what’s stuck there:

“We’ve tried a bunch of things (Graphite, et al.) but the one thing that’s really stuck has been the company’s Claude Code subscription — it’s the most definite value-add. We pretty regularly bump people up a tier as their usage increases vs other tools that mostly sit idle”.

An interesting signal at this company is that non-engineers have jumped onto Claude Code. Product managers, solutions engineers, and technical account managers alike are using it more than the median engineer, and they’re handling customer bug reports by opening Claude Code PRs directly:

“They probably use Claude Code more than the median engineer! They’ve been able to handle customer papercuts [pain points] with Claude Code PRs”.

2: How mid-to-large companies choose

At companies with 150+ engineers, it’s not about how a tool “feels”. Instead, existing vendor relationships may be decisive, and there’s often pressure from the C-level (leadership team), as well as security and compliance matters to address.

There’s also the new challenge of how to coordinate tooling rollouts across several departments and potentially hundreds of engineers. This is where a decisive CTO can cut through red tape to achieve faster adoption. Our first case study covers how one fintech business did precisely that.

EU-based software company (500 people, 150 engineers)

This place’s experience is a cautionary tale of what can happen when a leadership moves on AI tooling without a plan for what comes next. A senior engineer there says:

“In the summer of 2025, our leadership team came back from an offsite and declared we were now AI-first, which translated to everyone receiving a Copilot Business subscription if they happened to ask for one. Problem solved, right?”

But it wasn’t, as the Copilot rollout was immediately met with questions about alternatives:

“Our company’s pre-existing relationship with Microsoft was probably key: we already had M365, and then they rolled out Copilot to all devs.

People immediately had questions about other tools. A few of us had used Claude Code that summer, while others had used Cursor or Gemini CLI.

The pace of new models and tools in the second half of 2025 left the leadership team completely unprepared. They had also not budgeted for anything other than $19/mo Copilot subscriptions”.

They got “stuck” and unable to approve any new tools for six months. The attempt to create a formal approval process is stalled, thanks to legal and IT being gridlocked, with the European Union’s AI Act causing concerns and governance questions:

“We are facing adoption challenges thanks to bureaucracy and EU regulation. I’m positive this process has led to developers using unsanctioned tools at their own expense.”

Meanwhile, their default Copilot setup uses GPT 4.1, a 10-month-old model. Many developers there don’t know if they can change the model or use coding agents. This creates a vicious cycle where the tool feels underwhelming, which suppresses adoption and makes it harder to justify further investment in better options.

Cloud infrastructure company (900 people, 300 engineers)

A principal engineer responsible for AI tooling at this company describes the constant push-pull between developer enthusiasm and executive scrutiny:

“We started with Copilot because it was easy to procure, since we were a Microsoft customer for M365. Then switching to Cursor took forever. Pricing keeps shifting. Meanwhile, execs read a doc and keep asking “why aren’t we on Claude Code?”

The answer to this also came from the exec team: pricing. Execs simply did not want to invest in the tools, and pricing remains a persistent headache. Claude’s team plan is ~$150/month, Cursor’s is ~$65, and this company’s C-level was not comfortable with going from Copilot’s $40/month to Cursor’s $65/month. The principal engineer also worries that costs will keep mounting, even with approval to move to Claude Code’s $150/month:

“Claude Code and Codex are definitely eating the costs right now… we all know that won’t last. If my execs push me on this, I will need to say — ‘okay, our developers got much slower in 6 months, but now we need to pay $250/month, per developer, to get higher limits’”.

Public travel company (1,500 people, 800 engineers)

A staff engineer at this business highlighted vendor lock-in as a primary concern:

“Our main concern is avoiding vendor lock-in with a single solution. With this in mind, I expect to continue evaluating AI tooling this year as things keep evolving rapidly”.

They rolled out GitHub Copilot last year and are now evaluating Claude Code as a replacement. They remain cautious, given that the per-engineer cost is steep with Claude.

Public tech company (2,000 people, 700 engineers, productivity space)

The engineering leader in charge of dev productivity at this business calls security the biggest challenge:

“The biggest hurdle for us is security. We are looking for some amount of compliance, and I’ve found dev tools startups aren’t prioritizing that until they are late Series A/Series B. This helps focus us and ensure that what we are evaluating has passed some muster in the industry, without us feeling like we’re late to the game”.

Unsurprisingly, the tooling selection process is more in-depth at companies of this size, with many vendors as options. Here’s how they go about things:

“There’s an amount of instinct involved in knowing how to prioritize vendors. Our process is this:

• What we’ve heard from friends and colleagues at other places

• Chatter on Twitter/Reddit/Hacker News

• Knowing how to cut through hype

Evaluation is more organized and beta trials are common, he says:

“Every tool has to move a metric. Those that directly impact a metric which we already care about get approved faster. The tools that could theoretically impact metrics, but don’t have directly-measurable impact, take more work. The weaker the metric story, the stronger the narrative has to be.

We like to capture at least two weeks of beta usage on a tool before making a call on expanding or ending it.”

3. Measurement problem: metrics are needed but none work

If there’s one theme that unites every company in this deepdive regardless of size, it’s the struggle to measure whether AI tools actually work.

Execs want data, but engineers distrust the data that exists. Meanwhile, vendors’ own metrics are mostly useless.

Among our research sample, the EU-based software company debated options and only found bad or worse ones:

• Using the ‘lines of code written by AI’ metric creates bad incentives, a sentiment shared by AI enthusiasts and skeptics alike at the company.

• Even if they were dead-set on lines of code, is there a way to isolate those lines of code which create business value?

There’s also the point that some of the most valuable uses of AI don’t lie in the writing of code, but in research, generation of ideas, debugging, etc, which makes measuring code generated by AI tools a dead end. Meanwhile, vibe-coded scripts and tools that never hit production can feel like real productivity breakthroughs.

In the end, this company chose lines of code generated by Copilot as the “official” metric, which met a predictable response:

“You can imagine how poorly this was received by devs! It doesn’t even account for the fact that this metric from Copilot is purely based on telemetry from specific IDEs. So, even if you’re using Copilot CLI to write code and maxing-out your premium requests, it will not be counted.”

The principal engineer at the 900-person cloud infrastructure company was more blunt:

“My engineering org is getting hooked on AI, but execs want metrics on value-add. I don’t want to push vanity usage metrics just to justify spend, but outside of vanity metrics, I have nothing of value to show!”

This principal engineer dismissed developer-productivity vendors’ own measurement approaches:

“I talked with DX and one of the other vendors, they are just DORA+Velocity metrics combined with anything they can get from APIs of Cursor, Claude etc. Sure, this all looks good on paper: Team A is faster and they use AI. But is AI usage and speed a real correlation?”

A fundamental question remains unanswered, says the principal engineer:

“How can we make effective use of our AI agent subscriptions? So far, in my experience, there is no answer to this — not even the hint of one”.

4. How Wealthsimple measured and decided

Wealthsimple is a Canadian fintech company, employing about 1,500 people, around 600 of whom are engineers. I talked with CTO Diederik van Liere about how they choose AI code review and AI coding tools. For AI code review tools they run a thorough measurement process, and for AI coding tools it was more of a push from Diederik. He shared exclusive details on their exact measurement process, and how they landed on Graphite for code review and Claude Code for coding:

Choosing an AI code review tool via a “shootout” process

Read more

On the first Monday of this year, Balint returned to Craft with a new AI tool they’re calling “Craft Agents”. It’s his take on a more opinionated Claude Code, built on top of the Claude SDK. The company has mandated every engineer and non-engineer to try adding the new tool to their workflows – and they say the results have been jaw-dropping.

During January, the Craft team has completely changed how they work, and now feel more productive than ever. Also, non-engineers are hooked on using AI with Craft Agents.

Is it a sign of how mature startups in tech are changing how they build software, using AI tools? That’s one topic tackled in this deepdive.

First, some background. Craft has more than 1 million active users, over 50,000 paying customers, and an engineering team of 20 with a median tenure of nearly four years. They care deeply about engineering excellence and product quality: the startup won Apple’s Mac App of the Year award in 2021, and built their own, custom rendering stack to boost user experience above the competition.

Craft’s AI-first makeover is not a “YOLO” (you only live once) approach like a young startup might try as a way to create some hype. It’s an experienced engineering operation deciding that AI tools have reached an inflection point which means the company needs to change how it works, or be left behind.

My family ties to Craft mean I’m “in the loop” on things at the business, and it’s this insight that makes me confident this is worth looking at. Balint shares details below that are relevant for anyone in tech currently wondering how “AI-native” startups work in the age of AI, which aren’t AI labs or AI vendors.

Today, we cover:

1. Three years of AI experimentation with no “stickiness.” The engineering team was an early adopter of AI, but consciously avoided the “Copilot temptation” of shipping substandard AI features which lack true purpose.

2. Unexpected breakthrough: a “visual Claude Code” experience. Nontechnical users of Craft were struggling with a terminal experience, so Balint imagined what Claude Code with a UI and strong opinions” would look like. He then built this vision in two weeks using Claude Code.

3. How non-engineers use “visual Claude Code.” Customer support: the heaviest user of Craft Agents and Craft could soon ditch Zendesk, as a result. Marketing: building webpages with zero engineering input. HR: building new Bamboo HR integrations on their own.

4. New way to build software? Fast iteration without code reviews, rejecting pull requests but “weaving in” their ideas, and not using SDKs much in development.

5. Dramatic change in dev teams. Difficult migrations take a week instead of months, one-person responsibility squads, while some devs struggle with fast change, and even quit.

6. AI changes software that’s worth paying for. Enterprise-only capabilities could become standard consumer-grade offerings, and AI makes it easier to switch vendors. For example, Craft may soon pick an API-first vendor, instead of using Zendesk.

7. Predictions for the tech industry: the death of pull requests for open source, the rise of “remixing”, devs who value impact the most will thrive, and perhaps hiring freezes while companies figure things out.

Note: I have no financial affiliation with Craft Docs, own no shares in the company, and have not been paid to write this article. Obviously, there are family ties, which in this case enabled me to convince him to share more details than he originally intended! More in my ethics statement.

If Balint’s name is familiar, it’s because we did a podcast episode covering his unconventional, design-first engineering approach at Craft. You can meet Balint and myself at The Pragmatic Summit, coming up on 11 February, in San Francisco.

1. Three years of AI experimentation with no “stickiness”

Craft is a tool for docs, notes, tasks, and anything else to do with capturing ideas. Launched in 2020, its signature characteristics are low latency, high performance, seamless movement between devices (iPhone, Android, iPad, Mac, web, and Windows), and a delightful user experience.

The team has long experimented with AI while avoiding the hype by refusing to ship AI features that would be gimmicks at best. Below is a timeline of their AI-related activity:

2022: basic AI assistant

They launched an AI assistant, “Craft AI Assistant”, two days before ChatGPT launched in 2022, and weeks before Notion’s own AI assistant became available. That early assistant was pretty simple:

• Used OpenAI’s GPT-3.5 API

• Ran on an AWS gateway acting as an authentication proxy

• Used HTTP streaming resources

It was also limited: the context window was only 4,096 tokens and prompts needed to be squeezed into that limit. The assistant was also “one-shot”: users asked a question, it replied, and that was that! There was no more dialogue.

The AI assistant drove substantial user growth. Many users downloaded Craft just to play with it, but it quickly became obvious that the assistant was not “sticky”, which means it was simply not genuinely useful for enough users. After an initial engagement spike, people didn’t use the AI assistant repeatedly.

2023-2024: resisting “Copilot everywhere” slop temptation

The team kept experimenting with ways to add AI features that were useful for Craft, and ran experiments such as knowledge-based search using RAG, summarization, and other retrieval features. But nothing really stuck with users.

Balint was keen to avoid the “Copilot fail” moment he observed at Microsoft, where the tech giant rolled out the not-particularly-useful Copilot across its entire product suite, only for customers to quickly come to associate the shiny new Copilot with uselessness. Balint said:

“We knew we had one shot at showing users that AI is useful for them. And until we had a ‘wow’ moment ourselves, we would not go all-in on AI. We didn’t want Craft to have the same negative associations with AI as Copilot has”.

During this time, the company did a lot of experiments. In September 2023, they held an in-house hackathon focused on AI, for which I was a judge. As I recalled from the experience:

“This [hackathon] was one of the best I’ve seen, with around 80% of staff taking part, including designers, product managers, and customer support representatives within engineering teams. The winning team consisted of some customer support folks and engineers, who built tooling which improved the customer support team’s workflow by solving some persistent pain points.”

There was no shortage of enthusiasm, but that by itself doesn’t create a great user experience.

December 2024: reasoning models mark turning point

Little over a year ago, Balint rolled his sleeves up and spent two weeks experimenting with the latest reasoning models at the time, like OpenAI’s GPT-4o.

This led to his first “wow!” moment as a software engineer with the 4o model. A highly requested user feature was for Craft to recognize shapes which users draw by hand. This was a problem Balint estimated would take a few weeks to properly implement. So, he gave it a go with the 4o model and to his amazement was able to implement it within a day by learning on the 4o model. The feature was shipped, users approved, and it marked the first time Balint had to admit they couldn’t have shipped a feature without AI.

After that, Craft’s approach with AI became more ambitious. He assigned five senior folks from among the 20 engineers to experiment fulltime with how AI could be added to the product. They spent six months building a mobile-first agent called Chaps, similar to Peter Steinberger’s very popular Clawd bot, which ultimately was never released to the public.

By October of 2025, Balint concluded that agents had started to work pretty well, but they still weren’t ready to be a standalone product. Meanwhile, the five-strong team had built up knowledge about tool calling and orchestration, which they ported back into the Craft AI assistant.

In December 2025, Craft’s AI assistant got large updates which took it from being a “meh” experience for most users, to one which was helpful and popular. This new, improved AI was a lot more “sticky”, but still wasn’t the “wow!” experience Balint wanted.

2. Unexpected breakthrough: a “visual Claude Code”

That month, Balint wanted to use more top-tier models like Opus 4.5, via Claude Code, so he built a prototype terminal UI (TUI) on top of the Claude Agent SDK: a framework for building production-ready agents with Claude Code as a library.

This tool, named Craft Terminal, connected Craft Docs as a data source to Claude Code via the Claude Agent SDK, and worked surprisingly well. Balint used it to organize his own messy Craft Docs workspace, search in it, and ask for insights. It worked really well; much better than any previous knowledge base lookups the team had built.

The customer service team started using the Craft Terminal, but needed more sources than just Craft Docs for their work. A typical Customer Service flow resembles this:

• Look at an incoming Zendesk ticket

• Check if there is a runbook for this type of ticket in Craft

• Run the runbook, and often access other databases

Balint added the concept of “Sources” to the Craft Terminal: the ability to add other data sources – APIs, databases, MCP servers – on top of Craft Docs. The customer service team then started creating automations for common support tasks like sending discounts to emails on academic domains. The implementation of using APIs for sources is pretty clever: an API source acts like an MCP server to the agent, but Craft Agents hijacks the request and adds the credentials needed without the agent having access to those sensitive credentials.

However, the terminal interface was not beloved by Craft’s users. On December 18, he shared the Craft Terminal Agent with 15-20 external beta users. Feedback was positive and they liked it, but there were complaints about the terminal environment:

• Multitasking was painful

• Reviewing complex plans was awkward

• The whole experience felt “locked away” to most users who are non-technical and not developers

As a designer and UX lover at heart, Balint felt the terminal was holding the tool back. He asked:

“Can we turn the terminal into something that’s more like an email client or a Slack client? Something that is more natural to use for anyone.”

So, last Christmas during the holidays, Balint decided to build a UI on top of Claude Code – and gave himself two weeks. Previously, it took him up to four months when he started Craft to build a v1 of Craft Docs that felt good. But this time, he did it all with AI, using Claude Code. As an additional challenge, he used Electron to build it, a tech stack he’d not used before.

Embarking upon this project, Balint decided that if he could build a polished application in an unfamiliar tech stack within two weeks, it would be proof AI was capable of delivering a step change for Craft. If so, he would push everyone at work to start using AI coding tools. If the project failed, at least he would still learn about these models’ progress.

In the end, Balint hit his self-imposed milestone: by January 5 of this year, he had completed a polished Craft Agents app that ran on Mac, Windows, and Linux, thanks to it being built on top of Electron. The result is a far more friendly experience of using Craft Docs. Below is how it looked with the terminal interface reimagined as a more “email-like” UX:

This tool was more than just a nice UI on top of Claude Code. Balint also included:

• A concept of “sources:” data sources to connect to the agent; anything from databases, through to APIs and MCPs.

• Support for running parallel agents, visualization, and switching between them.

• Support for workflows via a label system.

• A permissions system defining whether data sources can only be read, or if they can also be written to. A readonly agent run mode, “ask to edit” mode, and a mode for making edits without asking.

• Expanding Claude Code’s “skills” concept.

• Theming options for the tool (added because there was time to spare at the end).

Internally, Craft Agents was an instant hit. During release week customer support, marketing, and HR all jumped on the tool, and people started automating previously manual tasks. In an unexpected development, non-technical folks started using Craft Agents more than devs!

Most devs were satisfied with Claude Code’s interface, and a few started to use Craft Agents because of the nicer multitasking (with parallel chats), and the interface being much easier to scroll, read, and view for things like long code changes.

On 19 January – almost exactly a month after the first line of code was written – Balint open sourced Craft Agents under the Apache License 2.0.

3. How non-engineers use “visual Claude Code”

I wanted to understand why Craft’s non-engineering teams have been so enthusiastic about using Craft Agents, and what’s changed as a result. As mentioned, the biggest adopters have been in customer support, so I sat down with Tamas Fazekas, head of customer support at Craft, who explained:

Custom workflows are the core of the team’s usage. They’ve built workflows for:

• Bug triaging.

• Bug processing: incoming user reports are run through the Craft source code for validation, for efforts to identify root causes, and to add context for the engineering team.

• Daily updates which summarize the work done by a user.

• Education: institutions can use Craft Docs for free. This workflow checks if a domain is an academic one. It approves legit domains and flags up fishy ones or request patterns.

Let’s take a look at a few workflows using this tool:

Triaging workflow with agents

Parallel agents are helpful for processing triage tasks in bulk. When there’s a large number of customer tickets, the tool automatically kicks off parallel agents to go through triage:

When triage is complete, the agent provides a report to the customer support person who kicked off the work.

Data enrichments are used in almost every workflow. The customer support team defined a skill called “Get User Data”. It uses a Craft backend API to look up the user profile based on the user’s email in the Zendesk field, and adds the user’s plan type, billing status, feature flags (feature access), and usage metrics to the ticket.

The support agent no longer needs to open a separate tool to get all these details.

Bug processing workflow: issues reported faster, automatically to engineering

The “bug report processor skill” has greatly sped up dealing with bug reports. It’s similar to Claude Code’s skills feature. The skill defines data sources (Zendesk and Linear) and defines the required output (a structured analysis with tags and categories). It instructs the agent to:

• Identify the platform and area which the bug belongs to, and tag it.

• Crosscheck with Linear (Craft’s issue tracking system). If there are similar issues, link to that issue. If no similar issues are found, create a new issue and assign it to the relevant developer team.

• Do a technical root cause analysis.

• Draft a ready-to-send response for the customer.

• Create engineering tickets for developers, including pointing to code references, where applicable.

Here’s how this skill is defined:

Below is an example of how it updates the Zendesk ticket with a suggestion on how the customer support agent should respond:

The customer support team have been pretty delighted about how their workflows have gotten better. Here’s Cusomer Experience (CX) lead, Peter Sajevics, on what’s changed:

“A massive win is we have to escalate so much less often than before! In the past, we would often have to ping engineers to check if a bug was a possible code issue. We no longer have to do this: the agent does it and automatically escalates when needed.

We no longer need to try to reproduce a bug on the latest version of the app which was reported on the old version because the code review lookup can tell us if anything relevant changed in the code paths.

We no longer need to look up subscription details in a separate tool because the agent fills in this detail.

I feel that tickets which used to take 20-30 minutes to process are down to 2-3 minutes.

My sense is that the biggest win is not even that we spend less time on each ticket, but that we can now process a much larger volume of tickets than before. Parallel processing of tickets is now much easier”.

Skills built by customer support

During just two weeks of using Craft Agents, the customer support team have built these skills:

• /triage: categorize tickets with parallel workers

• /bug-report: process bugs and create Linear issues

• /education: handle education license requests

• /daily-report: generate an overview of the current ticket queue with Zendesk as a source

• /get-user-data: enrich the context with user details from Craft’s database

• /feature-request: process feature requests, gather more info when needed, store requests in Craft for processing later

Use cases in other non-engineering teams

Today, all non-engineering teams at Craft are heavy users of the tool:

• Marketing: building websites without devs. The marketing team builds lots of websites: pages to share details about launches, feature descriptions, comparison pages, and more. Previously, a web engineer would pair with the marketing team on rotation, and engineers dreaded this as unchallenging work, while marketing felt they had less support. But a marketing intern started to use Craft Agents to build new web pages and tweak existing ones, and now the whole team does.

• HR: automating tedious work. An HR person built a Bamboo HR plugin to handle aged-based holiday allocations necessary for Hungary. Other automations include generating files compatible with the payroll system, which used to be done manually.

• Finance: automating personal workflows. One person in finance built a tool that exports Revolut’s business account into CSV, then cross-references it with the employee Slack channel where employees post invoices, and matches these to create a format to submit to the accounting tool.

Additional permissions system to avoid accidents

The Craft team uses this tool for real, customer-facing work. But of course, LLMs are nondeterministic and can make mistakes which go on to cause problems in production. To deal with this, Craft Agents has an additional permissions system. Sessions can run in one of three modes:

• Explore: readonly. The default mode for agents. They can read all sources but not write to them.

• Auto: the agent can write to data sources without user input.

• Ask to Edit: needs prior user confirmation. This mode is typically only used for onboarding – users often use Explore to discuss, and Auto to get work done.

When starting a new session, the default mode is “Explore”.

Engineers use Craft Agents – but less than non-devs

Some engineers have onboarded to Craft Agents, but most prefer Claude Code for the majority of product development work. The reasons that a few engineers prefer Craft Agents is its easier overview of parallel agent execution, and a more pleasant interface for reviewing lengthy code changes since scrolling is more convenient than reviewing on a terminal.

Creating custom themes

Craft Agents comes with several pre-built themes, like Ghostty, GitHub, Tokyo Night, Rosé Pine, and a dozen more:

You might think this is enough choice, but it’s not! One customer support person instructed Craft Agents to design a new theme imitating the Matrix. The agent obliged, and custom themes started spreading in the office. Here are some popular ones:

“Remixing” Craft Agents is already a thing

Craft Agents is open source which means anyone can fork the code and make changes. The Apache 2.0 license is permissive: you can fork, modify, and distribute your fork however you want. The only constraint is that you must include a copy of the Apache 2.0 license text itself with the distributed work.

Forking and modifying for personal needs is already happening. Here’s an example from researcher, Lisa Skorobogatova, who remixed Craft Agents by instructing the agent to make changes to its own code to support projects, and allow drag-and-drop to move chats into projects:

4. New way to build software?

Until now, software at Craft was built the usual way:

• Plan changes ahead of time when it makes sense to.

• Write the code.

• Code review.

• Deploy to production.

But with Craft Agents, engineering is also done differently:

Read more

The Pragmatic Engineer is not a typical publication, and so this is also not a typical role. I’m looking for someone to help research and compile Tuesday deepdives like the one on Cursor, on Claude Code, on Stripe, and many others. This position will include directly talking with engineers at interesting companies, researching both public details and details made exclusively available to us, and compiling what we learned into detailed reports.

If you’ve worked at startups or Big Tech for a while, would enjoy working full-remote, keeping up with the cutting edge of the industry sounds interesting, and you’d enjoy doing something that can start as part-time: read more and apply here. Applications close Monday, 26 Jan.

One trend in tech is that more startups are hiring for “product engineers” or “product-minded engineers”, who can implement products and also come up with strong product and feature ideas, then build them. This trend of engineers’ involvement from the ideas stage through to shipping looks set to accelerate with AI tools generating ever more code.

My recent analysis of what happens when AI writes almost all the code mentioned that nimble startups were already recruiting “product engineers” who can create their own work, and act as blends of mini-product manager and software engineer. I said this indicates that being more product-minded could become a baseline at startups because it’s increasingly important to specify what an AI tool should build.

But how do you get better at being a product engineer?

Obviously, pairing with a product manager, staying close to the business, and finding a mentor who’s a great product engineer are strong options. But if these aren’t all available in your workplace, there’s now a book dedicated to the topic.

Entitled “The Product-Minded Engineer”, it’s written by former software engineer and current product manager, Drew Hoskins, and published by O’Reilly:

A few years ago, I published an article named “The Product-Minded Software Engineer” which offers tips for software engineers to grow their “product muscle”, and it’s timely that a fellow engineer has invested in writing a guide about this increasingly pertinent subject.

After hearing Drew was working on his book, I got in touch, reviewed a draft version, and asked if he’d consider sharing an excerpt in this newsletter. Graciously, both Drew and O’Reilly agreed. Drew will also be a speaker at The Pragmatic Summit in San Francisco next month, on 11 February, discussing tactics for leading product engineering teams in an AI-native environment.

In today’s issue, we cover:

1. Author’s background. Twenty years as a software engineer at Microsoft, Facebook, and Stripe – and today as a product manager at Temporal.

2. Writing the book. Why create this guide now?

3. Importance of good errors and warnings at product-level. Excerpt from Chapter 3: “Errors and Warnings”, about why designing the right approach to errors has a massive impact upon products used by developers and nontechnical users, too.

My usual disclaimer: as with all my recommendations, I was not paid for this article, and none of the links are affiliates. See my ethics statement for more.

1. Author’s background

With experienced tech professionals who cross over into being published writers, I find it’s always useful to understand something about their background, and Drew has an impressive one spanning more than two decades:

• Microsoft: Software Development Engineer (2002–2009). Worked on the C++ compiler backend, static analysis tools, and the Windows UI developer framework.

• Facebook: platform/product infra engineer (2009–2015). Worked on the Facebook API and the initial version of the Facebook App Center. Then founded and led a product infrastructure team building the core data APIs for internal engineers. Still used pervasively today, EntSchema turbocharged Facebook’s Ent framework with codegen, reflection, and a sandbox experience. This later led to the popular open-source Ent framework in Go.

• Oculus: Software engineer, E7: Senior Staff-level engineer, (2015-2017) Led the effort to rebuild Oculus’s web platform to Facebook’s infrastructure, after the social media giant acquired Oculus. Tech lead for Oculus’ Platform SDK.

• Stripe: Staff+ software engineer (2018–2023). Tech Lead on the Stripe Connect product, then founded and led the Workflow Engine, a framework built on Temporal.

• Temporal: Staff Product Manager (2024–present). Product manager at Temporal, an open source durable execution workflow service, working on developer experience and agentic orchestration.

Drew went from working on APIs and platform teams, to leading large engineering efforts, and starting new teams and initiatives in his workplaces – before heading over to the “dark side” of product management at a developer tools company. To me, Drew seems the ideal professional to write such a book because he has plentiful experience of working as a software engineer when it was required to understand the business, and he’s now a product manager working with fellow engineers on Temporal.

2. Writing the book

Drew told me more context about this project:

What was the trigger to start writing this book?

“I had written a book outline on usable API design for O’Reilly, and the main themes were product-thinking and user empathy; topics I’ve long wished more engineers engaged with. But Louise Corrigan at O’Reilly liked those themes more than the API topic, and suggested I make product-thinking itself the subject of the book.

I liked how this pivot mirrors my personal career journey; of my interest in API design blossoming into a broader interest in products and users”.

How long did it take to write?

“It was an 18-month process end-to-end, starting the day after I joined Temporal – so that was an intense period! I upgraded a lot of three-day weekends to four-day weekends, and also did some writing on cruise ships. I wrote the whole thing myself, but used friends, and especially Claude, for research. I also sought lots of Alpha and Beta feedback because I believe “it takes a village”. The two biggest inputs were my own career experience and concepts from the design and product communities. It’s well-known stuff, but nobody bothered to inform engineers about it”.

Who’s the best product-minded engineer you worked with?

“John Carmack, with whom I overlapped at Oculus. He’s amazing because he’s super-deep technically in areas like graphics, yet doggedly pursues the most important product goals.

One year, he decided the community needed to level up in building performant VR apps for a mobile compute envelope, so he mentored the entire community in marathon sessions. Another time, he decided the Oculus platform needed more great apps, so went to Netflix and Mojang, worked with those teams, and heroically brought the Netflix and Minecraft VR apps into existence”.

What’s your advice for mid and senior-level software engineers who want to be more product-minded?

“My suggestions:

• Ask “why” a lot. Don’t expect to always get clear answers, not even from EMs and PMs.

• Switch your viewpoint. Go from the system level, to the user lens, and then back again.

• Use scenarios. Simulate and sequence user interactions until this becomes routine. Writing scenario tests is often a good start.

• Customer support. Spend time on user support and think about more permanent fixes while you engage and unblock users”.

As a product manager, what can devs do to be seen as product-minded and be invited to do more product work?

“I try to have devs help me author use cases/scenarios. I also invite them to come along on customer calls if they want. If they have an idea, I ask them to justify it with scenarios. If they start throwing use cases back at me without prompting after a couple of months, I know they’re on the journey”.

What is one technique for using AI tools that you’d recommend devs try, in order to be more product-minded?

“It’s easier than ever to gather user signal with AI tools. My team at Temporal has a Claude Code skill for gathering customer signal: the tool searches our internal Slack, community Slack, Miro Insights, GitHub issues, and Gong, and aggregates it all into a report with lots of links to chase down customers and requests. Many of those tools in turn have AI assistants that make all this much easier to do!”

3. Excerpt: “The importance of good errors and warnings at product-level”

The excerpt below is from “The Product-Minded Engineer”, by Drew Hoskins. Copyright © 2025 Drew Hoskins. Published by O’Reilly Media, Inc. Used with permission.

From Chapter 3: “Errors and Warnings”

The Value of Diagnostics

Crafting well-structured diagnostics with useful messages is an incredibly valuable and high-leverage way to spend your time. For many applications and platforms with complex and open-ended inputs, diagnostics are the primary interface—the vast majority of the user’s time will be spent dealing with errors and progressing to the next one.

Filling out electronic forms is all about being told about your missing or malformed input. My coding time is at least half dealing with errors and lint rules. Even writing in a word processor has become a constant process of looking at underlined text and being asked to proofread or rephrase.

And yet, as we design software, because errors often don’t appear in screenshots, marketing materials, or API method listings, they can be out of sight and out of mind.

Autonomous agents shine a bright light on this problem. They are now regularly presented with error messages resulting from their actions and instructed to correct their mistakes based on them. If the message isn’t sufficiently helpful, they fail at their task. The process of trying different things is slow and costly. Because agents are billed based on usage, the costs are directly measured.

Tip: Diagnostics may be the most important interface of your product.

Scenarios for Diagnostics

When considering errors, warnings, and their associated messages, it is essential to think about a broad range of scenarios, starting with identifying edge cases, to understanding how developers can automate reactions, and how end users will understand and act upon them. Improve your ability to understand your users’ knowledge, generate user stories, and simulate user interactions, and you’ll improve your diagnostics. For users, we provide contextual and actionable errors. For developers, we carefully select our error types, codes, and metadata so that those who receive them can recover gracefully.

In the rest of this chapter, you’ll learn how to craft refreshingly useful warnings and errors. We’ll explore how to:

• Understand the scenario—the persona who will benefit from the error and their situation

• Provide enough context to our users for them to understand the error

• Provide actionable error messages that suggest what to do about the problem

• Choose error codes and types carefully to allow upstream developers to serve their users

• Raise errors at the API or UI layer so that messages can be written with full context about what the user’s trying to do

• Shift left; that is, fire errors as early as possible to speed up your users, and before bad things happen

In Chapter 8, I’ll address how to list out edge cases to figure out what errors to check for in the first place. For now, I’ll focus on crafting errors once you already know what they are.

Categorizing Error Scenarios

When writing errors, you need to make a few main choices. First, a user-facing choice:

• What is the error message?

There are also choices of concern to developers so that they can catch errors and automate responses:

• What is the error’s class or code?

• What metadata is needed to pinpoint the problem?

Thus, when you craft errors in virtually any application or platform, you must think of two categories of user scenarios: the human one and the programmer one.

There are further divisions in the developer scenarios: are you communicating with members of your team who work on your codebase, or those from other teams or companies? This is especially important if you are building an API or service where upstream developers can catch your error and act upon it.

So, the first step is to pitch your message to the right person in the right circumstance. We’ve all seen errors that didn’t seem to be meant for us, such as when websites show code listings to end users. To determine your audience, start by deciding your error’s category. For our purposes, the five shown in Table 3-1 cover most cases.

Start by mentally categorizing any error you write. This gives a huge clue as to who you’re talking to—your own team, other developers, or users—and when the errors can be fixed—at runtime or during development. This will help you write with the right vocabulary and suggest helpful actions (see Table 3-2).

These five types of scenarios reveal drastically different strategies. For example, if an assertion triggers in production, it’s usually catastrophic. If the code is in a state the authors didn’t foresee, it will lead to unpredictable behavior—most likely in the form of a crash or a poor error message. Occasionally there are worse consequences, like data corruption. In some programming languages, assertions are stripped out in production to optimize their execution, meaning that you shouldn’t rely on them for anything load bearing. In no case is the end user persona expected to interact with them successfully.

For some applications, all end users are not the same; in which case, messages should be tailored to each persona. A classic example is a Preconditions Not Met error caused by the user not having the necessary access. Is the user an administrator or an end user? This determines whether we will provide them with direct instructions or instruct them to contact an administrator.

Knowing your personas will help you speak to the user’s ontology. (Ontology was defined in Chapter 2 as a structured graph of known concepts.) Consider “PC Load Letter,” [a reference to a segment before this excerpt] which tried to ask users to reload the printer’s paper tray. It was actionable—it told the user to load the paper—but it failed because it was speaking to the wrong persona. “PC” stood for “paper cassette” and “Letter” referred to a size of paper— 8.5”x11”. Perhaps instead, they should have labeled the paper trays A, B, and C and said, “Reload tray B.”

Categorizing Errors in Practice

Let’s work an example to show how to use product thinking to categorize errors.

Which of the five categories does a divide-by-zero—in Python, a ZeroDivisionError —fall into?

Imagine you are writing a method to compute the average value of an online metric over a time window.

Look at the return statement. If this method threw a ZeroDivisionError when the metrics array was empty, callers would be quite confused—they’d need to know the innards of your function to understand.

Tip: Users and developers should never have to understand your implementation to understand an error.

Thus, unless your code is literally a calculator, a divide-by-zero error is an Assertion, designed to be found at test time and telling your team that the code needs improvement. Avoid it—do some upfront validation before attempting the division.

So, we’re going to validate, but what scenario category would that validation fall under? The circumstance that led to the len(metrics)==0 condition could have been any of those listed in Table 3-3.

As I’ll discuss in the next section on messages, you’ll want to suggest different actions in each of these cases and therefore will need distinguishing checks in the code. Further, you will need to perform these validations at a moment when you have the necessary context.

In this section, we categorized diagnostics as either interacting with developers or end users and distinguished between scenarios that were actionable at runtime and those that were actionable only during development. Next, we’ll build on this to author awesome messages.

Warning and Error Messages

Writing diagnostic messages combines system thinking with user thinking. Know precisely what happened, but shift to your user’s perspective. Explain what they need to know in terms they understand. Otherwise, obscure warnings like LaTeX’s “underfull hbox (badness 10000)” will result.

Users seeing a diagnostic will want to know two things:

• What precisely happened to cause the error, in terms from the product’s ontology? This should help them know the impact of the failure and provide clues as to how to remediate it.

• What can they do about it, if anything? Actionable diagnostics will directly help them accomplish their task.

Let’s tackle those two goals one at a time. But first, let me introduce an example that will thread through the next few sections.

Case Study Introduction

Channelz is a fictional software as a service (SaaS) company building a workplace communication tool like Slack, Microsoft Teams, or Discord.

Elise works on the API team, and her teammate Deng is the tech lead.

In Channelz, one can write direct messages to coworkers or send them to “channels,” which are groups of employees organized around a specific topic; the API engineering team might have a channel called #team-api-eng. Elise’s user handle is @elisek and Deng’s is @deng.

Channelz is building out an API that can be used to send messages from bots, either directly to users or to channels. Their customers want to use it to send various notifications.

Before coding, Elise sketches a quick developer interface design and shows it to Deng. Channelz messages can go to a set of individuals or to a channel to alert employees when something has gone wrong or a job has been completed.

The method in the Python SDK they ship to customers will look like this:

She sketches some use cases for Deng:

Deng looks at Elise’s design and asks her to list failure scenarios as well. Elise has shown successful usages of the API, when callers already know what to do, but what about before then? If their users’ coding session is a journey, Elise has shown only the end. It’s as if somebody asked for directions with an online maps search, and she responded with only a pin on the destination.

Elise comes up with a few scenarios. (I will teach probing for edge cases in Chapter 7. For this chapter, I’ll skip that step.) She raises one important scenario that we’ll obsess over here: what if the user or channel passed into the API is invalid?

[We now skip ahead to the middle of the chapter, skipping through the section titled Provide Context.]

Make Error and Warning Messages Actionable

In many circumstances knowing what happened is only half the battle. Users often need to be given suggestions or told what to do. And for read operations—and increasingly with AI—you can even correct the mistake for them, as with Google’s “Showing results for: [correction]” feature, as well as coding or writing assistants automatically fixing your code or language.

We’ve all spent countless hours of our lives dealing with error messages, figuring out what to do, sometimes discovering after much investigation that the fix is simple.

In this section, you’ll see how to routinely improve your diagnostics. To achieve this, you’ll need to empathize with your audience, starting with the scenario categorization we did previously.

Returning to our Channelz example, suppose you called the API:

bot.send_message(message=”The sky is falling!”, channel=”@barnyard- friends”)

and got this error message:

Cannot deliver a Channelz message to channel ‘@barnyard-friends’: channel does not exist.

Can you tell what went wrong? It may take a bit to figure out, and if you’re not super familiar with Channelz nomenclature, you may not realize that channels are prefixed with #, not @.

It would be better to give our users a couple of options for what to do:

Cannot deliver a Channelz message to channel ‘@barnyard-friends’: it is prefixed with @. Did you mean to pass it into ‘users’? Or did you mean ‘#barnyard-friends’?

Channelz could even query accounts to see if #barnyard-friends exists—and is public to the user—and show:

Channel @barnyard-friends is prefixed with @, but we found a channel, #barnyard-friends. Is that what you meant?

By eliminating chunks of “search space” for your users, suggesting actions can save hours or prevent them from giving up and churning out of your product. Even a simple “try again in a minute” after a System error can increase completions of workflows.

Sometimes, the advice gets complicated. You may need to introduce users to some concepts, like the concept of channels, or guide them through a number of decisions. Where appropriate, link out to documentation dedicated to resolving that error. For example, if this case were more complex, you might do:

See https://channelz.io/docs/errors/invalid_channel to learn how to resolve this error.

Raise Errors at the Interface

If it were straightforward to write good error messages, engineers would do it more often. One of the main reasons they don’t is because it often takes diligent code organization to pull it off.

To write the best error messages, you need two pieces of information: What happened in the system? And, what was the user trying to do? The trouble is, in real life a different piece of code has each piece of information.

One place, near the API or UI boundary, knows who the user was and what they were trying to do. It is best placed to tell the user what to do without mentioning implementation details the user won’t be aware of.

The other, deep in the bowels of your processing code, is where the problem happens.

This came up when we were categorizing errors. When we divided by zero, Python’s division operation knew the exact law of mathematics being violated, but had no idea how the division was being used and therefore no hope of giving a good error.

The best place to raise errors is usually at the interface between the system and the user, where we can combine the bottom-up and top-down knowledge. (An alternative is to pass all the user context down, which I’ll explore in a later section.)

Generally, people raise at the interface in two ways:

• They raise errors proactively, using upfront validations at the API boundary.

• They use error handlers to intercept a lower-level error and repackage it into an appropriate form.

To illustrate each, let’s return to Channelz.

Upfront Validations

The Observability team at ChickenLittle has another problem. The function powering their on call messages is alert_team:

get_team_metadata reads a database of on call rotations that people at ChickenLittle set in a user interface at https://corp.chickenlittle.io/oncalls/.

What happens if team_metadata[oncall_user] is invalid? Recall that send_message will raise this error:

Error: Cannot deliver a Channelz message to ‘[@gooseyloosey]’ because ‘@gooseyloosey’ has been deactivated.

This is accurate, but when Goosey Loosey left the company, users couldn’t figure out how to fix the problem. So the Observability team added a validation upfront to tell folks what to do:

When a user is missing, this method is here to help:

Cannot send a Channelz alert to team ‘barnyard-friends’. On-call user ‘@gooseyloosey’ does not exist. Update your oncall rotation at https://corp.chickenlittle.io/oncalls/barnyard-friends.

This higher-level API alert_team captures the intent of the caller much more completely than send_message, which is a powerful basis for providing great errors.

Tip: Throw errors from the outermost layer of your API or application code where you can capture the user’s scenario.

The approach of upfront validation did the trick here, but it wasn’t perfect. Can you spot any issues?

Let’s look at an alternate technique of repackaging errors that your dependencies raise and see if it works any better.

Repackage Errors

There were two problems with the upfront validation. It’s expensive—it uses an extra roundtrip with the Channelz API. More important for our purposes in this chapter, the validation would need to duplicate edge-case-checking code already being done inside the Channelz API, such as checking whether @gooseyloosey’s account was deactivated. That info is still actionable—suppose the employee changed their Channelz handle because they changed their name. That would warrant a separate fix.

Since she’s given such good support before, the Observability team complains to Elise. They would rather write something like this:

Notice the from error clause in the last line. This is Python’s way of expressing “chained exceptions,” which preserves the inner error. Many programming languages have similar mechanisms. This would be the output:

ChannelzUserNotFoundError: User @looseygoosey’s account has been deactivated.

The above exception was the direct cause of the following exception:

ValueError: Cannot send a Channelz alert to team ‘barnyard-friends’.

On-call employee ‘@looseygoosey’ does not exist.

Update your oncall rotation at https://corp.chickenlittle.io/oncalls/barnyard-friends.

This is a longer message, but it contains everything the user might want to know. Observability asks Elise for specific errors in Channelz’s SDK so that they can do this.

With all these employees leaving ChickenLittle, Elise hopes the sky is not falling over there for her favorite design partner. Then, she and Deng investigate making their exceptions more programmable.

Since we’re diving into programmability, I’ll end our discussion of actionability and raising diagnostics from the interface layer here, turning my attention to making errors actionable for developers and their users.

[We now jump to the end of the chapter, skipping sections Raise Programmable Errors, Raise Specific Errors, Group Errors According to Scenario Category and Keep Information Around for Diagnostics]

Diagnose Early

Giving early diagnostics is often called shifting left and has numerous benefits for the system and for users.

Tip: Shift left. Give diagnostics to your users as early as you can.

For the system, it reduces resource usage by cutting off useless code paths. This can be critical, for example to fend off denial-of-service attacks. It also protects that same code from processing unpredicted inputs, preventing bugs like data loss.

Users benefit even more. Erroring early saves them time—think about how much time you save getting an error in your IDE rather than when deploying to production. Also, users are more likely to remember what they did when they get quick feedback, and to be more certain of the right action.

Here are four common techniques for shifting left:

• Do static validations

• Validate upfront

• Let them test

• Request user confirmations

Engineers widely practice the first two, but unfortunately not the last two. I came here to evangelize them all.

Do Static Validations

When validating any input, static validations can be performed cheaply and without deep inspection or network calls, while dynamic validations require more work or context.

Tip: Separate your cheap checks from your expensive ones and do the cheap ones early and often.

Many programmers are familiar with the pleasures of static typechecks and linters, but this comes up in products, too.

For example, when taking user input in application forms, look for cheap ways to tackle the users’ most common mistakes early, and provide inline advice telling them what needs to be corrected. Postal codes can be verified to be of the right length for their countries. Credit card numbers, UPC codes, and ISBN numbers have built-in checksum digits that help protect against common errors like typos and transpositions. These techniques shift left and also provide more certainty to diagnoses. If the checksum fails on a number, we know it’s not just missing from our database—it’s just flat wrong.

Validate Upfront

When I talked about crafting errors at the surface of an application or API, upfront validations provided a tool for more actionable and understandable error messages. But validations are also a tool for shifting left.

Imagine you went to the laundromat to wash your clothes, and you washed them, only to discover that only one dryer was online and it was booked for hours. A sign on the door would have saved you from a lot of wet, moldy clothes!

Similarly, when moving money, it’s essential to validate that the destination account is valid before withdrawing it from the source account.

Multistep workflows like laundry, money movement, and money laundering often benefit from upfront validation.

Let Them Test

If there’s one set of user scenarios I see most often neglected by platform engineers, it’s testing. They think of customers successfully using their production product without considering the trials their customers go through to get there.

If you’re building a developer service, your customers will adore you—and you’ll get adopted faster—if you build a fake. Fakes are high-fidelity versions of production systems that share as much code with the production path as possible, while taking certain shortcuts to avoid flaky runtime dependencies such as databases and online services that would make tests slower and less reliable.

Channelz should build a fake service for their messaging API. It would ideally:

• Run in-memory or in a lightweight local process so that it can be tested by its users in their test environments and locally on their development machines

• Allow users to input a bunch of users and channels so that they can then test various pieces of code that use the API

• Act just like the production service for as many scenarios as possible

For a real-world example, Stripe, a provider of payment and billing APIs and UIs, provides a popular “test mode” in which one can do realistic transactions without actually moving money around. For example, customers can pass in special credit card numbers to induce various Invalid User Argument and Precondition errors. The card number 4000 0000 0000 9995, for example, simulates a card_declined error with an insufficient_funds subcode.

Stripe users spend far more time with test mode than with production, leveraging it when writing integration tests and testing user interfaces. For example, if a merchant wants to test their credit card input form, they can plug those magic numbers in manually and see how the site behaves.

Fakes are great ways to shift errors far to the left—to before developers even get properly authenticated or submit a production request.

Request User Confirmations

Confirmations are roadblocks in your app or platform that help users verify whether they have done the right thing. They are louder than warnings but not as intrusive as errors.

They frequently appear in user interfaces and developer tools when heuristics detect something odd about the user’s input. They explain why it might be unexpected or dangerous, and provide the opportunity to correct it or confirm it.

Google’s “Did you mean” is a fine example. When a user searched for “compture,” maybe they really were referring to Compture, the band. So, Google can’t just return an error, but they can report their suspicion to the user upfront rather than relying on them to figure it out after digging through search results.

Or instead, when you have just asked a user to fill out several pages of forms, show them a summary of what they’re about to submit so they can check for errors.

Confirmations allow you to shift left at times when you can’t be certain that the user is wrong without running the input through the whole system, but you can take your best guess.

One common method to allow upfront testing is with a “dry run” followed by a confirmation. In a stock-trading app, this is as simple as multiplying the number of shares someone plans to purchase by the current share price to show users how much they will spend.

Here’s a more complex dry run at Channelz. Suppose Elise wants to add retry policies to her API, designed to be used like so:

What happens if ChickenLittle authors a really spammy retry policy that retries once a second, forever? This could cause an accidental Denial-of-Service (DoS) style attack on Channelz, which would surely result in ChickenLittle’s traffic being throttled.

Elise could shift left with a heuristic-driven warning by simulating the retry policy and flagging bad behavior:

Your retry_policy (currently initial_interval_seconds=1, backoff_coeffi cient=1.0, max_attempts=Infinite) will result in 601 attempts in 600 seconds. This exceeds our limit of 50. You may ignore this by adding .ignore(Errors::RetryPolicySpamminess) to your policy.

Elise is not perfectly confident in her exact heuristic, but she can move fast and help her users out by providing this sort of confirmation that stops development temporarily. If her heuristic is off, users will complain, and she can adjust it upward.

In developer tools, --force flags override confirmations.

Confirmations aren’t foolproof—users might blithely confirm or copy/paste a -- force flag. So don’t use them for inputs you really can’t accept. That said, most developers I know are accustomed to absolutes—either the input is allowable, or it’s not. You’ll find that thinking about a middle case—the input is probably wrong—unlocks an incredible amount of richness and expressiveness in your interfaces.

In this section, we discussed the merits of shifting left and captured some of the many user scenarios that would benefit from the practice.

Since shifting left sometimes means we can’t perform a full, perfect validation, I explored techniques such as service fakes, heuristic-based confirmations, and static validations like checksums to address the most common user mistakes as early as possible.

Chapter Summary

As you think about writing your next few diagnostics, start by shifting your focus to your users’ experience and consider what they know, what they need to know, and what they should do. If you’re writing an error, bucket it into its scenario category— System, User’s Invalid Argument, Precondition, Developer’s Invalid Argument, or Assertion. This will help you to reason about constructing the metadata and message.

In your messages:

• Give context. In most cases, echo back the operation being done and the bad data that was passed in.

• Use concepts from the product’s user-facing ontology.

• Suggest actions, and be willing to suggest alternatives if there’s not just one.

You’ll be more likely to do a good job of all of this if you raise the error to show the user from the interface boundary. For developers, include enough context and structure with your error so that other developers can do their jobs. Organize your errors into hierarchies to provide developers with both generic and specific ways to serve their users.

Finally, find creative ways to shift left for common or important scenarios, such as via techniques like confirmations, static checks, and fakes.

Exercises

1. Search the web for “Windows Blue Screen of Death Evolution” and watch as a few decades of Windows teams struggle to improve their error messages. (a) Decide which personas its messages are tackling. (b) Later in the video, you’ll see the Windows 11 screen. Analyze its message for the personas you picked.

2. For the next questions, pretend that you work on an e-commerce app. Users can fill their shopping cart and check out. The app finalizes orders via a submit_order API with the following fields: user ID, shopping cart ID, and credit card CVV code—the three- or four-digit code to verify a credit card. Your mobile client has the ability to do static verifications before sending up data to your API.
   
   What error scenario category are invalid user ID and invalid shopping cart ID?

3. What error scenario category is an invalid CVV code?

4. What about an empty CVV code?

5. Write an actionable end-user-facing error message for a missing CVV code.

6. Shopping carts can expire and are automatically deleted from the system after 24 hours. What scenario category of error is raised when the user tries to resume their session by adding to a deleted cart?

7. Can you think of a way to shift this expired shopping cart error left so that the user doesn’t need to re-find each item?

Takeaways

That concludes the excerpt

Many thanks to Drew and O’Reilly for sharing this extract from the new book. I very much recommend it if you’re interested in strengthening your “product muscle” as a software engineer, and it comes with lots of actionable examples. You can subscribe to Drew’s Substack, follow him on LinkedIn, and read more about his personal journey to becoming a product-minded engineer. Drew is also a speaker at the upcoming The Pragmatic Summit.

Buy The Product-Minded Engineer

(or get it on Amazon)

What I really appreciate about the excerpt is that Drew goes into detail about the significance of those parts of a product which devs rarely think of, such as simple things like error messages. With well-crafted errors/warnings, you can reduce confusion, improve a product’s usability, and improve “stickiness” (i.e., when users “stick around” with a product despite it having issues).

My sense is that being a product-minded engineer is more than a simple checklist of skills, but I appreciate that Drew breaks down the important areas to focus on, like using your own product, collecting customer feedback, understanding users, understanding the product better, its architecture, and more.

I’m unsure to what extent 20+ years of lived experience can be captured in a single volume, but Drew makes a superb effort at doing so. In your own workplace, if there are standout product-minded engineers – or developer-minded product managers – from whom you can learn tips and tricks, then this book may seem familiar. For the rest of us, “The Product-Minded Engineer” is a great introduction to becoming a product-minded software engineer at a time when demand for this role is rising alongside the rapid growth of AI in our dev workflows.

Speakers might be familiar from the podcast — and this is a rare opportunity to meet them in-person. We’ll have Laura Tacho, Simon Willison, Chip Huyen, Kent Beck, Martin Fowler, Nicole Forsgren, Thomas Dohmke and others join. Engineering leaders from the top AI companies and startups will talk about how exactly they build — folks from Cursor, Codex, Linear, Ramp, Vercel and Uber, and more. And in a fun crossover, Alex and Sahn from ByteByteGo will also take the stage.

The exact agenda:

We already have 250 confirmed attendees — experienced engineering leaders and experienced engineers — and just 150 seats left. We also have more delightful surprises planned for the event, and on the day — those who attend will learn of these. If you are thinking of joining, apply now, while there is still space.

If you want to meet in-person, meet folks who have been on the podcast or in the deepdives, and connect with an incredible group at the event: this is the event to do it at.

Talks will be recorded and shared with paid newsletter subscribers in a timely matter — and they will be published in batches for all newsletter subscribers later on as well.

Apply to join the Summit

At some point in the lifespan of a successful product, a member of the team decides it has become too messy, and that what it needs is some kind of system/UX library/UI organization, since what used to be creative, clever UI approaches have become chaotic. At this point, the idea of a reusable components library often starts being seen as a solution.

This usually happens long after the product has found its product-market-fit and while it’s growing in usage, with 10+ designers and frontend engineers working on it. I’ve seen this scenario play out several times: at Skype, Skyscanner, and at Uber, where the outcome was the Base Design System, which was shared externally.

But how is a design system like this created, and what are practical tips for doing so? To find out, I turned to an expert on the topic, Michael Abernethy, who’s principal frontend engineer at the data security company, Rubrik. He’s worked on user interfaces and design systems for 25+ years, and has led Rubrik’s design system engineering team for the past four.

Today, we cover:

1. Wake-up call. Feedback from the CTO kicked off efforts to make the UI look more professional at Rubrik.

2. What’s a design system? More than a style guide, it includes standardized building blocks, and becomes a unified source of truth for the design and engineering team.

3. Building a design system. UX researchers, designers, and design system engineers collaborate. A step-by-step guide on how the team at Rubrik did so.

4. Design system libraries in the AI era. The new technology can be helpful for many tasks, but generating a consistent design system isn’t one of them. A look into areas where AI has helped Rubrik’s design system, plus emerging use cases like using Figma’s MCP, or running design system subagents.

5. When not to build a design system library. For startups and smaller teams, using a pre-built design system library like Material UI, Chakra UI, or Base UI can be a great alternative.

6. Investing in a custom design system library. Advice about good engineering skills and traits for when building a design system, the timelines – and Big Tech’s different approach.

7. Getting started and inspiration. A five-step, practical approach for building your own design system.

At growing tech companies, a design system is almost always put in place, eventually. I hope this is a useful guide about how to contribute to it – or take the lead – as an engineer.

With that, it’s over to Michael:

1. Wake-up call

At Rubrik, our “design wake-up call” is still talked about. It was six years ago, when the CTO, Arvind Nithrakashyap, casually remarked during a team meeting that the product looked “like a college project”.

The UI he was talking about was this early version:

His observation hit home because it was right! The product’s user interface did indeed look like it had been thrown together by engineers trying to build features as fast as possible – because that’s exactly what happened. This feedback was enough to prompt a UI overhaul that would see us revamp the whole product’s UI/UX for the better.

Most engineers will understand the urge to build quickly and ship lots of features in order to hit seemingly-impossible deadlines. Questions about how a product looks often take a back seat behind getting people to use the product. And let’s be honest, few of us engineers are famous for our artistic sensibilities, so a user interface we deem “good enough” during the early MVP stage may not cut it when pitching to potential enterprise customers.

When we eventually completed this project at Rubrik, we were able to put a “Version 1.0” stamp on what we called the “Aura Design System Library”. It consists of 90+ React-based components, ranging from simple components (buttons, checkboxes, tabs) to complex components (page layouts or tables connected to GraphQL endpoints). Our work visibly improved the product, which went on to win awards and achieve industry-wide recognition.

2. What’s a design system?

The best, easiest way to think of a design system is as a brand for your software. It’s the unifying source of truth for a company’s or product’s entire identity across the suite of digital assets. But it’s also much more than just a logo. It includes:

• Rules and guidelines

• Patterns and visual language

• Colors, fonts, spacing, and image styles

• Reusable code components

• Style guides and documentation that define the design system’s rules

Basically, it’s all the small things that go into making a website unique, so that customers instantly recognize the software. Like any strong brand image, a great design system is an incredible asset for a business because it creates an identity for software that’s as distinctive as Nike’s swoosh, or an apple with a bite taken out. Take a look at these two screenshots from popular websites:

Without my help, you can probably identify the products in the screenshots above as Airbnb and Spotify. Chances are, you also have opinions about companies when you see their websites. That’s the power of branding – and of a great design system.

A design system is important for users, too. If they become accustomed to what a button or menu looks like in one part of the software, they know what a button or menu looks like in the rest of the application. A UX researcher might say this reduces the cognitive load of using the software.

Design system engineering

A design system engineering (DSE) team takes the concept of a design system, the branding, and the user benefits, and converts all this into code by building a design system library of reusable components that adhere to the strict standards of the design system. At Rubrik and at most Big Tech companies it works like this:

1. The design team creates the look and feel of the software in Figma.

2. The DSE team converts that into working React code (or Angular, or iOS, or Android, or the framework they use).

The design system library is the suite of reusable Frontend components that the DSE team creates. These are then bundled up into a stand-alone, JS/iOS/Android library and used across the entire product line. Using the library ensures a visually cohesive brand identity.

Smallest design system: a button

Imagine you’re a member of a Series B startup that’s just raised $20M in funding, and has found a product-market fit. Business is growing fast, but your UI is inconsistent, so you decide to standardize the design with a library and to improve the UI and UX.

Soon, you’re in the kickoff meeting of this effort with a few designers and design system engineers (with their DSE hats on) who are ready to crank out the new components. You start with the most common component of any design system library: the button. Below are some decisions to be made about this seemingly simple component:

Color. Which shade of teal is best for users, balancing readability with other colors on the page, and considering every possible background color on which the button could appear?

Text color. Which text color makes the button most readable to users on all devices, keeping in mind that not every user visits your software on a $6,000, 6k Pro Display XDR Monitor.

Height. How many pixels tall should the buttons be in order to draw attention without wasting vertical space and messing with page layout?

Border radius. What kind of curves fit best with the design aesthetic of the application?

Shadows. Do you want the design to be flat, or for the buttons to appear to rise off the page to grab attention, implying they can be pressed?

Font. Which font gives the button a modern, readable look? Should it match the rest of the site, or be slightly different to draw attention?

Answering just these six questions can take considerable time, including multiple back-and-forth discussions. The meeting may typically take an hour, and the bad news is that it will barely scratch the surface of everything that must be agreed. A proper discussion about a button will also include:

• Color: Should there be extra background color options for less important actions?

• Border: Should there be a color border, and if so, how thick?

• Icon: Should icons in the button be allowed? If so, should they go before or after the text? What spacing should be used between icon and button?

• Events: What should the button look like when the keyboard navigates to it? What about when hovering and clicking?

• Animation: Should there be an animation when the button’s hovered over, should there be one when it’s clicked, and what about when something is loading?

• States: What should the button look like when disabled? What about the cursor in the disabled state?

• Mobile: Do you need to adjust any design decisions for iOS, Android, or both?

• Accessibility: Do you need to adjust any decisions for color blind users?

• Localization: Do you need to rethink decisions for different languages, or right-to-left writing systems?

All of these calls need to be made about a button – which is just one component! A simple design system usually has 15-20 components, and a complex one can have 50-100. Designing a full-fledged design system involves a lot more time and effort than many people realize because all these details add up.

Benefits of a design system

As mentioned, it’s a lot of effort to create a design system – but it brings several wins when complete. From the engineering viewpoint, common benefits are:

Faster delivery times. By creating an entire suite of reusable components, engineers eliminate the time and burden of creating the same component over and over on their own. Having a library also removes the need to “reinvent the wheel” by recreating new components. With a mature design system library, creating a page in the UI is almost as easy as dragging and dropping components onto the screen and wiring them up.

Higher quality. By utilizing skilled frontend/UI engineers to create the design system library, the quality of the entire product suite can be instantly visually upgraded. The UI’s quality can go from amateur to professional with the work of a few engineers.

More devs can contribute to the frontend. In my observation, most full-stack engineers lack expert-level skills in CSS and React. However, a well-written design system library encapsulates the difficult React/CSS/HTML code within the library, and allows full-stack engineers to contribute across the frontend without needing expertise in more niche areas like CSS.

“Premium feel” software. I have a saying that if you want to charge enterprise prices, your software needs to look like an enterprise product.

I feel many engineers don’t internalize the significance of premium interfaces. If your product charges a high price, customers expect a superior UI that goes beyond a simple and basic style. For example, it would be hard to entice an enterprise to pay $1M/year for software that looks like the image below. This site uses a standard, free, off-the-shelf Bootstrap theme with default colors and a font that doesn’t look good on the page. It looks the same as thousands of other free websites out there, making it hard for your customers to think “premium” when they see it.

Better collaboration across teams. With a shared library and design, engineering and product management can “speak the same language”, so that when a designer references a component called a “StatBox,” everyone understands what it means, for example. Even better, a tool like Figma allows design and engineering to virtually share code between teams.

Cost savings. Time is money, and a system like this makes engineers more productive because they deliver features faster, and the company can increase revenue quicker. That’s the theory, at least!

3. Building a design system

We’ll use an actual component to go through the steps of building a design system: the Range Bar component in Rubrik.

UX research

User experience (UX) folks are usually members of the design team who meet customers and talk through problems in simple terms, seeking feedback about usability, ease-of-use, and desired functionality.

During this research, one theme might be that users dislike having to type numbers to specify a minimum and maximum storage capacity. They’d prefer an easier way to use the component in the UI.

The UX team gathers this feedback and puts together the component requirements:

• A visual way to see min/max without typing

• A way to remove the burden of calculating standard storage sizes (i.e: powers of 2)

• A simple way to navigate with a few clicks.

Here’s a rough sketch of the component they create:

The designer

The design system team takes it from here. They receive the written details and rough sketch from UX and then create a visual guide for how it should look and behave across the application.

Figma is the tool of choice for almost every design team, and it’s the same at Rubrik. A designer creates the component, and realizes the best solution would be to expand the functionality of the existing Slider component by adding a “start” and “stop” thumb to the component, and automatically snapping it to a power of 2:

The designer also decides to include the ability to type in the minimum and maximum numbers, and to round up the numbers when the field is blurred. When the Figma design is complete, they pass it to the DSE team to turn into code.

Design system engineer

The design system engineer receives the Figma file and examines it from an engineering perspective, asking questions like:

• Implementation constraints. Has the designer used designs that are impossible to implement in CSS, such as alpha-blended non-rectangular shapes, or gradients in places that aren’t supported?

• Scalability. Has the designer used interactions that don’t scale well, like interactions or animations that trigger too many re-renders?

• Performance. Has the designer created the potential for performance degradation; e.g., dozens of shadows in a moving element?

• Edge cases. Has the designer considered edge cases like whether the user can set the min and max to be the same value, and considered error states such as how an invalid input should be displayed to users? Have they considered responsiveness (i.e: does it look good on both small and ultra-wide screens)?

• Accessibility. How does it display keyboard focus, and how does a customer use the component without a mouse?

• Localization. Does the design work in languages with many long words, and what about right-to-left writing systems?

Design system engineers and designers work closely to iron out these details, and once they’re sorted, the engineer starts coding.

Coding typically starts in Figma’s Dev Mode. This is a simplified view tailored for cases where engineers want to implement Figma designs. They can see key information about the design in one place, like the sizing of the component, spacing, etc. There’s even the option to copy/paste the CSS from Figma and insert it directly into the code.

Coding

Below are key things to consider when beginning to put together the code of the component:

• Naming. A good name is one that’s agreed by design and engineering, so that both use the same terms. Good naming also helps the Figma MCP (Model Context Protocol) work better. For example, for this component, “RangeSlider” feels more suitable than MinMaxStorageSlider. It’s more expressive and doesn’t expose implementation details that might change later.

• API. For React components, the API means setting the prop names to be descriptive and consistent with prop names in the rest of the design system library.

• Future usage. The engineer should think about how this component will be used in the future, and support likely use cases. For example, should the RangeSlider allow the option to make the min or max ranges unchangeable? That decision could change how the component is coded in React.

Coding a high-quality component comes with plenty of nuance, which is why it helps to have experienced engineers build them. Below are some small details that are important to get right:

• Performance: Will the component stay performant if the user slides the RangeSlider’s thumb up and down quickly? Do the events need to be debounced?

• Theming: How will the component look in a different theme? Rubrik has three themes for its product, and components need to look good in them all.

• Accessibility: For example, how is the component used without a mouse? For a component like a RangeSlider, accessibility can be quite complex. In Rubrik’s case, we decided to add support to control the component with the four arrow keys (left/right control the minimum, up/down control the maximum).

There are three less-discussed areas which become increasingly important to a well-built modern design system library.

Analytics: Often an afterthought, but perhaps shouldn’t be! At Rubrik, we recently faced a dilemma where we wanted to massively increase our analytics usage, and started to log pretty much everything. The challenge was that there were already 1,200 different pages in the application! Adding analytics to each didn’t make sense.

Instead, we added analytic tracking code into our design system library’s components. For example, when a button is pressed, it (abstractly) sends a tracking event to our analytics platform. As design system engineers, we went through each component and added analytics depending on how users could interact with it.

With this approach, we started to log around X4 more events, which were all relevant for analyzing our application’s usage. The best part was that devs didn’t need to add a single line of code to get more granular analytics – they just needed to use components from our design system library!

Async awareness: React’s v18 and v19 have introduced and expanded the notion of “async React,” which aims to improve user experience and the code’s performance by processing less important rendering in the background, and prioritizing certain renders. For example, an input box that’s being typed in should prioritize displaying the key typed, but can defer the display of an error state if a key press triggers a validation error.

At Rubrik, we’ve started re-coding our interactive components to use the Async React approach. These kinds of optimizations in the design system library can make an application feel snappier and quicker and also reduce the chance of another engineer introducing lag into the UI.

Loading states: Modern UI like React Server Components rely on loading data in the background as the page renders. As a result, displaying loading states becomes more critical, as a component may be rendered immediately even if the data it displays isn’t available yet.

At Rubrik, we’ve re-coded our components that display information to handle loading states. This allows devs using each component to pass in a “loading” property and ensure the application keeps a consistent look, even when data isn’t present.

Testing

As with any quality software, we want to be thorough with testing. At Rubrik, we use several layers of testing to ensure the design system library minimizes regressions:

Unit tests (including writing lots of them with AI). We use unit tests extensively, which is a common approach for any standard React component.

Over the past year, we’ve come to rely heavily on AI to write unit tests, and have found that it not only creates time savings, but also hits more edge cases. With AI, we can generate tests with extremely high code coverage from surprisingly short prompts.

Storybook. This is an open-source tool for publicly displaying and documenting UI components outside of the main application. It allows engineers to create “stories”, which are examples of components in different states and configurations. Stories allow Frontend engineers outside of the DSE team to visualize the components without the complexity of a full application.

Storybook also allows other engineers to toggle and adjust the props into the components and copy/paste the resulting code. Engineers will then have the ability to visually customize the code to their exact needs.

At Rubrik, we use Storybook extensively and utilize it as a showcase of all available React components within our application. Nearly every Frontend Engineer visits our internal Storybook site on a weekly basis.

Chromatic: This is a visual regression testing tool. When making a change to a single component, an engineer needs to check that it works correctly across all four major browsers and supported themes. Plus, they need to check that any component dependent on the updated component works correctly.

Chromatic ties into Storybook and takes a visual snapshot of every component, comparing it to the previous snapshot to check for pixels that have changed, which it flags up. Devs can then examine how a change affects the rest of the library. It’s a great way to test regressions at scale.

At Rubrik, Chromatic runs on every pull request that touches any file within our design system library. We do this by adding Chromatic as a blocking reviewer in GitHub for all these files. For us, Chromatic takes about 15 minutes to run against the design system library suite. If it finds no regressions, the bot removes Chromatic from the blocking reviewers list. If something has regressed, the blocking reviewer stays and the engineer who submitted the PR resolves the issue.

Thanks to running on every pull request, Chromatic has prevented 100+ bugs from shipping to production at Rubrik. Without it, I suspect most of those changes would’ve slipped through and have only been discovered later.

4. Design system libraries in the AI era

Can a team use AI to create a complete, well-designed, complex design system library from a single prompt? Today, that would be asking too much of AI, but maybe one day soon it will be possible. At Rubrik and in my own experiments with AI and design system libraries, I’ve found several cases where AI is extremely helpful, and also times when it’s not. In some situations, it even damages the work.

Read more

Unexpectedly, LLMs like Opus 4.5 and GPT 5.2 did amazing jobs on the mid-sized tasks I assigned them: I ended up pushing a few hundred lines of code to production simply by prompting the LLM, reviewing the output, making sure the tests passed (and new tests I prompted also passed!), then prompting it a bit more for some final tweaking.

To add to the magical feeling, I then managed to build production software on my phone: I set up Claude Code for Web by connecting it to my GitHub, which let me instruct the Claude mobile app to make changes to my code and to add/run tests. Claude duly created PRs that triggered GitHub actions (which ran the tests Claude couldn’t) and I found myself reviewing and merging PRs with new functionality purely from my mobile device while travelling. Admittedly, it was low-risk work and all the business logic was covered by automated tests, but I hadn’t previously felt the thrill of “creating” code and pushing it to prod from my phone.

This experience, also shared by many others, suggests to me that a step change is underway in software engineering tooling. In this article – the first of 2026 for this publication – we explore where we are, and what a monumental change like AI writing the lion’s share of code could mean for us developers.

Today, we cover:

1. Latest models create “a-ha” moments. It’s not just devs working at AI vendors who noticed much more capable models, but also independent software engineers.

2. Why now? Model releases in November and December seem to have been the tipping point: Opus 4.5, GPT-5.2 and Gemini 3.

3. The bad: declining value of expertise. Prototyping, being a language polyglot or a specialist in a stack are likely to be a lot less valuable, looking ahead.

4. The good: software engineers more valuable than before. Tech lead traits in more demand, being more “product-minded” to be a baseline at startups, and being a solid software engineer and not just a “coder” will be more sought-after than before.

5. The ugly: uncomfortable outcomes. More code generated will lead to more problems, weak software engineering practices start to hurt sooner, and perhaps a tougher work-life balance for devs.

6. Product management vs software engineering: merge or separation? Product managers can now generate software easier – needing fewer engineers to realize their goals – but software engineers also need less product management. Both professions are set to overlap with another more than before.

1. Latest models create “a-ha” moments

Over the past few weeks, some experienced software engineers have shared personal “a-ha” moments about how AI tooling has become good enough to use for generating most of the code they write.

Jaana Dogan, principal engineer at Google, was very impressed by how far Claude Code has come:

“I’m not joking and this isn’t funny. We have been trying to build distributed agent orchestrators at Google since last year. There are various options, not everyone is aligned, etc. I gave Claude Code a description of the problem, it generated what we built last year in an hour.

It’s not perfect and I’m iterating on it, but this is where we are right now. If you are skeptical of coding agents, try it on a domain you are already an expert in. Build something complex from scratch where you can be the judge of the artifacts”.

Thorsten Ball, software engineer at Amp reflected (emphasis mine):

“For more than 15 years, I thought I loved writing code, loved typing out code by hand, and loved the “cadence of typing”, as Gary Bernhardt once called it; sitting in front of my editor with my fingers click-clacking on my keyboard.

Now, I’m not so sure.

2025 was the year in which I deeply reconsidered my relationship to programming. In previous years, I had the occasional “should I become a Lisp guy?”, for sure; but not the question “do I even like typing out code?”

What I learned over the course of the year is that typing out code by hand now frustrates me”.

Malte Ubl, CTO at Vercel:

“It’s been a crazy holiday period: I built 2 major open-source projects (one unreleased, and the other a full implementation of a bash environment in TypeScript for use by AI agents).

• I started writing a book

• I fixed a bunch of other things

It’s a very different world now that we’re in Opus 4.5 world, and these would absolutely not have been possible without it. Opus + Claude Code now behaves like a senior software engineer whom you can just tell what to do, and it’ll do it. Supervision is still needed for difficult tasks, but it is extremely responsive to feedback and then gets it right.

I don’t want to be too dramatic, but y’all have to throw away your priors. The cost of software production is trending towards zero”.

Longtime readers may recall Malte from his reflections on 20 years in software engineering, including 11 at Google.

One could claim that the voices above have an interest in this topic given they work at companies selling AI dev tools. But engineers with no pull towards any vendor also made similar observations:

David Heinemeier Hansson (DHH), creator of Ruby on Rails, described how his stance on AI has flipped due to the improved models:

“You can’t let the slop and cringe deny you the wonder of AI. This is the most exciting thing we’ve made computers do since we connected them to the internet. If you spent 2025 being pessimistic or skeptical about AI, why not start 2026 with optimism and curiosity?

Just last summer, I spoke with Lex Fridman about not letting AI write any code directly, but it turns out part of this resistance was simply based on the models not being good enough at the time! I spent more time rewriting what it wrote, than if I’d done it from scratch. That has now flipped.”

Adam Wathan, creator of Tailwind CSS, reflected:

“Any time I have to type precise syntax by hand now [instead of using AI] feels like such a tedious chore. Surprisingly and thankfully, programming is still fun, probably more fun [with LLMs].

My biggest problem now is coming up with enough worthwhile ideas to fully leverage the productivity boost.”

From “AI slop” to rocking the industry

One widely circulated “a-ha” moment has been from Andrej Karpathy, a cofounder of OpenAI. Andrej has not been involved in OpenAI for years, and is known to be candid in his assessment and critique of AI tools. Last October, he summarized AI coding tools as overhyped on the Dwarkesh podcast (emphasis mine):

“Overall, the models are not there. I feel like the industry is making too big of a jump and is trying to pretend like this is amazing, and it’s not.

It’s slop.

They’re not coming to terms with it, and maybe they’re trying to fundraise or something like that. I’m not sure what’s going on, but we’re at this intermediate stage. The models are amazing. They still need a lot of work.

For now, autocomplete is my sweet spot. But sometimes, for some types of code, I will go to an LLM agent”.

Two months on, that view had been thoroughly revised, with Karpathy writing on 26 December (emphasis mine:)

“I’ve never felt this much behind as a programmer.

The profession is being dramatically refactored as the bits contributed by the programmer are increasingly sparse. I have a sense that I could be 10X more powerful if I just properly string together what has become available over the last ~year, and a failure to claim the boost feels decidedly like a skill issue.

There’s a new programmable layer of abstraction to master in addition to the usual layers involving agents, subagents, their prompts, contexts, memory, modes, permissions, tools, plugins, skills, hooks, MCP, LSP, slash commands, workflows, IDE integrations, and a need to build an all-encompassing mental model for strengths and pitfalls of fundamentally stochastic, fallible, unintelligible and changing entities suddenly intermingled with what used to be good old fashioned engineering.

Clearly some powerful alien tool was handed around, except it comes with no manual and everyone has to figure out how to hold and operate it while the resulting magnitude 9 earthquake is rocking the profession.

Roll up your sleeves to not fall behind”.

The creator of Claude Code, Boris Cherny, responded, sharing that all of his committed contributions last month were AI-written:

“The last month was my first as an engineer when I didn’t open an IDE at all. Opus 4.5 wrote around 200 PRs, every single line. Software engineering is radically changing, and the hardest part even for early adopters and practitioners like us is to continue to re-adjust our expectations. And this is *still* just the beginning”.

We previously did a deepdive on how Claude Code was built with Boris and other founding members of the team.

2. Why now?

Model releases in November and December seem like the tipping point where AI got really good at generating code:

• Gemini 3 by Google (17 November): Google’s best coding model to date

• Opus 4.5 by Anthropic (24 November): the best coding model from the company, which has become the default model for Claude Code

• GPT-5.2 by OpenAI (11 December): powering Codex, it’s similarly impressive as Claude Code with Opus

I’ve used both Opus 4.5 and GPT-5.2 and they seem very competent at coding, which isn’t a niche opinion. Here’s Peter Steinberger, a software engineer with ~20 years’ experience and the creator of PSPDFkit, gave his impression of GPT-5.2:

“The step from GPT 5/5.1 to 5.2 was massive. Now that GPT 5.2 is out, I have far fewer situations where I need [a custom-built CLI agent called ‘oracle’ to get the agent unstuck]. I do use [GPT 5] Pro myself sometimes for research, but the cases where I asked the model to “ask the oracle” went from multiple times per day to a few times per week.

I’m not mad about this; building ‘oracle’ was super fun and I learned lots about browser automation, Windows, and finally took time to look into skills, having dismissed the idea for quite some time. What it does show is how much better 5.2 got at many real-life coding tasks: it one-shots almost anything I throw at it”.

Simon Willison is an independent software engineering expert in LLMs whom I pay attention to, and he pointed out the same:

“It genuinely feels to me like GPT-5.2 and Opus 4.5 in November represent an inflection point - one of those moments when the models get incrementally better in a way that tips over an invisible capability line. Suddenly, a whole bunch of much harder coding problems open up.

It’s possible Gemini 3 Pro should be included in that group as well, but I’m not seeing quite the same level of astonished buzz from hardened software engineers around that model that I am for the other two”.

The first-ever The Pragmatic Engineer Podcast episode was with Simon, fittingly entitled AI tools for software engineers, but without the hype – and Simon continues to live up to that sentiment.

Wild prediction coming true?

One forecast which many – including me – were sceptical about was made by Anthropic’s CEO, Dario Amodei, last March, when he said:

“I think we will be there in three to six months, where AI is writing 90% of the code. And then, in 12 months, we may be in a world where AI is writing essentially all the code”.

Yet it came to pass in December, when 100% of Cherny’s code contributed to Claude Code was AI-written. Playing devil’s advocate, one could point out that Claude Code is closed source, so the claim is hard to validate. And of course, the creator of something like Claude Code wants to showcase its high-performance capabilities. But I’ve talked with Boris and trust him; also, my own experience of using Claude Code tallies with his: I let Claude Code generate all the code I end up committing.

When the code is not how I want it, I do more prompting to get the LLM to fix it. Like Boris, I’ve ceased writing code by hand because I don’t have to do it, and the models seem capable enough. I can still do it, but it’s simply faster to leave it to the model.

Even for code that I know well enough and can navigate in my IDE, I’ve noticed that prompting the agent can get edits done just as quickly as I would do it, if not faster. Nonetheless, I do find myself going to the IDE as I like to know that I can do the work.

For my own use case, the claim feels true enough that AI can generate 90% of the code I write, using TypeScript, Node/Express, React, and Postgres as technologies, in my case. I understand the same is increasingly true of languages like Go, Rust and other popular languages and frameworks which LLMs have plenty of training data for. Interestingly, LLMs seem to be pretty good at C as well, as observed by Redis creator, Salvatore Sanfilippo.

The remainder of this article assumes AI coding tools WILL become good enough to generate ~90%+ of the code for many devs and teams, this year. The most likely candidates for this landmark look like being startups seeking product-market fits – for whom throwaway work is no big deal – and also greenfield development projects, where no existing codebase needs to be ingested or understood before building something new.

Of course, there will also be plenty of occasions when it won’t make sense to totally rely on AI coding tools, either due to them not working well enough in some codebases or contexts, or when devs purposefully choose not to lean on them.

It’s worth exploring the directions in which the software engineering profession could go in environments where almost all code is generated by AI via prompts, rather than being typed out by a developer. This would be a sea change that impacts the profession, but how? Let’s run the rule over the good, bad, and the ugly possibilities, starting with potential negatives.

3. The bad: declining value of expertise

Some things that used to be valuable will be mostly delegated to the AI:

Prototyping. Platforms like Lovable and Replit explicitly advertise themselves as ways for nontechnical folks to build software. In a recent ad, Replit teamed up with legendary NBA star Shaquille O’Neal, who vibe-coded an app for getting the top 5,000 pickup lines in online history, among others:

Still, with AI tools, product folks, designers, and business people can build their own prototypes, and no longer need a dev to make an idea real. Alongside this, it’ll become a baseline expectation for each and every developer to be able to generate concept apps fast.

Being a language polyglot will probably be less valuable. Engineers who are experts in multiple languages have traditionally been coveted because engineering teams often like to hire experts in their own stack. So, a team using Go would often prefer to hire senior engineers with Go knowledge, and the same with Rust, Typescript, etc. I’d qualify this with the fact that some forward-thinking teams ignore candidates’ expertise in their specific language and assume that competent engineers can pick up languages on the go.

But with AI writing most of the code, the advantage of knowing several languages will become less important when any engineer can jump into any codebase and ask the AI to implement a feature – which it will probably take a decent stab at. Even better, you can ask AI to explain parts of the codebase and quickly pick up a language much faster than without AI tools.

The end of language specializations and frontend/backend specializations? In the early 2000s, I recall most job adverts wanted candidates for a specific language: ASP.NET developer, Java developer, PHP developer, and so on. From the mid-2010s, more companies started to hire based on specialization in the stack: backend developer, frontend developer, native iOS/Android developer, cross-platform mobile developer. On the backend, it became accepted that a dev who knew a language like Go pretty well could pick up Typescript, Scala, Rust or other languages as necessary. Roles where for specific languages still matter were for native mobile, where the iOS and Android frameworks were different enough to require deep expertise in one or the other.

But today with AI, a backend engineer can prompt decent frontend code, cross-platform code, or even attempt native mobile code. With this tool, I struggle to foresee startups hiring separate frontend and backend devs: they’ll just hire a specialist whom they trust will use AI to unblock themself across the stack.

Implementing a well-defined ticket is something AI will increasingly do, like taking a JIRA or Linear ticket that’s well-defined – like a bug report or small feature request – and implementing it. Even today, the team at Cursor has an automation where all Linear tickets are automatically passed to Cursor, which one-shots an implementation. A dev can then choose to merge it or iterate on it. The more context is passed and the better the models are, the more likely the output will be mergeable.

This will be a major shift, especially in hierarchical workplaces where project or product managers have long been writing detailed tickets for devs to implement with no questions asked!

Refactoring will probably be ever more delegated to AI. It’s already pretty good at refactoring and the tools will probably only get better. Refactoring by hand will be much slower than spelling out what type of refactor you want the AI to do. Of course, in the pre-AI era, modern IDEs also offered powerful refactoring capabilities for speeding up tedious tasks like renaming functions or classes that did the renaming across the entire codebase, extracting functions.

Of course, there will always be the risk that AI messes things up, especially with large refactorings. This is why having ways to validate code will surely be even more important this year.

Paying careful attention to generated code – in some cases? Generating code from AI prompts can lead to verbose code, or duplication of existing code instead of using an abstraction. But there are times when this is perfectly acceptable, such as when building proof of concepts, or when topics like program efficiency are unimportant.

Software engineer Peter Steinberger is building greenfield software, and said he’s stopped reading code generated by AI:

“These days, I don’t read much code anymore. I watch the stream and sometimes look at key parts, but I gotta be honest, most code I don’t read. I do know where components are and how things are structured, and how the overall system is designed; that’s usually all that’s needed.

The important decisions these days are language/ecosystem and dependencies. My go-to languages are TypeScript for web stuff, Go for CLIs and Swift if it needs to use macOS stuff, or has UI. Go wasn’t something I gave the slightest thought even a few months ago, but eventually I played around and found that agents are really great at writing it, and its simple type system makes linting fast”.

Even so, reading code will remain important when extending existing mature software, or when security issues need to be avoided. In general, if shipped code doesn’t work and hurts the business, then you’ll want to both test and review it for correctness.

4. The good: software engineers more valuable than before

Read more

AWS released an incident summary three days later, revealing the outage in us-east-1 was started by a failure inside DynamoDB’s DNS system, which then spread to Amazon EC2 and to AWS’s Network Load Balancer. The incident summary overlooked questions such as why it took so long to resolve, and some media coverage sought to fill the gap.

The Register claimed that an “Amazon brain drain finally sent AWS down the spout”, because some AWS staff who knew the systems inside out had quit the company, and their institutional knowledge was sorely missed.

For more clarity and detail, I went to an internal source at Amazon: Senior Principal Engineer, Gavin McCullagh, who was part of the crew which resolved this outage from start to finish. In this article, Gavin shares his insider perspective and some new details about what happened, and we find out how incident response works at the company.

This article is based on Gavin’s account of the incident to me. We cover:

1. Incident Response team at AWS. An overview of how global incident response works at the leading cloud provider, and a summary of Gavin’s varied background at AWS.

2. Mitigating the outage (part 1). Rapid triage, two simultaneous problems, and extra details on how the DynamoDB outage was eventually resolved.

3. What caused the outage? An unlucky, unexpected lock contention across the three DNS enactors started it all. Also, a clever usage of Route 53 as an optimistic locking mechanism.

4. Oncall tooling & outage coordination. Amazon’s outage severity scale, tooling used for paging and incident management, and why 3+ parallel calls are often run during a single outage.

5. Mitigating the outage (part 2). After the DynamoDB outage was mitigated, the EC2 and Network Load Balancer (NLB) had issues that took hours to resolve.

6. Post-incident. The typical ops review process at AWS, and how improvements after a previous major outage in 2023 helped to contain this one.

7. Improvements and learnings. Changes that AWS is making to its services, and how the team continues to learn how to handle metastable failures better. Also, there’s a plan to use formal methods for verification, even for systems like DynamoDB’s DNS services.

Spoiler alert: this outage was not caused by a brain drain. In fact, many engineers who originally built the service, DNS Enactor (responsible for updating routes in Amazon’s DNS service) ~3 years ago, are very much still at AWS, and five of them hopped onto the outage call in the dead of night, which likely helped to resolve the outage more quickly.

As it turns out – and as readers of this newsletter likely already know! – operating distributed systems is simply hard, and it’s even harder when several things go wrong at once.

Note: if you work at Amazon, get full access to The Pragmatic Engineer with your corporate email here. It includes deepdives like Inside Amazon’s Engineering Culture, ones on Meta, Stripe, and Google, and a wide variety of engineering culture deepdives.

1. Incident Response team at AWS

Amazon is made up of many different businesses of which the best known are:

• Amazon Retail: operates the shopping websites Amazon.com, and 21 regional versions including Amazon.co.uk, Amazon.de, and others.

• AWS: everything to do with Amazon Web Services. Retail is a major customer of AWS.

These organizations operate pretty much independently with their own processes and set ups. Processes are similar but not identical, and both groups evolve how they work over time, separately. In this deepdive, our sole focus is AWS, but it could be assumed Retail has some similar functions, like separate Incident Response teams.

Regions and AZs: AWS is made up of 38 cloud regions. Each one consists of at least 3 Availability Zones (AZs), which are at least one independent data center, connected via a low-latency network. Note: “Region” and “AZ” mean slightly different things among cloud providers, as covered in Three cloud providers, three outages, three different responses.

Incident Response is a dedicated team at AWS, staffed by experienced support and infrastructure engineers, who do the following:

• Monitor all 38 AWS regions, continuously. Health KPIs (Key Performance Indicators) are set up, so it’s easy to tell if a service is healthy or not. These KPIs vary by service, but typical ones include monitoring API success rates and response latencies, as well as synthetic canaries.

• Get alerted when a region becomes unhealthy, or KPIs suggest it might be.

• Coordinate incident calls when an outage happens within a region or AZ.

• Oncall rotation 24/7: someone is always available to respond to an alert.

Interesting detail: the oncall team for Incident Response is distributed across Seattle, Dublin, and Sydney, which is a “follow the sun” rotation, meaning nobody is oncall when it’s nighttime where they are.

Oncall team members are a mix of senior+ engineers and relatively junior engineers onboarded after training in how to handle alerts and run calls. For significant events like this latest outage, an oncall “AWS Call Leader” is paged automatically. These are more tenured folks, usually principal up to distinguished engineers.

All AWS Service teams and products run their own independent oncall teams. The AWS Incident Response group is a “safety net” which coordinates large-scale events.

AWS has a “you build it, you run it” mentality: teams have autonomy to decide which service they build, the technologies used, and how they structure it. In return, they are accountable for the service meeting the uptime target; for production services, this means having an active oncall. We cover more about oncall practices in Healthy oncall practices, and detailed compensation philosophies in the Oncall compensation article.

Gavin’s background at AWS

For this article, Gavin McCullagh shared the story of how the incident unfolded from his point of view. Gavin joined Amazon’s Dublin office in 2011 and worked for many years as a DNS and Load Balancing Specialist, including working on Route 53 Public DNS, and Virtual Private Cloud (VPC) DNS Resolver. Nowadays, Gavin works in a team focused on Incident Response and Resilience for AWS services and customer applications. This includes the AWS Incident Response team, and services like Application Recovery Controller and Fault Injection Service.

Since his early days there, Gavin has been a regular in large-scale incident calls as his first team – Load Balancing – was regularly paged into major incidents because it is central to AWS, and telemetry from load balancers can greatly help with debugging.

2. Mitigating the outage (part 1)

All times below are in Pacific time.

19 Oct, 11:48 PM: the incident begins and regional health indicators degrade, triggering an alert. Minutes later, AWS Incident Response oncall is paged. The AWS call leader also joins the call. Gavin joins despite not actually being oncall because he figures he might be able to help – which turned out to be a good hunch.

Rapid triage

Triage starts immediately. At this point, 100+ services were showing problems within us-east-1 due to the broad nature of the outage. From experience, the Incident Response Team has learned that it pays to start systematically debugging common layers of the stack. For such a broad outage, it’s usually a core building block with an issue.

Below is a rough checklist of what the team works through:

The Incident Response team begins by going through a checklist in roughly this order:

1. Investigate whether there are power failures or significant network failures

2. Check if the software-defined network and load balancing layers are healthy

3. Look for issues within DNS and authentication/ authorization services

4. Confirm that core services are healthy which most AWS products build on; like KMS, DynamoDB and S3

Two problems at once

As the team went through the list, two coincidental issues were uncovered within a minute of each other.

Sunday, 19 Oct, 11:48 PM: a networking failure event starts. Network packet loss is detected in the “border” network which connects the us-east-1-az1 availability zone to the wider AWS backbone and internet. A Network Border Group is a unique group from which AWS advertises public IP addresses.

11:48 PM: DynamoDB degradation starts. As packet loss is being detected, DynamoDB also starts to degrade.

11:52 PM: AWS Incident Response automatically detects an impact upon services in us-east-1, and pages the incident response oncall network engineers, and affected service oncalls.

11:54 PM: Automated Triage suggests Networking. Incident response’s automated metric triage system completes, noting a) a change in network performance (increase in packet loss) correlated with the time of the event start, and b) that DynamoDB’s metrics have changed at the same time. Typically, this would suggest a network issue is the cause of the DynamoDB issue.

Monday, 20 Oct, 00:08 AM: after failover to a secondary conference system (more later), there’s an investigation into the networking event. As the network event is “lower in the stack” and initially appears to explain the impact on other services (including DynamoDB), the team focuses on fixing the networking issue.

00:26 AM: Red herring. While the Incident Response team is busy resolving the networking failure in the AZ, someone posts in the incident Slack channel:

“Hang on, why is DynamoDB not resolving in DNS, packet loss would not cause that?”

This is when the Incident Response team realizes that while the networking event has some impact, something bigger is also happening that’s not caused by the outage. Whatever it is, it’s impacting DynamoDB.

This smaller network outage is lower in the stack and started a few seconds before the first DynamoDB alerts, which led the experienced team off track into mitigating the smaller outage.

20 Oct, 00:27am: the call is divided. The DynamoDB investigation is split into its own call, so two technical calls run in parallel:

1. DynamoDB outage: the focus of the dedicated call

2. Networking outage: resolving the “smaller” outage continues

Resolving the DynamoDB outage

To bring more senior leaders into the call, Gavin messages three longtime DynamoDB frontend engineers, and asks Incident Response to page them, too.

00:31am: realization that it’s a DNS issue. Using the ubiquitous “dig” tool, the team sees that something is wrong with DynamoDB’s DNS records.

00:35-00:50am: engineers who designed and authored the DNS Enactor service join. The root cause of the issue with DynamoDB ended up being identified as a race condition within a DNS Enactor service, as previously covered.

The quorum (collective) of seven DynamoDB engineers includes folks who built and authored a good part of the DNS Enactor and related services, and who have a combined Amazon tenure of over 50 years. With domain experts on the call, the incident responders let them figure out what has gone wrong with DNS Enactors, while looking for ways to mitigate faster.

As the team will later learn, the cause of the event is an edge case bug in how the three Enactors interact, as covered below.

Hitting the “automation paradox.” The DynamoDB team has built a system with DNS Enactors that automatically does a lot of complicated work to keep their DNS updated. But now that it’s gone wrong, it’s time to manually fix the DNS records.

Here’s where the automation paradox kicked in: the team had never had to manually overwrite the DNS zone files before, as they had a system that could reliably do this! So, it took some time to package up and deploy the change to DNS records via manual intervention.

01:03 AM: a parallel mitigation, before the “full fix.” The problem is that the DNS records for DynamoDB are empty due to the race condition. The first order of business is to restore something that works and relieves customers – even before understanding exactly how things went wrong.

The incident response team prepares a “quick” partial mitigation: within the internal private network, call us-east-1 “Prod” (yes, this really is the name!) and apply an Response Policy Zone (RPZ) override on the DNS resolvers to force the DynamoDB IPs into place, for only this internal network. As it hosts many AWS services, this will help restore services like Identity and Access Management (IAM) and Secure Token Service (STS).

01:15 AM: mitigation. The incident team performs an override to force DynamoDB IPs into the us-east-1 Prod DNS on DynamoDB’s most common domain name. Then, the incident team performs a second override of DynamoDB IPs on DynamoDB’s other domain names. At this point, services like IAM, STS, and SQS in us-east-1 recover.

02:15 AM: fix for public DynamoDB DNS found and applied. The team figures out how to fix the public DNS records for DynamoDB, and applies this to restore DynamoDB for all customers.

The engineers look through the zone and realize the top-level alias record for the DynamoDB domain is pointing at a non-existent tree of records (the old “plan” was deleted). They also note that a set of backup trees which the system maintains are still present. As a result, their task is to stop the automation system to prevent it from interfering, delete the broken alias record, and create a replacement that points at the rollback tree.

02:25 AM: DynamoDB recovered. The first part of the incident is resolved. However, Amazon EC2 continues to have issues, and Network Load Balancer (NLB) would have problems later that night.

3. What actually caused the outage?

AWS runs three independent DNS Enactors, and each takes DNS plans and updates Route 53. DNS Enactors are eventually consistent: meaning that the DNS plans in any single Enactor will eventually be consistent with the source of truth. This is unlike strongly consistent systems where you can be certain that after writing something, the whole system is consistent.

Using an optimistic locking mechanism is a common solution to prevent multiple writers from changing the same record. The AWS team wanted to achieve this: have only one Enactor writing to Route 53 at any single given time.

You need to have some kind of key-value database to implement optimistic locking. For example, you could use DynamoDB for this:

The problem with this approach would have been that DynamoDB itself depends on DNS Enactors, so this would create a circular dependency:

Instead, the AWS team solved the locking problem in a clever way by using Route 53 as the database for optimistic locking. As Senior Principal Engineer Craig Howard explains:

“We didn’t want to add any other dependencies [to DNS Enactors]. Right now, all we depend on is the DNS Plan, and Route 53 (Amazon’s DNS service.) Can we use Route 53 to create a lock? Well, you can!

You create a TXT record, and the existence of that TXT record indicates who the holder of the lock is. The value of the text record is the host name of the lock holder, and then you put the epoch [length of time] in seconds for how long this lock is alive for.”

The AWS team used the transactional characteristic of Route 53: the Route 53 control plane ensures that “CREATE” and “DELETE” operations for TXT are transactional – they either succeed or fail.

So, this is how a lock is created:

CREATE “lock.example.com” TXT “<hostname> <epoch>”

As lock creation is transactional, whenever a DNS Enactor wants to create a new lock for its own use, it first deletes the old lock because it can be certain that lock creation /deletion is transactional. Then it creates the new lock:

DELETE “lock.example.com” TXT “<hostname> <old epoch>”
CREATE “lock.example.com” TXT “<hostname> <new epoch>”
// Insert updated Route 53 DNS records

Lock contention is what started the outage. In this case, one Enactor was very unlucky and failed to gain a lock several times in succession. At that point, the plan it is trying to install is older than anyone had ever envisioned. By the time this “unlucky” Enactor gains the lock, an up-to-date Enactor takes over and its “record clean-up” workflow detects a very old plan and deletes it. This was an older plan than the team expected.

Craig explains how this strange race condition played out in this re:Invent talk, shared earlier this week.

What started the outage? As the team will later learn, the cause of the event is an edge case bug in how the three Enactors interact. They are designed to operate independently for resilience, with each Enactor optimistically taking a lock on the DNS zone, making their changes, and releasing the lock. If an Enactor fails to obtain a lock, it backs off and then tries again later.

4. Oncall tooling & outage coordination at Amazon

Let’s hit pause on the incident mitigation to look into some unique parts of Amazon’s oncall process.

Read more

Begin researching developer productivity, and you’re likely to soon encounter the work of Nicole Forsgren, as lead author of the best-selling book Accelerate, and creator of the widely-adopted DORA and SPACE developer productivity frameworks. And, beyond SPACE is the DevEx framework for productivity measurement, created by Abi Noda in collaboration with Nicole.

When I heard that the pair’s work together has continued in a new book about developer experience, I wanted to know more, and had the chance to sit down for a conversation about this big project of theirs.

Frictionless was released two weeks ago, and Nicole and Noda have been nice enough to share an excerpt from their new title with readers of the Pragmatic Engineer, below.

The back cover of Frictionless reads:

“AI can generate code in minutes — so why does shipping software still take forever?

The answer is friction: the invisible barriers that turn quick wins into endless delays. While your competitors ship daily updates, your developers burn out fighting broken tools instead of solving real problems”.

Personally, the lack of impact by AI coding tools to date upon the speed of development and quality of software, is amusing to me. If I take the apps I use, websites I visit, and services I access via software, I don’t see signs of faster iteration, higher quality, and fewer bugs since AI started shaking up the tech industry. An exception might be AI labs that release new features at eye-catching speed, often faster than much nimbler startups, but it could be argued those begin as experimental features.

What I like about Frictionless is that it gives a step-by-step approach for developers to smooth their own workflows and be more productive. If it helps teams and companies to move faster and produce more and better software – this is a win for everyone!

This article begins with our conversation, followed by two excerpts from the new book. We cover:

1. How Frictionless relates to Accelerate, DORA, and SPACE. An obvious question, but I had to ask.

2. Writing process. Frictionless took more than 3 years to write – including a “full rewrite”

3. Making the business case to reduce developer friction. Excerpt from the new book, chapter 9.

4. How AI changes (and doesn’t change) the metrics you need. Chapter 59 of the book.

To hear more about developer productivity from Nicole, see this podcast episode with her in The Pragmatic Engineer Podcast.

My usual disclaimer: As with all my recommendations, I was not paid to recommend this book, and none of the links are affiliates. See my ethics statement for more details.

1. How Frictionless relates to Accelerate, DORA, and SPACE

When developer productivity is discussed, your book Accelerate often gets a mention. How does Frictionless relate to it – and what makes it different?

Nicole: “Great question! The two are complementary but different in their foundations. Accelerate distills years of research connecting software delivery performance to business outcomes—it tells you what matters and why, but leaves the how to the reader. Frictionless is built from years of hands-on work advising teams on DevEx improvements, and it’s designed to help you actually do the work.

Where Accelerate is research-backed insights, Frictionless is a practitioner’s guide. It gives you a seven-step process you can start wherever you are, plus organizational change frameworks, metrics guidance, and 100+ pages of workbooks. Many of the practices are informed by research, but the book is ultimately about what we’ve seen work in the real world when teams try to improve developer experience.

So, if Accelerate is the ‘why,’ Frictionless is the ‘how’”.

DORA, SPACE and this book

To what extent does the new book build upon the DORA and SPACE frameworks?

Nicole: “They’re all deeply connected, Frictionless is really the natural evolution of this work.

DORA and Accelerate was about improving software delivery through technical, cultural, and process changes. Yes, we focused on business outcomes, but a core goal was always to make work better for engineers. Frictionless extends that same idea: start by identifying inefficiencies in the system, talk to developers about where their work is difficult, and you can improve both business outcomes and developer experience.

SPACE gave us a framework for measuring complex work like software development. Measurement is critical to any DevEx initiative, and we dedicate a full chapter to how AI is changing what we need to measure and how we should think about metrics differently.

But here’s the key difference: Frictionless goes beyond measurement. Metrics alone don’t create change. You also need to sell the initiative to leadership, advocate for change across your organization, plan communications, manage organizational change—all the messy, human work of actually making improvements happen. That’s what this book is designed to help with”.

2. Writing Process

It’s always interesting to learn what goes into writing a book. What made you both decide to take on this project?

Nicole: “This is a great question, since I didn’t think I would write another book. However, I realized a couple of things after talking to teams and leaders across many companies and industries: there was increased attention upon, and a push to improve, developer experience. Also, many questions and challenges were very similar: how do I start, how do I prioritize, and how do I convince the business that DevEx is important?”

Abi: “Over the past several years, I’ve worked hands-on with Developer Productivity (DevProd) leaders at hundreds of organizations. The same questions and challenges came up repeatedly, so a book felt inevitable. When Nicole reached out about teaming up in 2022, it was a no-brainer to finally make it happen. As a fun fact, I bought the domain developerexperiencebook.com in February 2021 – so about a year before we started writing the book!”

How long did the writing take?

Nicole: “The whole process took about 3.5 years, but it wasn’t linear. I had a false start where I’d written several thousand words and realized it was way too pedantic; basically a research book with chapters explaining methods, but nothing immediately actionable. I scrapped it and started over.

This was the point when I reached out to Abi. We’d written a couple of papers together before, and his work at DX was giving him incredible insights into the key challenges teams face, and what actually works to improve DevEx. The combination is perfect: I brought the frameworks and experience from working with teams, and he brought the frontline practitioner perspective and patterns he was seeing across hundreds of companies.

Together, we built something practical: the step-by-step process, the workbooks, the real-world examples. Much better than my first attempt!”

Where did you get inspiration for the practices shared in the book?

Nicole: “Honestly, it all comes from actual work: collaborating or advising actual teams and actual DevEx improvement projects”.

Abi: “As Nicole mentioned, we didn’t go reaching for ideas or conduct research specifically for the book. Everything we put into it comes directly from our existing research and practical experience”.

What are your favorite parts of the book, and do any details stick out in particular?

Nicole: “I love the section on treating metrics like a product. But if I had to pick one story, it’s how Dave Anderson was able to drive improvements at Amazon. What made it stick with me was how he could influence real change without authority by sharing a dashboard”.

Note from Gergely: longtime newsletter readers and podcast listeners may find Dave Anderson’s name familiar: he authored the popular guest post, A Day in the Life of a Senior Manager at Amazon, and has been a guest on the Pragmatic Engineer podcast.

Abi: “Translating developer experience into business terms (Chapter 9) is a topic that’s especially meaningful to me. Developers – and most engineering leaders – instinctively understand how essential a strong developer experience is. But helping non-technical leaders see its strategic importance can be challenging.

In the chapter, we highlight the story of Mike Fisher, former CTO of Etsy, who led a transformative developer-experience initiative that not only accelerated engineering teams, but was backed by hard, quantifiable ROI”.

What has the feedback about the book been like, so far?

Nicole: “It’s been really positive, overall. A lot of folks are finding ways to get unstuck, or engage with people at work in new ways. I’ve had a few folks insist the AI transformation will change everything, but the core of the book is really about how we build software, how to spot inefficiencies, and how to effect change.

And, as many of us are seeing, AI amplifies everything. Sometimes, it’s how much we can prototype and experiment, and other times, it’s our friction points and bottlenecks. So, while workflows may change, the fundamental importance of reducing friction doesn’t”.

Abi: “Like shipping a product, putting a book out into the world is exciting and also nerve wracking! I’ve been grateful for all the support we’ve been receiving, as well as feedback.

I think this book lands at a critical moment for the industry. AI is creating enormous excitement and rapid change, and that makes the fundamentals of measurement and developer experience more important than ever”.

Frictionless is organized into five parts:

• Part I: Understanding DevEx

• Part II: The three essential parts of DevEx (feedback loops, flow state, cognitive load)

• Part III: Making the business case [to invest in DevEx]

• Part IV: Improving DevEx: a 7-step process

• Part V: Evolving and sustaining DevEx

With that, let’s jump into two full chapters of the book, from parts III and V.

The excerpts below are from Frictionless, by Nicole Forsgren and Abi Noda. Copyright © 2025 by Nicole Forsgren, Abi Noda. Published by Shift Key Press. Used with permission.

3. Making the business case to reduce developer friction

Chapter 9 from Part III: Making the Business Case.

Understanding developer experience and its business impact is just the beginning. The real challenge is securing the resources and organizational support needed to make meaningful improvements. Most DevEx initiatives fail not because of poor technical execution, but because they can’t effectively communicate their value to decision-makers.

In this part, we explore how successful teams have translated developer pain points into compelling business cases that resonate with executives. You’ll learn how to connect DevEx improvements to financial outcomes, align with existing strategic priorities, and position your work as a solution to leadership’s most pressing concerns. These approaches are designed to be durable—while the specific tools and technologies you advocate for will evolve with AI and other innovations, the fundamental methods for investigating problems, gathering data, and communicating business value remain constant. Whether you’re seeking initial funding for DevEx improvements or defending existing investments, these chapters will show you proven approaches for building leadership support.

Translate developer experience into business value

C-suite executives speak the language of business outcomes.

While “improving developer satisfaction” or reducing lead times” might resonate with engineering leaders, executive decision-makers need to see clear connections to financial impact. Business leaders are evaluated on metrics like revenue growth, profitability, and market share—so your DevEx proposals must connect directly to these financial outcomes.

This doesn’t mean executives don’t care about developer well-being. Rather, they need to understand how developer experience investments translate to business results. This financial framing is crucial when selling your strategy (Step 5) and demonstrating value (Step 7).

Talk about recovering time: Convert developer hours into dollar value. Developer time has clear financial value. You can:

• Measure hours spent on unnecessary toil or waiting for builds/tests.

• Multiply this time by the fully loaded hourly cost of developers.

• Present these savings as “recaptured productivity dollars” or “free headcount.”

For example, if your team of 50 developers each wastes 5 hours weekly on build-related delays at a fully loaded cost of $100/hour, that’s $25,000 in weekly productivity losses—or $1.3 million annually.

Alternatively, express this as “free headcount,” for example, 50 developers × 5 hours lost weekly = 250 total hours wasted. Based on a 40-hour workweek, that’s equivalent to gaining 6.25 additional developers without increasing your budget. This framing resonates particularly well when hiring freezes are in place or during tight labor markets. To safeguard your credibility, be sure to actually measure real, credible time savings.

The financial impact is substantial. When teams improve developer experience, those benefits compound. A $250,000 developer who recovers just 10 percent of their time by avoiding tooling friction represents $25,000 in reclaimed value. Multiply that across an entire engineering organization, and it’s equivalent to dozens or hundreds of “free” developers.

But the benefits don’t stop there—faster toolchains enable organizations to build, experiment, and iterate more rapidly, creating business value that can climb into the millions. And that’s before accounting for the compounding effects on delivery timelines, team morale, and customer satisfaction.

Etsy’s $2.5M DevEx investment: A business case in action

As Etsy’s engineering team rapidly grew from 250 to nearly 1,000 developers, leadership recognized that productivity would suffer without investment in developer experience.

CTO Mike Fisher’s approach to securing buy-in was straightforward: Translate improvements into business language. Instead of explaining technical changes to deployment processes, he told executives the work would save 50 percent of engineering hours spent waiting for deployments—equivalent to a quarter of a full-time engineer’s salary. The result was an initiative that dedicated 20 percent of engineering capacity to improve DevEx in four key areas: Crafting products, developing and deploying, building with data, and reducing toil.

The initiative, which took 18 months, was so successful that it led to ongoing investment in developer-experience work, integrated into Etsy’s standard operations. Their key insight: Scale people, process, and technology together, or productivity suffers regardless of growth.

Talk about saving money: Quantify cost savings. Improved developer experience delivers tangible cost savings that directly impact your financial statements. By enhancing automation, streamlining workflows, and eliminating redundancies, you can substantially reduce operational expenses.

For example, strategic DevEx investments can:

• Decrease vendor costs through process optimization and tool consolidation.

• Reduce cloud computing expenses by improving test efficiency (eliminating redundant tests).

• Lower infrastructure costs through optimized build processes that require less processing power.

• Minimize expensive production incidents through better quality controls.

When making your business case, focus on these hard cost savings with specific projections whenever possible. This includes both the immediate savings from reduced delivery costs—as developers ship features faster with less friction—and the long-term savings from improved talent retention. Executives respond strongly to initiatives that demonstrate clear paths to expense reduction while maintaining or improving output quality.

Strategic standardization: How Block delivered millions in savings

Block is a leading fintech company made up of brands like Square, CashApp, and Tidal. Rather than pursuing standardization for its own sake, they centered their DevEx strategy on consolidating tools to create well-supported golden paths for developers.

When making their business case, Block’s team could have focused on abstract benefits such as developer satisfaction or code throughput. Instead, they focused on the dollar-impact that engineering friction and avoidable reliability incidents were having on the business..

The business-focused approach paid off. Their developer experience initiative was funded and delivered impressive results in 12 months. Thanks to investments in better developer environments, golden paths, and AI tooling, they saw millions of dollars in documented savings and double-digit improvements in developer satisfaction scores.

Talk about making money: Accelerating revenue through DevEx. While developer experience initiatives aren’t direct revenue generators, they significantly accelerate your existing revenue streams. For software-driven companies, revenue growth depends on five key factors:

1. Feature velocity (how quickly new capabilities reach customers).

2. Feature quality (how well new capabilities meet customer needs).

3. Experimentation speed (how quickly you can test and iterate on new ideas).

4. Service reliability (how consistently your products perform).

5. Security posture (how well you protect customer data and trust).

All five factors depend on effective developer workflows. Consider the AI transformation, where companies racing to integrate AI capabilities into their products are pursuing increased revenue, competitive advantage, and expanded market share. However, these AI features must be:

• Designed and coded.

• Reviewed for quality.

• Built and tested.

• Integrated into existing systems and products.

• Securely deployed.

• Maintained and improved.

When developers wrestle with suboptimal tools and fragmented processes, these technical bottlenecks become business problems. Your company faces delayed market entry, compromised quality, and security vulnerabilities—all directly hitting revenue.

Consider this: If competitors release AI features quarterly while your team needs six months due to workflow inefficiencies, you’re not just accumulating technical debt, you’re experiencing actual revenue losses and diminishing market relevance. Smart DevEx investments create a multiplier effect where your existing teams deliver more value faster, enhancing revenue streams and capturing market opportunities before competitors do.

Accelerating revenue growth: Capital One’s DevEx transformation

Capital One dramatically transformed their market position by strategically investing in developer experience and cloud infrastructure.

Before their DevEx overhaul, launching new financial products took 6-9 months, limiting the company’s competitive agility. By standardizing development environments, implementing modern CI/CD pipelines, and embracing cloud-native technologies, Capital One slashed delivery times from months to weeks or days.

The business impact was fast and measurable. Faster deployment cycles meant Capital One could capture market opportunities before competitors, respond quickly to customer needs, and optimize revenue-generating features through rapid iteration.

Capital One secured executive buy-in by directly connecting their DevEx investments to accelerated time-to-market and customer acquisition—metrics that translate directly to revenue growth. Their story demonstrates how improving developer workflows isn’t just a technical win—it’s a powerful revenue accelerator.

Talk about proven correlations: Link technical metrics to business outcomes. Sometimes the most powerful business cases come from identifying correlations between technical metrics and business outcomes.

In How to Measure Anything, Douglas Hubbard writes, “If you can correlate two things to each other, and then if you can correlate one of them to money, then you can express them both in terms of money.”

At eBay, a breakthrough analysis showed that every 10 millisecond improvement in page speed translated to millions of dollars in increased web purchases. This correlation transformed how the business viewed page speed—from a technical concern to a critical revenue driver deserving investment.

A specific example of this approach is the developer experience index (DXI), developed by DX. The DXI score is the average of 14 survey questions that ask developers about different aspects of their development process. By correlating this score against self-reported time loss, DX has found that each one-point gain in DXI score translates to saving 13 minutes per week per developer, equivalent to 10 hours annually. This correlation allows organizations to predict financial impact from DevEx improvements.

You can apply this same approach to developer experience:

• Correlate developer satisfaction scores with reduced time-to-market.

• Link build system reliability to fewer production incidents.

• Connect onboarding experience improvements to faster new-hire productivity.

However, be cautious of trying to tie a developer productivity metric straight to revenue—a common mistake in this space. DevEx leaders attempt this, they don’t know how to get started, and then they give up trying to convince their management of the value of their work.

Storytelling matters more than the exact numbers

When building your DevEx business case, perfect is the enemy of good. Many leaders get stuck trying to calculate perfectly precise ROI figures or collect perfect data. This perfectionism is rarely necessary.

In most situations, reasonably credible data is sufficient—what really matters is how you weave that data into a compelling narrative. While data points make good headlines or proof points, they need a clear, concise story to be meaningful to decision-makers.

Remember: Even the most impressive numbers rarely speak for themselves. Your presentation skills, narrative structure, and understanding of your audience’s priorities will ultimately determine whether your business case succeeds. We discuss this more in Step 5: Sell Your Strategy.

4. How AI changes (and doesn’t change) the metrics you need

The excerpt below is Chapter 59 from Part V: Evolving and Sustaining DevEx

AI changes what developers do every day, so your metrics need to change too.

The good news? The measurement fundamentals you just learned still apply. As AI tools become woven into daily development work, you’re probably wondering: “Do I need to throw out everything I know about measuring developer experience?” The short answer is no—but the longer answer is more interesting.

The fundamentals still matter. The SPACE framework and DORA metrics remain relevant because developers still need fast builds, reliable deployments, good documentation, and strong collaboration—and AI often amplifies the pain of bad foundations rather than masking them. Flow and friction haven’t gone anywhere: Developers still need focus, and slow pipelines, flaky tests, and unclear ownership still break it. And while AI shifts what developers do, it doesn’t change their need for clarity, autonomy, mastery, and purpose.

Applying SPACE to AI-Augmented Development

In Chapter 58, we introduced the SPACE framework as a way to balance your metrics across five dimensions. That framework remains just as valuable when measuring AI-augmented workflows—you just need to ask slightly different questions:

• Satisfaction: How do developers feel about AI tools? Do the tools improve or frustrate their work experience? Are they satisfied with the quality of AI suggestions?

• Performance: Are outcomes improving with AI assistance? Track defect rates, feature completion times, and time-to-resolution for incidents.

• Activity: What are the AI usage patterns? Monitor suggestion acceptance rates, what types of work get delegated to AI, and where AI gets used most (tests, docs, debugging).

• Communication: How does AI affect collaboration? Are code reviews shifting from syntax to architecture? Are developers asking AI instead of peers? Is documentation improving or declining?

• Efficiency: Where does AI save time versus create overhead? Measure workflow changes, time saved on routine tasks, and friction from context gaps or validation effort.

Consider adding a sixth dimension: Trust. How much do developers trust AI-generated code, comments, and recommendations? This affects both adoption patterns and the value teams ultimately derive from AI tools.

AI doesn’t just speed up existing work—it changes workflows. Developers aren’t just writing code; they’re reviewing, prompting, and steering AI. And here’s what most people miss: Most AI usage involves reasoning about information rather than generating it. Developers spend more time using AI to write summaries, compare alternatives, conduct smart searches, and synthesize research than generating code. This means existing metrics that capture time spent coding or lines of code written miss the bulk of AI-augmented work—the time spent crafting prompts, reviewing suggestions, and making decisions based on AI synthesis.

New metrics can help answer new, emerging questions. Track prompting efficiency—how many attempts developers need before getting a useful AI suggestion. Measure validation effort—the time spent reviewing and correcting AI-generated code, which reveals whether AI truly saves time or just shifts where developers spend it. And monitor trust calibration—when developers trust AI too much (shipping bugs) versus too little (wasting time double-checking correct code). These metrics reveal friction that traditional measures like commit frequency or lines changed simply can’t capture.

AI usage telemetry can reveal important patterns and changes in workflows. Capture what types of work developers do themselves versus what they hand off to AI agents—this shows where AI is valuable and may reveal new system blockers. Deep insights also come from watching how AI reshapes team dynamics: Are engineers asking AI instead of peers for help? Are code reviews shifting from syntax nitpicks to architecture discussions? Are new developers getting up to speed faster because AI explains your internal codebase?

Most importantly, collect this data thoughtfully. AI instrumentation typically involves much more detail than other methods. Aggregate at the team level rather than tracking individuals, be transparent about what you’re measuring and why, and work with HR and legal partners early to establish clear privacy boundaries. The goal is to improve developer experience, not create surveillance.

Your metrics product also needs to adapt. Create segmented views that compare AI and non-AI performance. Build feedback loops that reveal where AI adds the most value. And introduce visualizations that track how AI affects collaboration patterns.

Use AI to build your AI metrics

There’s a helpful irony here: While AI changes what you need to measure, it can also help you build that measurement infrastructure faster.

Use AI coding assistants to:

• Generate data pipeline code for collecting AI telemetry

• Create dashboard prototypes and visualizations

• Write scripts to analyze prompt patterns and acceptance rates

• Build data transformation logic for aggregating team-level metrics

What once required weeks of custom development can now be scaffolded in hours. Just remember to validate AI-generated code carefully—especially for instrumentation that feeds critical decisions. And leverage AI for the scaffolding and boilerplate, while keeping human expertise focused on the measurement strategy and interpretation.

Mixed methods become even more important with AI. Logs tell you what AI suggested and what got accepted, but only surveys and interviews tell you why. Your telemetry might show developers rejecting 80% of AI suggestions for authentication code—but you need to talk to developers to learn whether that’s because AI hallucinates security vulnerabilities, doesn’t understand your internal frameworks, or because developers don’t trust it for security-critical work. The “what” reveals patterns; the “why” reveals the root cause and tells you what to fix.

The shift from output to outcomes becomes critical. AI makes it painfully obvious that lines of code was always a bad metric—but more importantly, it forces us to connect our metrics to what actually matters. Developer productivity isn’t the goal; faster customer value delivery is. That means measuring problem-solving speed (time from idea to working solution), breadth of exploration (how many approaches developers can test), and cognitive load (whether AI helps or overwhelms). These outcome-focused metrics tell you what actually moves the needle on business value, not just code volume.

Behind the scenes, your infrastructure needs to evolve—and not just for developer workflows. Traditional telemetry doesn’t capture AI interaction loops, so you need new instrumentation for prompting workflows, completions, and accept/reject events. These real-time metrics reveal patterns and friction that retrospective analysis can miss.

But don’t stop there—instrument agentic workflows too. Tracking what agents do reveals which tasks are best suited for automation and which require human judgment. It also shows how delegation to agents affects developers’ mental models of the system: Does offloading routine tasks free developers to think architecturally, or does it create blind spots where they lose understanding of how things work?

Start by auditing your existing metrics through an AI lens. Ask: “Does this metric still capture what we care about?” and “What important tasks, workflows, and feedback loops does our data miss?” Don’t throw everything out and don’t start over from scratch—add data that fills important gaps and helps you answer new questions.

Your metrics strategy should also anticipate questions stakeholders will ask about AI adoption: “Where is AI providing the most value?” “Are we seeing quality trade-offs?” “Which teams are benefiting most?” Your metrics should answer these, not just operational velocity questions. Having these insights ready builds confidence in your AI implementation strategy while helping teams make better decisions about where and how to leverage these tools.

The bottom line: AI changes what you measure, but not why you measure it. You’re still trying to understand and improve developer experience—the SPACE framework still applies, as shown in the callout above. The fundamentals remain sound; you just need new instruments to capture the new reality.

End of excerpt.

Takeaways

Gergely, again. Thanks to Nicole and Abi for talking with me, and sharing excerpts from Frictionless. You can buy the book here, and download a free DevEx workbook that accompanies the title.

Get the book

Before AI tools started to spread, larger tech companies invested in developer experience. Amazon calls this area “builder experience”, Spotify uses “R&D experience”, and other places use terms like “engineering experience”, or “maker experience” as per Frictionless.

With AI coding tools in use, there are expectations that productivity should increase for us devs – basically; assuming these AI tools are any good, products should be built faster, and iteration speed should improve. But we seem to be going back to basics in figuring out why this happens at some companies, but not at others.

Engineering leaders finally have a case for the resources of time and money to reduce/remove processes, improve tooling, and make developer experience better at both small and large teams.

I like Frictionless because it doesn’t talk about how to use AI differently or “better”: instead, it provides suggestions about removing annoying friction from devs’ day-to-day lives.

Thanks, Nicole and Abi for writing this book; hopefully it makes the case easier for reducing developer friction at your workplace, too!

For more reading on developer productivity, check out these deepdives:

• How tech companies measure the impact of AI on software development

• Measuring developer productivity: 17 real-world examples. Examples from Google, LinkedIn, Peloton and other companies

• How Uber measures developer productivity

• Measuring engineering efficiency at LinkedIn

• DevEx framework: an overview