Developer productivity with Dr. Nicole Forsgren (the creator of DORA)

Stream the Latest Episode

Available now on YouTube, Apple and Spotify. See the episode transcript at the top of this page, and a summary at the bottom.

Brought to You By

• DX — An engineering intelligence platform designed by leading researchers.

• Sentry — Error and performance monitoring for developers.

—

In This Episode

We’ve previously covered a lot on the surprisingly slippery topic of developer productivity: from how Uber measures it, how LinkedIn does it, an overview of the innovative DevEx framework, and a deepdive of dev productivity metrics used by Google, LinkedIn, Peloton, Amplitude, Intercom, Notion, Postman, and 10 other tech companies.

Every time developer productivity comes up: DORA and SPACE are almost certainly mentioned. So I could not be more excited to have Dr. Nicole Forsgren on the podcast. Nicole is the creator of the widely adopted DORA and SPACE frameworks, co-author of the award-winning book Accelerate and the DevOps Handbook (2nd edition), and author of the State of DevOps reports. She is currently a Partner at Microsoft Research, leading developer productivity research and strategy, and is currently working on a book about developer experience with Abi Noda. It’s safe to say that Nicole is one of the foremost experts in developer productivity — if not the foremost expert.

In this episode, we discuss:

• Why PRs and Diffs are incomplete as a solo metric and how to view them in context

• The importance of a holistic set of metrics for evaluating productivity

• An overview of DORA’s four key metrics, its strengths, and its limitations

• The evolution of processes and tools since DORA, including SPACE

• What developer experience is—and concrete ways to improve it

• Common characteristics of highly productive engineering teams

• How faster onboarding might challenge Brook’s Law

• How AI tooling is impacting developer productivity and best practices for experimentation

• And much more!

Takeaways

My biggest takeaways from this episode:

1. Measuring the number of PRs is controversial for a good reason. Measuring any single one “output” can be misused: and when devs know it is being measured, they will optimize for it.

However, measuring PRs is important: but to do it well, don’t look at it as an individual performance metric. Instead, use it to understand how well (or not well) systems across the team and company are working. For example, what systems are getting in the way of PRs taking long to merge?

2. DORA and SPACE both have their own limitations. DORA is very well-defined in the metrics it measures. However, a massive limitation it has is how it only measures from commit to production.

SPACE can be used to measure the complete developer workflow – including e.g. planning, coding, and even post-release. In turn, it is a more vague framework where you need to put in more effort to make it useful for your envirnment.

3. AI developer tools don’t change DevEx fundamentals. DORA and SPACE are still relevant. AI tools might be able to improve iteration speed, or make certain tasks more helpful. It’s really an open question how exactly this will play out. There’s a fair chance these tools significantly change how we do development. So if you’re a dev: experiment with them!

4. Developer experience (DevEx) is not necessarily great at large companies and poor at startups. Google is known to have standout developer experience thanks to so much investment it put into systems – but not all large companies are like this!

Startups have less resources to invest specifically in improving DevEx: but then again, they have less infrastructure in-place! The more custom infrastructure you have, the more painful DevEx tends to be (and the more in-house tools need to be built to mitigate this)

The Pragmatic Engineer deepdives relevant for this episode

• Measuring Developer Productivity: Real-World Examples

• A new way to measure developer productivity – from the creators of DORA and SPACE

• Measuring Engineering Efficiency at LinkedIn

• How Uber is Measuring Engineering Productivity

• Measuring software engineering productivity

Timestamps

(00:00) Intro

(02:03) PRs and Diffs and how to view them in the right context

(07:42) EngThrive at Microsoft

(10:26) The importance of having a holistic set of metrics in evaluating productivity

(17:00) The four key metrics of DORA

(23:57) The evolution of processes and tools since DORA, including SPACE

(26:40) An explanation of developer experience — and ways to improve it

(30:44) Devex at startups vs. larger companies

(34:20) Why measuring developer productivity is so difficult

(39:05) How to make a case for platform teams

(44:34) Common characteristics of highly productive teams

(51:01) Brook’s law and how faster onboarding might make it irrelevant

(52:49) Onboarding for internal transfers

(54:18) Shifting culture towards technology first

(58:36) How middle management can improve engineering culture

(1:03:36) How AI tooling is impacting developer productivity

(1:06:42) Potential use cases for AI

(1:08:40) A case for experimenting with AI coding tools and how to maintain flow state

(1:15:30) Rapid fire round

A summary of the conversation

Measuring Developer Productivity

• PRs and diffs can be both good and bad signals when measuring developer productivity. Many leaders focus on the traditional economic definition of productivity: the rate of output or output per input.

  • PRs can give a view into the work being done, but senior engineers may have lower PR output due to other responsibilities like unblocking others, architecture, mentoring, and recruitment. It's important to consider the context and who is being measured.

  • A constellation of metrics is better than a single metric for a holistic view. So yes: it can be helpful to get data on PRs – as long as you get a lot more other data points as well!

• Microsoft's EngThrive: this framework (which Nicole works on) uses multiple metrics across dimensions, inspired by the SPACE framework, alongside qualitative feedback.

• Easy-to-operationalise measures can be useful. However, they only show a few things and may ignore important aspects.

• SPACE: a framework that groups metrics into:

  • Satisfaction

  • Performance

  • Activity

  • Communication

  • Efficiency.

  • Nicole is a co-author of the framework. It’s one that is getting a lot wider adaptation. See an overview of the SPACE framework

• Absent engineering leadership is unhelpful. No matter what metrics you use: if engineering leadership is not connected with the work, outcomes are likely to be a lot worse.

• Metrics collected in a holistic way, on the other hand, can guide decisions about company-wide blockers.

DORA frameworkand its evolution

• Nicole is the co-creator of DORA

• DORA can refer to the entire research program. More frequently though, people tend to refer to the 4 DORA metrics:

  • Deployment frequency

  • Lead time for changes

  • Change failure rate,

  • Time to restore service

• These 4 metrics are a good indicator of how well a development pipeline is working

• DORA can be adapted for different environments, such as air-gapped systems, by redefining the scope of the deployment pipeline.

• DORA does not show the full picture. DORA metrics serve as a signal for how well a team is doing, but it does not show the complete picture. DORA focuses on commit to production. This is a part of the engineering process that can be made pretty efficient.

• Fast feedback is important, as is the ability to set up environments and provision resources quickly.

• SPACE evolved from DORA. SPACE considers the entire end-to-end toolchain. SPACE metrics can be applied to specific components, like PRs. DORA is essentially one of many instances of SPACE.

Developer Experience

• Developer experience is a developer's lived experience. This can be “good” – it can also be “bad!”

• A good developer experience minimises friction, blockers, and confusion

• Security and compliance: these are important! However, automation can minimise manual processes for developers. A secure and compliant dev process does not need to be cumbersome.

• Delays and cognitive load from inefficient processes negatively affect developer experience.

• Both large and small companies can have good or bad developer experiences. Large companies have more resources but may struggle with legacy systems and bureaucracy. Startups can move fast but lack infrastructure. Google has an exceptional developer experience because they invest in it and don't tolerate friction.

Improving Engineering Teams

• Most work in software is invisible, and systems are increasingly complex.

• Leaders and executives often don't feel the pain of developers.

• To make a case for investment in tooling: use both data and stories. Present the trade-offs realistically.

  • Acknowledge that there's a tipping point where further investments won't deliver much more progress.

  • Frame requests in terms of trade-offs and potential for repurposing teams.

• High-performing teams:

  • Exhibit psychological safety, curiosity, and openness to better ways of doing things.

  • Onboarding time is a great indicator of team efficiency.

    • A “dummy pull request” early during the onboarding phase (e.g. day 1!) can significantly increase productivity. Try it!

    • If onboarding is slow: adding people to a late project can backfire. But if onboarding is fast: this can actually work!

• Engineering culture transformations:

  • These require changing the entire company culture, not just the tech culture.

  • Changing how people do their work can lead to cultural impacts.

  • Introducing faster tools and ways of working changes people's lived experience.

  • To improve the engineering culture, summarize observations, seek feedback, and involve others in finding improvements.

  • Aim for quick wins with visible impact, like hack days to address paper cuts.

Impact of AI on Developer Productivity

• DORA and SPACE relevance:

  • DORA metrics should remain relevant

  • SPACE is still applicable for assessing satisfaction, performance, activity, communication, and efficiency.

• Changes AI is likely to bring:

  • There’s some chance that AI tools einvent development and software engineering, particularly in IDEs and testing.

  • Coding assistants: AI-powered ones can improve code readability and unit test success rates

  • PR reviews: an obvious place for AI to help

  • Support roles like release engineering and deployment: could be a use case

• Experiment with AI!

  • Experimenting with AI tools and adapt to your workflows.

  • Assess AI tools to see what works, and focus on problem-solving and architecture.

  • When you find the workflows where AI is helpful: you might be able to preserve flow better

  • Turn off autocomplete and autosuggestions if they are an interruption!.

• Consider AI as a different way to work, such as "driving and reviewing" or "guiding".

• LLMs are changing software engineering rapidly.

Book recommendations

Nicole recommends reading:

• Inspired by Marty Cagan

• Outlive by Peter Attia

• Ender's Game for some good fiction reading

Resources & Mentions

Where to find Dr. Nicole Forsgren:

• X: https://x.com/nicolefv

• LinkedIn: https://www.linkedin.com/in/nicolefv/

• Website: https://nicolefv.com/

Mentions during the episode:

• Microspeak: Sats: https://devblogs.microsoft.com/oldnewhttps://newsletter.pragmaticengineer.com/p/developer-productivity-a-new-frameworkthing/20100914-00/?p=12873

• Measuring Software Engineering Productivity: https://newsletter.pragmaticengineer.com/p/engineering-productivity

• Hawthorne effect: https://en.wikipedia.org/wiki/Hawthorne_effect

• Accelerate: The Science of Lean Software and DevOps: Building and Scaling High Performing Technology Organizations: https://www.amazon.com/Accelerate-Software-Performing-Technology-Organizations/dp/1942788339

• What is an Air Gap?: https://www.ibm.com/think/topics/air-gap

• DORA: https://dora.dev/

• Quantifying the impact of developer experience: https://developer.microsoft.com/en-us/developer-experience

• A new way to measure developer productivity – from the creators of DORA and SPACE: https://newsletter.pragmaticengineer.com/p/developer-productivity-a-new-framework

• Ciera Jaspan on LinkedIn: https://www.linkedin.com/in/ciera/

• Emerson Murphy-Hill on LinkedIn: https://www.linkedin.com/in/captainemerson/

• Inside Stripe’s Engineering Culture - Part 1: https://newsletter.pragmaticengineer.com/p/stripe

• Inside Stripe’s Engineering Culture: Part 2: https://newsletter.pragmaticengineer.com/p/stripe-part-2

• David Singleton on LinkedIn: https://www.linkedin.com/in/davidpsingleton/

• Courtney Kissler on LinkedIn: https://www.linkedin.com/in/courtney-kissler/

• Brian Houck on LinkedIn: https://www.linkedin.com/in/brianhouck/

• Brook’s law: https://en.wikipedia.org/wiki/Brooks%27s_law

• Satya Nadella on LinkedIn: https://www.linkedin.com/in/satyanadella/

• Steve Ballmer on LinkedIn: https://www.linkedin.com/in/steve-ballmer-7087a8157/

• John Shook: https://www.lean.org/about-lei/senior-advisors-staff/john-shook/

• Does GitHub Copilot improve code quality? Here’s what the data says: https://github.blog/news-insights/research/does-github-copilot-improve-code-quality-heres-what-the-data-says/

• Two developers built a game that sold 1M copies. How?

• Reading Between the Lines: Modeling User Behavior and Costs in AI-Assisted Programming: https://www.microsoft.com/en-us/research/publication/reading-between-the-lines-modeling-user-behavior-and-costs-in-ai-assisted-programming/

• Abi Noda on LinkedIn: https://www.linkedin.com/in/abinoda/

• Jason Entenmann on LinkedIn: https://www.linkedin.com/in/jason-entenmann-06146875/

• Claude: https://claude.ai/new

• Anthropic: https://www.anthropic.com/

• OpenAI: https://openai.com/

• Sonnet: https://www.anthropic.com/news/claude-3-5-sonnet

• Inspired: How to Create Tech Products Customers Love: https://www.amazon.com/INSPIRED-Create-Tech-Products-Customers/dp/1119387507

• Outlive: The Science and Art of Longevity: https://www.amazon.com/Outlive-Longevity-Peter-Attia-MD/dp/0593236599/r

• Ender’s Game: https://www.amazon.com/Enders-Game-Ender-Quintet-1/dp/1250773024

• Tressie McMillan Cotton’s website: https://tressiemc.com/

• Anne Helen Petersen’s newsletter: https://substack.com/@annehelen

• Can You Really Measure Individual Developer Productivity? - Ask the EM: https://blog.pragmaticengineer.com/can-you-measure-developer-productivity/

• Measuring Developer Productivity: Real-World Examples: https://newsletter.pragmaticengineer.com/p/measuring-developer-productivity-bae

• Measuring developer productivity? A response to McKinsey: https://newsletter.pragmaticengineer.com/p/measuring-developer-productivity

• Measuring developer productivity? A response to McKinsey, Part 2: https://newsletter.pragmaticengineer.com/p/measuring-developer-productivity-part-2

• The Full Circle on Developer Productivity with Steve Yegge: https://newsletter.pragmaticengineer.com/p/steve-yegge

• Measuring Software Engineering Productivity: https://newsletter.pragmaticengineer.com/p/engineering-productivity

• Platform Teams and Developer Productivity with Adam Rogal, Dir. Developer Platform at DoorDash: https://newsletter.pragmaticengineer.com/p/platform-teams-with-adam-rogal

—

Production and marketing by Pen Name. For inquiries about sponsoring the podcast, email podcast@pragmaticengineer.com.