What Changed in 50 Years of Computing: Part 2
How has the classic book on software engineering, ‘The Mythical Man-Month,’ aged with time, and how have architecture approaches and tech org structures changed in half a century?
‘The Mythical Man-Month’ by Frederick P. Brooks, Jr. is a classic in the software industry. First published in 1975, it’s almost 50 years old. Brooks was the project manager of one of the most complex software projects in the world at the time: the IBM System/360 operating system. He published this book based on his personal experience of spending several years building it, and leading several hundred programmers.
I’ve been making my way through this book from the dawn of software, to see which predictions the book gets right or wrong, and what’s different about engineering today - and which things just never change. In Part 1 of this series, we covered chapters 1-3. In this article, we cover chapters 4-7:
Evolution of architecture approaches: The outdated separation of “architect” and implementer, the dated “technical manual” concept, software design and the “second-system effect,” & telephone logs.
Architecture approaches: what’s the same. A well-architected system balances simplicity and functionality, and it’s still hard to do. Processes are still needed for proposing and discussing architecture changes for mature products and larger teams.
Communication challenges on large projects: then and now. Surprisingly, communication best practices have changed little in 50 years; mixing informal, formal, and written forms for efficient projects.
Ideal structure of tech orgs. The tree structure was the most favored for tech organizations in the 1970s, and it still is the most popular choice today. Are simplicity and straightforwardness the reason why?
1. Evolution of architecture approaches
When it comes to how we design software, a lot has changed in 50 years:
The outdated separation of “architect” and “implementer”
How do you build a good software system? In Mythical Man-Month, Brooks gives two viable options:
Have a small “surgical team,” as discussed in Part 1.
Separate architecture from implementation in large projects. Not just by phase, but with different people doing different things.
Here’s how Brooks describes the role of the architect in Chapter 4: “Aristocracy, democracy and system design.”
“The separation of architectural effort from implementation is a very powerful way of getting conceptual integrity on very large projects. (...)
The architect of a system, like the architect of a building, is the user’s agent. It is [their] job to bring professional and technical knowledge to bear in the unalloyed interests of the user, as opposed to the interests of the salesman, the fabricator, etc.
Architecture must be carefully distinguished from implementation.”
“Implementers” work under the guidance of the architect, and write code which follows their design direction.
Brooks shares a real-world example of how this setup played out when building the OS/360 system. They had a team of 10 architects and a team of 150 “implementers” – so, a 1:15 ratio. The architecture team asked for 10 months to design the system and hand it over to the implementer. However, the “implementer” team didn’t want to sit idle with 150 people for 10 months, so they proposed to be part of the planning process. The architecture manager protested, but Brooks – as the decision maker – agreed not to leave the implementing team idle. The result was a disaster, as per Brooks:
“The architecture manager responded that if I gave the control program team [the ‘implementer team’] the responsibility [of specifying interfaces] the result would not in fact be on time, but would also be three months late, and of much lower quality. He was right on both counts. Moreover, the lack of conceptual integrity made the system far more costly to build and change, and I would estimate that it added a year to debugging time.”
Building a system with “conceptual integrity” is much faster and easier than in the 1970s. Brooks recognized that a large system built by hundreds of developers needs strict architecture guidelines and well-defined interfaces for it to work. It made more sense to have 10 architects come up with these interfaces for close to a year, and only then for the implementers to work on them.
So why does this approach feel dated these days? Here are some potential reasons:
1. Software teams are much smaller. For a new project or system, it’s rare to hire 100+ developers to build it. Instead, it’s usually a small, “core” team of 5-10 devs who build the core of the system, and other teams build features on top. This is even easier when teams use microservices, meaning each service can be developed independently and interact with other services via public APIs.
2. Defining and updating APIs is easier these days. It’s been more than 20 years since Amazon pioneered the principle of every engineering team exposing APIs to communicate with other teams, in the “Jeff Bezos API mandate” of 2002. When building online services or apps, it’s straightforward enough to require engineering teams to build and own their own services, and to expose APIs which other teams can use.
3. Prototyping is more rapid and commonly used. When building a large system from scratch, starting with a prototype is common enough. This prototype is a fantastic way to flesh out how the system should work at a high-level. This prototype can then be used as a living “architectural blueprint,” or be thrown away after the team confirms viability for their high-level design.
4. Architects uninvolved in implementation leads to worse results. I’ve been part of complex projects with 100+ developers working on a new system, like Uber rewriting its mobile app with ~400 developers, or rewriting its payments system with ~200 developers.
In both cases, we followed this approach:
The “architecture” team prototyping the new architecture to flesh out the design approaches. This took 2-4 months. “Architects” were staff or senior-level engineers, and not called architects. For the Uber app rewrite project, this was most of the Mobile Platform team. For the Payments rewrite project, they were the group “in the room” during planning and built up a strong rapport.
Then, all other teams joined in to build/port their functionality. However, the “architects” were in the weeds and working hands on with the teams, doing design reviews, code reviews, coding, and adjusting the design as needed!
Rollout: the original team was deeply involved in rolling out systems, being oncall, and resolving outages. This was so important, as the system that should have 99.99% reliability kept having production issues during the first few months, with availability much lower than this. The “original architect” team being involved and oncall was very helpful; they saw the assumptions in their original designs, and could fix them much easier.
Engineers designing a system who have “skin in the game,” is a sure way to build better systems, and for people to become better software professionals by experiencing first-hand how an initial design misbehaves in the real world.
5. It’s common for the “architect” team to be a subset of the development team. Let’s take a complex feature that takes 10 teams collaborating, and around 50 developers working on it. Is it sensible for everyone to start work at the same time? Of course not!
There will be a subset of this team which comes up with a plan on how to proceed, who are the “architects,” “leads,” or “tech leads.” Once the plan looks good, the whole group gets to work on building. The important detail is that the architects are hands-on in developing with their teammates and tweak the design, as needed.
With Brook’s approach, this didn’t happen. It feels like much of why not comes down to it being a lot earlier in the age of programming; “implementers” needed more guidance than today, architects had to invent things that did not exist, and there wasn’t the luxury of powerful languages, frameworks, and open source solutions.
For large projects with 10+ engineering teams, it’s not uncommon to have Technical Program Managers (TPMs) running such projects. We cover more in the issue, What TPMs do and what software engineers can learn from them.
6. “Architect” is an increasingly unpopular title in startups and Big Tech. Devs who act as architects on teams increasingly have titles like “senior engineer,” “staff engineer,” “tech lead” and “principal engineer.” I don’t know of any Big Tech or scaleup with an official “architect” title. As WhatsApp engineering director Mark Tsimelzon shared:
“Architects” is pretty much a banned word in Meta.
7. But traditional companies still follow the “architect” vs “implementor” approach. A few years ago, I talked with the chief architect at a financial services company, who was leading the redesign of the architecture. He said it was year 3 of the redesign, and that architects made the plans and developers carried these out. The project was dragging on, due to the implementing team misunderstanding designs, and this architect was brainstorming approaches on how to improve things.
My sense was that the company was old (founded more than 100 years ago) and extremely risk-averse. It was more important that nothing was broken, and architects spent more time interfacing with the business about this. For the business, a slower pace was an acceptable tradeoff in order to provide stability for their very profitable work.