Software Engineers Leading Projects: Part One
Why and when engineers at all levels could and should lead projects, and advice to prepare for leading these.
Q: My tech lead is stretched, leading several projects. What suggestions would you have to ease the load?
Let me flip the question: should the same person - like the tech lead or engineering manager - lead all projects on an engineering team?
At both high-growth startups and in Big Tech it’s common for engineers at all levels to lead engineering projects. As an engineer, building up the skills to do so is important if you want to grow into senior or staff roles. And as a manager, helping people build these skills will help you scale as a leader.
In this issue we cover:
What is a ‘project’? What do “projects” mean in the context of Big Tech and other tech companies? And what about Agile?
Why, when and who should lead projects? What are setups in which engineers taking the lead makes sense, and what are cases when it does not?
Advice for engineers to prepare for leading projects.
Advice for engineering managers to empower engineers to act as project leads.
Companies where engineers at all levels lead projects and how these project lead positions are referred to.
A subscriber-only guidance document for engineering project leads, used as inspiration by hundreds of teams.
In Part 2, we further cover:Kickoffs and milestones: why they are important, and what good kickoffs look like.
Stakeholders how to work with them, especially the problematic ones.
The “law of software project physics”: how people, scope and timeline are connected.
Day-to-day project management: practices worth using.
Risks and dependencies: spotting and mitigating them.
Wrapping up projects properly.
Leading several projects: managing workload once you’re an accomplished project leader.
This is part one of a two-part series about engineers leading projects. See part two for the more practical parts: day-to-day activities related to leading projects, managing risk, and wrapping up projects.
1. What is a ‘Project’?
If you work at a tech company, your team almost certainly owns a product, a platform, service, or other functionality. Your team will most certainly have a mission, and in better teams you’ll have some measurable health and business indicators like KPIs (Key Performance Indicators) that both the business and your team care about.
What will your team be working on in the upcoming period?
At most of Big Tech and at high-growth startups, the organization tends to run planning roughly every six months. I wrote in more detail on how this process works in the issue Preparing for the annual or biannual planning.
The outcome of planning is a roadmap for your team. This roadmap will include product and engineering initiatives, each initiative having some kind of impact estimate and a priority order.
My definition of a “project” is this: a non-trivial piece of work which delivers measurable impact.
1. Non-trivial piece of work. If the work can be done in a few minutes, is this a project? In practice, most people would call this a task. It doesn’t require much coordination with others. While it could be called a project, it would be silly to treat it as such, as projects carry an overhead that comes with coordinating something that is non-trivial and might involve several other teams and stakeholders.
2. Measurable impact. If you treat every piece of work that is non-trivial and requires coordination as a project, you’ll soon find yourself with a lot of work items. These will range from new feature requests, through complex bugs to be fixed, all the way to reducing tech debt.
There are benefits in declining to treat anything as a project before it’s clear what the impact of this work will be.
Being clear on why the work is important, or if it’s important at all. In teams where projects are added to the backlog with no estimate of impact, it is work that’s close to the hearts of team members which typically gets done. This could be useful work, or it could be busywork: there’s no way to differentiate. By having a rule that all projects need to have an estimated impact, those proposing the work need to think about the bigger picture.
Prioritization. If one project is estimated to generate $1M/year in new revenue, while the other could save $5,000/year in costs, their relative impacts can be compared.
A more democratic way to propose projects. Most tech companies want to involve engineers more in innovation. The best way to do it is to transparently share the current projects and their estimated impacts, and invite engineers to add suggestions of further potential impacts. This approach will help some engineers become more product-minded.
What about Agile?
In this issue I’m covering how projects often get done in Big Tech and high-growth startups. We’re talking about projects for which engineers often work across several teams and dependencies – like modifying multiple services to build their functionality – and at places where systems are complex enough that the absence of coordination could lead to the wheel being reinvented, or teams slowing themselves down unnecessarily.
In these scenarios, some level of upfront planning is done because it’s faster this way. However, as soon as I mention teams doing planning, and only then starting coding afterwards, people often suggest it must be a waterfall process with a BDUF (Big Design Up Front).
Spending time planning up front does not make this Waterfall: in fact, it often means faster overall work. Waterfall is a process where one team plans every detail of a project for six months, then hands this over to another team to build for three years, and then the project would fall in year four. These days, there are very few companies using waterfall processes, whereas twenty years ago this approach was more common. Waterfall has largely ‘dried up’ as a practice in the industry.
On the other hand, coming up with a problem, validating it with customers – e.g. by talking with them – then drafting a plan of how to build this and which systems need changing, and then shipping it in another few weeks, is pretty nimble. If we’re talking about a team with CI/CD systems in place, for example one that’s shipping daily to production, the changes to a prototype used by the team and early customers, then this checks all of the Agile Manifesto boxes.
However, the goal of building software is never to follow any manifesto. The goal of an engineering team is to achieve its business goals as efficiently as possible, while keeping the team and organization happy and healthy.
A project is the most common building block many organizations use to coordinate efforts that move the needle of a specific business goal. These projects can be as short as a week or two, but regularly take months from start to finish. Long-term efforts might span years. However, at tech companies with healthy engineering cultures, engineers ship business value multiple times per day, and ship projects that are considered complete at least every few months.
There will be teams which decide not to use project-based approaches and opt for following processes like Scrum – with sprints that last one or more weeks – or Kanban; taking off the next ticket from the backlog. For simple and well-defined pieces of work, the guide below is likely not needed.
This guide applies to more complex “work units” in any framework, agile or not. Any ticket that involves coordinating more people and working across several dependencies is a project that can benefit from the following approaches. This is especially true if you work in an environment where the business uses dates to coordinate and communicate with different functions. And most of Big Tech and also high-growth startups are date-driven.
2. Why, when and who should lead projects?
Benefits of Engineers Leading
There’s much that engineers, their team, and the company gains when engineers take on leading smaller or larger projects. Here are a few:
Growing skills that are hard to strengthen otherwise. Communicating with stakeholders, influencing without authority, empowering other team members, delegating down/sideways/up and mentoring: these are some of the skills that engineers leading projects will get to exercise. While many of these skills can be strengthened in other ways, leading a project is often the best bang-to-buck.
A more resilient team. Take for example a team in which the only person driving projects is the Tech Lead. What happens if this person quits or is out-of-office for a longer period of time? Now compare this with a team in which most team members have led projects and have no problem standing in to take over leading a project. Rotating leadership responsibilities builds a far more resilient team.
Empathy for “the other side” of the IC track. As an IC (individual contributor) software engineer, I had little understanding and empathy of the work tech leads and managers were doing until I led my first project. An engineer stepping up to lead will gain lots of empathy about what the activity really entails. They’ll be better partners to other project leads and to managers, gaining empathy that is hard to get without first-hand experience.
Investing in future managers and staff+ engineers. Both engineering managers and staff-and-above engineers need to have direct experience of leading. By empowering engineers to take on this responsibility earlier in their career, you’ll help many of them gain this skillset. Not all engineers will want to go down the staff or EM paths and it’s just as good that these engineers experience early on what leading is like. But for those who do have an interest in these career paths. you’re helping them progress faster.
Reasons Why Engineers Often Don’t Lead
If there are so many benefits, why do we see many organizations shy away from supporting more engineers to lead projects? There are several reasons.
Support from the manager. For any engineer to successfully lead a project, their manager needs to support them. This often means creating opportunities, encouraging people to take it on, and mentoring and coaching them on how to do it.
Managers who don’t want to mentor. Leading engineering projects is a skill that gets better with practice and feedback. Managers who are unwilling or unable to mentor new leads will often avoid delegating leadership responsibilities to engineers. Even if they do delegate them, without support engineers are more likely to fail, powering the negative cycle of the manager being convinced it was a bad idea to start with.
Fear of delays. A common heaistation managers have in delegating leadership is that the outcome will make the team and the manager look bad. In my view, this comes to back to whether the manager supports professional growth, wants to mentor engineers on their team, and if they are willing to have the back of the lead and manage expectations while first-time leads get more experience.
Engineers not wanting to lead. Some engineers will state they have zero interest in leading. The most frequent reasons I’ve heard are not having additional bandwidth and not having interest beyond the code. “I’m not paid enough to do more” is a cynical but not uncommon response. If someone genuinely does not want to take on more responsibility for whatever reason, it’s important that they should not be forced. Just as important though is outlining career paths and opportunities that can open for them, if they do want to take on more challenges, and also ensuring the underlying reason for declining the chance is not an unsupportive manager or team.
Expectations of ‘leading’ are not clear up front. Few people will want to take on a leadership role that is too vague. To succeed as a project lead, the concept of “leading” needs to be clarified by the manager. Managers who don’t do this will often be disappointed with how the engineer leads, even though it was they who failed to set up the person for success.
“It’s never been done like this.” Especially at traditional organizations, the notion of leading is often tied to years of experience, a fancy title and higher salary. In these organizations someone without these could be seen as a threat to a model that has worked for so long. All I can say is this is an attitude that rarely fosters growth or success.
Whatever the reason why engineers are not empowered to lead, I suggest finding the root cause and seeing if it’s still valid. When done well, there’s much to gain and little to lose by empowering engineers to take more ownership of the project they work on.
When should an engineer lead a project?
When is it a good opportunity to offer the lead of a project to an engineer? The answer to this question will depend on the complexity and scope of the project, and the number of teams involved. Here’s a simple framework I like to apply.
Coordinating large projects can be a full-time job. By large projects I mean those that encompass multiple teams, often across several locations and timezones, that call for dozens of engineers and a wide range of stakeholders. For example, the project of implementing GDPR compliance at a company with 8 different products and 500 engineers will keep more than one person busy just staying on top of the organizational details.
At Big Tech and larger tech companies, program managers or technical program managers (TPM) tend to own the running of projects with wide breadth. Product managers and engineering managers or leads sometimes step in to take on some of these projects, as long as they have the bandwidth to lead it and get the rest of their work done.
Smaller projects could be good opportunities for software engineers to lead. A Project with a small enough scope is when a team member could step up to lead the effort. By smaller projects I mean when the work is to be done mostly within the team, with few dependencies outside the team. A rule of thumb I use is that if managing a project will not take more than 20% of an engineer’s time, then taking on this responsibility could be a reasonable challenge for most engineers.
In organizations with hierarchical cultures, it will be the leads – engineering managers, tech leads – who take on leading all non-trivial projects. At these places, any project where a few engineers work together is headed up by the same person. The way engineers get to lead these efforts is when they are promoted to the lead or manager position. The downside of this approach is that leads might be stretched and otherwise capable engineers are not given opportunities to step up and lead.
I took the opposite approach when I became a manager at Uber. Even though I had the experience to lead most projects that my team was working on and we had a few tech leads to whom I could have delegated the other projects, it felt like a waste to follow this approach. Why put myself on the critical path of all projects? Would it not be better for everyone if I offered the opportunity for others to lead, support them and help them build skills that made them better project leads?
Engineers leading projects while not being tech leads was a decision that paid dividends for years. I set up a system where at first, the most experienced tech leads would lead projects. I supported them, gave feedback and encouraged them to delegate some leadership activities to less experienced engineers. Then on the next project, I rotated another engineer to take the lead role, asking an experienced project lead to support them.
I wrote in detail about this approach, its challenges and the results, in the article An engineering team where everyone is a leader:
“Initially, my team was eight engineers, from juniors to more senior members. Two years later, I had a team of triple the size, where most people had led a significant project, with mentorship/coaching support from a more experienced leader. Professional growth followed, with all the original eight people promoted to the next level, alongside a few others. Attrition was low, and people seemed to like being on the team.”
Engineers leading projects is increasingly common in companies with high engineering autonomy, like Big Tech and many high-growth startups. This article closes with several examples of companies that follow this model.
Who should lead the project?
As a manager, you’ll need to “match” projects with engineers, sooner or later. The idea of a self-organizing team regularly comes up, where the team self-selects its leaders.
While self-organizing might work, I would caution not to rely on this method. If you do, you’ll likely find that factors like people’s career aspirations and their experience level versus the complexity of the project, do not match up. Even worse, in a self-organizing setup there will be conflicts when two people want to lead the same project. As a manager, you’ll want to get ahead of this, balancing the interests of the team first and the interest of team members second.
Things worth considering before deciding which engineer should lead which project: