Software Engineers Leading Projects: Part Two
Practical approaches for engineers to run projects from the kickoff to wrapping them up.
👋 Hi, this is Gergely with a 🔒 subscriber-only issue 🔒 of the Pragmatic Engineer Newsletter. In every issue, I cover challenges at Big Tech and startups through the lens of engineering managers and senior engineers. Subscribe to get issues like this in your inbox, every week:
Q: What are practical approaches to run engineering projects within Big Tech or high-growth startups?
This is a question I first had back in 2016. When I joined Uber, we shipped almost all work via short, 2-12 week projects completed by small teams, led by engineers or engineering managers. In order to get better at project management, we hired an external training company to provide project management training.
This training was a disaster. It was completely disconnected from our high-paced work. The contents were so out of sync for what we needed that we canceled it midway.
This issue covers the ‘practical’ aspects of leading projects I wish that training could have delivered, back then. All of this is specific to software engineering teams in companies that have decent engineering cultures. It’s all directly applicable to Big Tech, and many of the suggestions apply to getting engineering work done within most tech companies.
Kickoffs and milestones: why they are important, and what good kickoffs look like.
Stakeholders how to work with them, especially the problematic ones.
The “law of software project physics”: how people, scope and timeline are connected.
Day-to-day project management: practices worth using.
Risks and dependencies: spotting and mitigating them.
Wrapping up projects properly.
Leading several projects: managing workload once you’re an accomplished project leader.
This issue is the second and final part of a mini-series about software engineers leading projects. In part one, I covered why and when engineers at all levels can and should lead projects, with advice on how to prepare for driving these.
1. Project kickoffs and milestones
I’ve observed that the biggest reason projects fail is that they started with misaligned expectations. However, this misalignment only became apparent near the end of the project, when the engineering team thought they were done.
Kickoffs help make sure there are no misunderstandings at the start, those ones that cause large chunks of work to be thrown out, or be redone entirely. I think about kickoffs as events where each question is clarified with the relevant stakeholder group:
The best way to de-risk any project is to do a kickoff: an event where all the project stakeholders confirm they understand the goals and approach of the project, and agree with the high-level plans.
Preparing for a kickoff varies depending on the complexity of the project and the number of stakeholders. Having a written document is essential, in my view – one that can be circulated beforehand, to which people can add comments ahead of time. This approach is a must in any organization where teams work remote or partially remote.
A Product Requirements Document (PRD) is a common format for describing the “why”, and “what” of a project. While engineering should be involved to some extent, at this stage it’s not about the engineering details, but about aligning all product, engineering, design, data science, and business stakeholders. See examples of PRDs at various tech companies.
1. The project kickoff to align everyone with the “why” and “what” is an event where all attendees go through the plans and can ask questions. A successful kickoff is one where when the project lead asks, “are there any questions on what the project is, why we’re doing it, or how we are approaching it?”
The ideal answer is that everyone is clear, based on the information available. In practice, I’ve seen a room explode with questions at this point: proving the value of this session. Much better to settle questions before the team starts building versus afterwards, when changes might see previous efforts go to waste.
I recommend the kickoff to be a meeting, with ten minutes dedicated to reading the proposal document, or the project lead walking attendees through it.
As counter-intuitive as it sounds, I recommend inviting all stakeholders with a relevant say to join the kickoff. This means inviting everyone from the business – including customer support – who is a stakeholder. While having more people in a meeting makes it a more expensive event time-wise, it’s an opportunity to capture early those misunderstandings which could lead to weeks of engineering work being tossed out.
2. The engineering kickoff to align engineers on the “how” follows the project kickoff, and involves the engineering team and relevant engineering stakeholders. Once the business goals and high-level approaches are clear, engineers dive into planning exactly how this approach will work.
Approaches vary: some teams do whiteboarding – or virtual whiteboarding – sessions. Most of Big Tech and many startups kick off with engineering planning documents that they circulate, like RFCs (request for comment), ERDs (engineering requirement documents) or ADRs (architecture decision records.)
I am a big fan of writing down the agreed engineering approach, as it has so many benefits. It forces engineers to clearly describe the proposed approach, thereby limiting misunderstandings. The plan can now be circulated for comments, broadening its reach. And the planning document will be available later for those working on the project to aid their onboarding and understanding of decisions made.
3. Establishing milestones aligns the engineering team with the “when.” Several engineering teams combine their engineering planning with the project’s milestones and estimations. I prefer to do these two separately.
The reason is that if you set engineering’s dates and milestones at the planning stage, there’s a high chance the team will immediately add shortcuts and make decisions that result in tech debt. By doing the engineering planning separately, the team should have an ideal plan that makes the best decisions possible, not just for the project, but for the long-term health of the systems and services.
With an engineering plan in place, now is the time to get a sense of what are shippable milestones. The more granular these milestones, the better. This is because all estimates will be off by some degree, and the only real way to know by how much is to hit a milestone. Milestones have value in themselves by bringing the final goal a step closer.
Once the milestones are agreed on, the team does a breakdown of the work this milestone needs, and a rough estimation. Some teams do T-shirt sizes, while others prefer engineering days. In the end, whichever method you use, in any date-driven company you’ll have to give a rough date estimate by when this milestone should be ready.
I prefer to have milestones that are small enough to achieve in no more than a few weeks. If the milestones end up involving lengthy chunks of work, I’d challenge the team to set intermediary milestones that capture a part of the work.
As a project lead, ensure that estimations are not binding on the team. Software projects are full of unknowns and risks, and the most anyone can reasonably ask of a team is for a rough idea of how things would look, based on the latest information. As the project progresses, risks and challenges will pop up which can add more complexity and change the estimates.
The business will, of course, want an idea of the final shipping date. There are several ways you can go about sharing this estimated deadline. Many leads add padding or inflate the estimates, then communicate this externally.
I have always preferred to add some padding, but also to educate the business that there’s no such thing as a fixed date. Commit to the business that they will get regular updates, then as the project passes its milestones, you can give dates with increasing confidence.
In this sense, software engineering and construction work are not too dissimilar. Anyone who has contracted house building or renovation will be aware that dates and budgets are estimates at best and it almost always ends up being more than estimated. Software projects are no different.
2. Stakeholders and managing them
Stakeholders are people who have some level of interest in the outcome of the project. For most engineering projects, stakeholders typically fall into a few groups:
Customers: those using the output of the project. For engineering teams building B2C (business-to-customer) products, these will be the end users. For B2B (business-to-business) use cases, they will be the business customers for the product. And for internal-facing projects – frequently built by platform teams – these will be internal teams.
Business stakeholders: non-tech groups within the company who have ‘skin’ in the project. These could include Legal, Marketing, Customer Support, Finance, Operations and many others. My observation is that without a good kickoff - that we’ll discuss later - engineering is often unaware of all business stakeholders. But if some business stakeholders are out of the loop it can cause delays to the project, later.
External stakeholders: teams at other companies that have a vested interest in the project. These are most frequently the vendors the company uses, or corporate partners in a project.
Product stakeholders: product managers, design, data science or other groups in tech which tend to work directly with product managers. Most of these stakeholders work both with the business stakeholders and the engineering team.
Engineering stakeholders: other internal engineering teams which are upstream or downstream dependencies for this project.
Categorizing stakeholders by their type of dependency is an exercise I suggest doing, based on their relationship with the project:
Upstream dependencies are teams you depend on. They need to do a specific type of work, or enable your team, for the project to get done.
Downstream dependencies are teams that depend on you. You’ll need to get work done to allow them to complete their part of the project.
Strategic stakeholders are people or teams you want to keep in the loop. They are often people or teams who can help unblock downstream dependencies.
Stakeholder management is a fancy way of saying you’re making sure everyone who needs to know what’s happening with your team, does know. Poor stakeholder management means you’ve left key people and teams out of the communications loop, which can eventually delay the project, or at the very least bring frustrated messages and many complaints.
Proactive, written communication on how the project is progressing is an underrated approach for managing stakeholders well. I suggest sending regular emails on the progress of the project, risks and how they are mitigated, and how the timeline has changed, or how confidence in the timeline is changing.
See the full example email here.
Problematic stakeholders are people you will no doubt run into as a project lead. What does “problematic” mean? It depends whether they are downstream, upstream, or a strategic stakeholder:
Upstream problematic stakeholders are those whom you are having issues with, trouble collaborating with, or in trusting. They might be teams or people whom you perceive as unreliable.
Downstream problematic stakeholders are often ones who keep bugging you about when you’ll be ready, and try to push you and your team.
Strategic problematic stakeholders are those who keep asking for updates or give you a hard time about any new challenges that you surface.
In all three cases, the biggest issue tends to be communication and trust with these stakeholders. Here are approaches that typically make things better, regardless of which type they are.
Talk face-to-face instead of doing it over chat or email. Get on a video call or grab them in-person and lay out the issues you’re observing, and talk through what you can both change to make things better. For example, if we’re talking about an upstream stakeholder who keeps nagging you on whether your team can ship a milestone sooner, walk them through where you are and what you need to do, give the feedback that this approach is not helpful, and come to an agreement on how you can both work better together.
Offer transparency and educate them. For upstream stakeholders and strategic stakeholders – who are both typically impatient – explain what needs to be done and why, and where you are right now. I find that offering regular, honest updates on your progress calms nerves, which is another reason I’m a fan of regular update emails. If you’re writing these, there’s a good chance that simply adding some of these problematic stakeholders to the mailing list resolves the problem.
Ask for support from your management chain or project leadership chain if you’re not getting anywhere with the above approaches. There might be cases where neither talking directly, nor offering more transparency helps. Pull in help from above in this case.
3. The “law of software project physics”
While the software we build is virtual, there is a “law of software physics” I’ve found to be strikingly accurate. Here it is:
If the above triangle looks familiar, it’s probably because of the resemblance of the project management triangle which is a commonly used model to describe the conntection between cost, time, scope and quality. For teams working at Big Tech and high-growth startups, I’ve found quality to be not up for negotiation, and “people” representing the tradeoff of working time better.
The people working on the project, the project timeline and the scope of the project are all connected. If one of these changes, at least one other needs to change in turn.
If the scope of the project increases, then either the timeline also needs to increase, or more people need to work on the project. The first option – the timeline increasing in line with the scope – is straightforward, although the business often doesn’t want to accept it. The second consequence – where more people work on the project – also holds firm in all my observations.
People working extra hours and overtime is the most obvious way to increase this people component. In this case, the “new” people are the same ones as before, but now they’re working more time. They have full context and don’t need any onboarding. But make no mistake: you now have more people hours being invested in the project than you planned.
Adding more people to a project without slowing it down is challenging, but is possible in rare cases. Whenever I state this view, I regularly get quoted Brooks's law from the 1975 book, The Mythical Man-Month. This “law” is a quote coined by author Fred Brooks:
“Adding manpower to a late software project makes it later.”
I would caution to not treat this statement as a universal truth. It is not true for projects where onboarding takes little time, the work is broken down and can be parallelized, when people with domain understanding are pulled in and where you add relatively few people, relative to team size.
Pulling in engineers familiar with the codebase to pick up standalone work, can speed up projects. I’ve not only observed this happen, but also succeeded in delivering projects with increased scope on the original timeline, by adding new engineers.
However, when adding more people there are always hard tradeoffs on onboarding time and on communications overhead. Onboarding takes time, and the process can take focus off the team. The easier the onboarding – e.g. by using an internal open source model – and the more domain knowledge the new joiners have, the less expensive this onboarding is. The lower the communication overhead is – thanks to clear documentation, a clean codebase and asynchronous processes – the less this overhead will be felt.
The reality is that for most projects with expanding scope, pulling in extra engineers is more a theoretical option than a practical choice. Those engineers need to come from some other team or project and it is rare to have idle engineers elsewhere. And for many projects, the work might not be sufficiently parallelizable for new joiners to simply pick up independent parts of it.
You should still be conscious of the ways in which getting more people may be an option: how overtime equals more person-power, and how it is realistic to ask for more engineers with domain knowledge, but only after you can confirm they would speed up the project.
If the deadline of the project is brought forward, something also needs to give. The most obvious likelihood is that the scope gets cut. What is less obvious is that the “people” dimension almost always increases. This is almost always done via working overtime, as mentioned above.
Adding more engineers to a project with a shorter timeline is still an option, though a risky one. If onboarding is short and those engineers can work on independent parts of the project, the risk is lower. However, unless the deadline is still months away, the onboarding time itself typically makes adding more people unpractical.
Changing the scope or the timeline is the easiest change to make. Pushing the team to do more hours is easy, but can bite you, if overtime becomes a regular thing and people burn out. Adding more people is several times more challenging and risky to pull off for any project than other alternatives are.