Code Freezes: Part 1
At the end of last year, code deployments were frozen at many tech companies. Why was this done, what are the benefits, and what are the tradeoffs of this approach?
If you’re a software engineer at a tech company, there’s a good chance the final two weeks of December were more peaceful than usual, not only because of colleagues going on holiday, but because of deployment freezes. These are often referred to as “code freezes,” short for “code deployment freeze.” In this article, I’ll use the term “code freezes” to mean both “code deployment freeze” and “deployment freeze.”
Today, we cover:
Big Tech and code freeze approaches. How do Meta, Amazon, Microsoft, Uber, Apple, Google and other larger tech companies go about deploy freezes during the holidays?
Code freezes at other companies. Details from Spotify, Twitter, N26, Podia, Bold.org, T-mobile and several startups.
Code freeze upsides. Fewer outages, people disconnecting and other reasons these freezes are helpful.
Downsides of code freezes. A rush to get changes in, risky refactorings, merge issues, and other problems the freeze introduces.
Companies which don’t do code freezes. A major e-commerce platform and a large SaaS provider don’t do code freezes. What did the teams there observe, as a result of this?
This is Part 1 of a three-part series.
In Part 2, we go into more depth, touching on:
Software product categories and code freezes. Web-based products, desktop apps, on-prem software, and embedded products tend to have very different deployment cycles. What are products where code freezes matter, and what are ones where they do not?
Mandating a code deployment freeze. Informal and more formal approaches to do so, and questions to ask before deciding on the strategy.
Code freeze trends across industries. What are trends within banking, fintech, e-commerce, wellness apps, and other groups?
Code freeze alternatives. Code chills, close slush, and business as usual.
“Wave and bake” instead of code freezes: Ocado. The grocery retailer platform has no code freeze and uses a unique approach with its isolated, single-tenant environments.
Do you need to implement a code freeze? Things that play into this decision, from the maturity of deployment tooling to wanting to create space for staff to recharge for a few weeks.
In Part 3, we cover:
Dates worth keeping in mind. Apple’s store slowdown, and a 3-day workweek at the end of the year.
Code freezes by funding lifecycle. The later stage a company is, the more likely it puts a freeze in place. Bootstrapped companies seem the least likely to do anything different during the holiday season.
Company size. The smaller the business, the less likely there’s a formal code freeze.
By industry. An analysis of 185 data points on deployment freeze approaches at tech companies, contributed by readers.
Advice for those doing a code freeze. Avoid the “thundering herd” in January!
Advice for those not doing a code freeze. Ensure your engineering team gets a well-earned rest during the holiday.
Interesting code freeze approaches. Inspiration from Monzo, Outreach, LinkedIn, Block/Square, Stick Fix, Atlassian and Klarna.
The survey data. Anonymized responses from 185 data points.
Let’s jump in!
1. Big Tech and code freeze approaches
It’s common for most of Big Tech to have two week-long code freezes during a year:
Thanksgiving week. The week of Thanksgiving is when Black Friday takes place. For most e-commerce businesses, this is when the single biggest daily revenues of the year are generated. Online retail companies or businesses whose customers are in this sector want to ensure maximum reliability of systems during this period. Additionally, many tech workers take at least a few days off during this week, and several take the whole week. A code freeze helps both with reliability and allows for more tech workers to take a vacation.
Christmas to New Year. In the US and Europe, it’s common for the majority of people working in tech to take off one or two weeks. It’s understood there will be fewer colleagues available to fix any breakages, so most companies ensure no such breakages happen by putting a code freeze in place. A code freeze also helps those on holiday to properly unplug, confident that no potentially troublesome new changes are making their way to production.
Meta puts a code freeze in place from November, which looks roughly like this:
The reason for the stricter cadence at the end of the year is to avoid issues ahead of and during the holiday periods of Black Friday, Christmas and New Year.
This practice operates for two reasons:
Minimize revenue lost due to outages, especially for advertising products.
Enables people to take time off to rest during Thanksgiving and at the end of the year.
Both the “yellow” and the “red” periods are enforced company-wide. “Red” periods are company-wide code freezes, with exceptions for some teams and scenarios. Teams working on internal tools and tests can usually push code changes, and outages – naturally – can also be mitigated by them. Teams working closely with revenue generating functions like Ads tend to be a lot stricter with code freezes, while smaller teams more distant from revenue-generating tools tend to have more leeway in pushing changes.
Amazon has a calendar of “Restricted” and “Blocked” days for deployments, formerly called “gray” and “black” days. These days are usually around Prime Day, Black Friday, Christmas, Amazon’s annual “Re-Invent” event and some other major calendar dates.
On Restricted days, all deployment pipelines have automatic deployment blockers in place. These blockers may be limited to specific regions, or only for AWS deployments.
Blocked days are usually announced at short notice during big, operational events such as an outage, or unplanned events when temporarily suspending deployment pipelines is the safest option. Deploying changes during Restricted days requires Director-level approval. Deploying during Blocked days needs VP-level approval.
On top of Restricted and Blocked days, teams define their own code deployment freezes. For most codebases, a freeze lasts for 1-2 weeks around the time of Black Friday, and 1-2 weeks around Christmas and the New Year.
Microsoft does not have a company-wide policy; organizations decide how they approach deployments.
For example, within the team responsible for the search engine Bing, there are two deployment freezes:
One week spanning Thanksgiving through to Cyber Monday
Two weeks spanning Christmas to New Year
Bing operates heavily based on experiments. During this time, experiments that were already running can continue, but new ones cannot begin.
Other organizations are more hands-off in their approach. For example, GitHub does not mandate code freezes centrally: it leaves these decisions up to teams. Most engineers are on vacation at the end of the year, and the pace is generally slower. A current software engineer at GitHub put it that GitHub has an effective code freeze, mainly thanks to barely anyone being around, without any of this having been mandated.
Uber historically put freezes in place during the week before Thanksgiving, and from mid-December until early January. This was because both Thanksgiving and New Year’s Eve saw peak usage. For example, during the early years, New Year’s Eve often saw a 5x traffic surge compared to other days.
The code freeze was meant to provide stability for the app during peak load periods. On top of the code freeze, Uber did capacity planning, load testing ahead of time, and had heavier than usual oncall during this time.
Apple puts code freezes in place both at the end of the year, around product launches, and for major events like WWDC – Apple’s flagship annual conference.
Unlike Meta, Google, or Amazon – where most teams ship straight to production – Apple likely has a larger percentage of its engineering teams ship on a cadence, such as those building the operating systems for Mac and iOS, those building the Safari browser, and those working closer to hardware. For products where releases are cut on a cadence, code deploy freezes are less of a change.
Where Apple differs greatly from Google or Meta is in how keenly it focuses on launches being ready for key dates, and planning most new hardware and software releases for those dates. The major recurring annual event is WWDC in June, and Apple also usually runs 2-3 other keynote events during a year. In 2021, there was a spring event in April, an iPhone event in September, and a Mac event in October.
Google tends to freeze deploys from mid-December until early January. Several teams I’ve talked with don’t deploy during this time.
Oracle has different policies for each organization. Many teams which ship core products tend to do quarterly releases, called “patches.” They usually enact a code freeze 6 weeks before the patch is released, when the patch enters thorough testing.
If this pace appears slow, then it most certainly is. However, Oracle’s core products are enterprise ones for which stability is more important than speedy iteration. This means slow releases and engineers I’ve talked with note that the pace does feel snailike, but it’s done to provide the reliability which business customers value much more.
Another organization within Oracle puts a code freeze in place at the end of the Q2 of their fiscal year: from mid-November until mid-December. The company’s fiscal year runs from 1 June – 31 May, and the code freeze ties in midway throughout this period. It’s curious this period ends in mid-December, as teams tend to be cautious when deploying at the end of the year as more colleagues are unavailable should something go wrong.
Twitter is a company worth noting, because of how drastically the engineering culture has changed. Before Elon Musk acquired the company, there were no company-wide freezes. Teams would decide at their own level if and when to pause deploys. For example, the Recommendations team would do a deploy freeze for the festive period, and put a holiday oncall in place.
But this year, Twitter continued to make large changes during the holiday period, including Musk unplugging a server rack on Christmas eve, and Twitter suffering a hours-long outage four days later, on 28 December. There’s a fair chance these two events were connected.
2. Code deploy freezes at other companies
Spotify implements no code freezes during the year, including at year-end. It does, however, request teams and engineers be extra careful, and to delay risky changes if lots of people are out-of-office. Otherwise, the end of the year is not much different from any other time.
Auth0 – recently acquired by Okta – not only does code freezes during Thanksgiving and at the end of the year, it also has a so-called Wellness Week in place for the tech team, when nobody is expected to work outside of the oncall for existing systems, in the event of an outage. It’s an open invitation to take off the whole week, pretty safe in the knowledge nothing important will be missed. Teams still need to staff oncall during this time, but oncall is expected to be quiet with no code changes making their way to production.
N26: freeze at the end of the year. N26 is a neobank in Europe and a competitor of Revolut. The company employs more than 1,500 people and was valued at $9B during its most recent fundraise in 2021. N26 has a production deployment freeze in place during the end of the year, with exceptions for urgent fixes or urgent deployments.
Podia ‘closes up shop’ for the final week of the year. Podia is a ~30 person SaaS platform, enabling creators to sell digital goods. The CTO, Jamie Lawrence, shared that while they don’t do code freezes, the whole company logs off from work between 24 December and 2 January:
“Between 24 December and 2 January is the only time we have a ‘code freeze’ in place. I don’t believe in code freezes and don’t use them at other times of the year (e.g. Black Friday) but since everyone is off work, no one should be making changes and there is reduced availability should anything go wrong.
Why do a code freeze if everyone is off work? Because sometimes people get bored and tinker around on work projects or fix bugs that come in over Christmas. This way just makes it clearer that we don’t expect any code changes. Emergency fixes are still possible but need to be reviewed & approved by me.”
Talking with Jamie near the end of 2022, he was clear that the goal of the end-of-year “code freeze” is to give people time off, not because they’re worried about making breaking changes. He shared that some of the riskiest refactors made it into production at the end of the year:
“I expect developers to exercise good judgment and not deploy just before they leave for vacation, but this applies throughout the year – you should be available to support the code you deploy.
In fact, we’re finishing the year with a two week tech debt sprint which is arguably riskier work than typical feature development. We removed Webpack, renamed database models, cached some API responses, etc. They were all shipped but we also need to exercise our judgment.
As an example of considering what it is we’re deploying: in the last week we also made all the changes required to move from Ruby 2.7 to 3.1, but won’t deploy to production until January."
Bold.org: code freeze at end of year. Bold.org is a company dedicated to fighting student debt. They are reasonably small with around 30 employees, profitable, and have a holiday code freeze in place. As principal engineer Orestis Ioannou shares:
“We do a code freeze between 23 December - 2 January. We encourage people to just take time off and rest and we freeze code so that nobody is forced to work. Take the time you need, but if you want to work then we have some tech debt tickets to work on. Still no need to release when most of the company is out.
This code freeze encourages people to take time off and to recharge. Nobody should be working through the holidays, unless they really want to.”
A summer and a winter freeze at a Scandinavian company. A large Scandinavian retail insurance company has two code freezes of 3 weeks per year:
Summer freeze: 3 weeks, from mid-July
Winter freeze: 3 weeks, from mid-December
Each freezen is in place because of people’s holiday schedules. Employees have 5-6 weeks of vacation per year, and most take it during this time. A software engineer working at this company shared that although the freezes greatly reduce productivity, leadership accepts this drop. In return, there are few to no incidents during this time, and it’s a given that most people can take time off.
A T-mobile subsidiary: two months. A site reliability engineer at a T-Mobile subsidiary in a European country shares the reason for an unusually long freeze:
“Internet-facing systems like webshops have deployment freeze periods from 18 Nov to 9 Jan. This is because the webshops generate most of their annual revenue between Black Friday and Christmas.
In addition, the availability of TV/streaming services was critical due to the soccer World Cup, this year.
Internal systems have a shorter code freeze period, from 19 Dec to 2 Jan.”
Vendors: often based on what their commercial partners want. A staff systems development engineer at a Series E company which supplies financial services for mobile network operators, shared that code freezes are in place because partners don’t want any changes at the end of the year:
“In our case, code freezes are mainly imposed on us by our partners. Our partners are mobile network operators in a mega-region and they put code freezes in place from 12 December to 4 January. We end up following their cadence.
So, in our case, code freezes are more of a ‘necessary evil.’ In general, I like these code freezes, as long as they are not too rigid. For example, as long as they do not restrict updates that are far away from the critical path.”
A last-mile delivery startup: end-of-year freeze. A startup based in Japan builds a B2B application used in last-mile delivery. A mobile engineer shares why they have a code freeze in place:
“We have a full frontend (mobile and web) and partial backend code freeze. We still push stuff to our staging infra, but not to prod. Breaking changes would be catastrophic for our delivery partners during the end of the year, when they see higher than normal volumes.
The code freeze is integral to building a product consumers can trust, especially at the highest volume times in the year.”
A clean energy startup: end-of-year freeze. A software engineer working at a climate startup valued at more than $1B shared its approach:
“It's well known that many issues come from new features/code in general, so freezes definitely help quiet things down for holidays/major events.
For us, all non-critical deployments are prohibited. This starts from before Christmas Eve and lasts to just after New Year. Merges are OK, just no deployments. Only exceptions are for high-priority hotfixes.”
End-of-year freezes at lots of companies. In the Pragmatic Engineer survey, so far filled out by more than 100 software engineers, the majority of companies have a deployment freeze in place at the end of the year. No production deploys are done this time. I’ll share more findings from the survey in Part 2 of this series. Please take part by filling in the survey if you wish!
This does not mean development stops; engineers and teams continue to work on branches. But these branches are either not merged to main, or the main branch is not deployed to production. For companies which operate staging environments, changes are deployed to staging. Read more about how different companies ship to production in this article.
3. Upsides of code deploy freezes
Common upsides are these: