The Pulse #61: What’s going on at Google Cloud?
Google Cloud caused an outage at a scaleup, after lowering a quota; and months later the same outage repeated. Also: what were perf reviews like at Meta, and do two GCP zones share the same building?
👋 Hi, this is Gergely with a 🔒 subscriber-only issue 🔒 of the Pragmatic Engineer Newsletter. In every issue, I cover challenges at Big Tech and startups through the lens of engineering managers and senior engineers. Subscribe to get issues every week. 👇
The Pulse is a series covering insights, patterns, and trends within Big Tech and startups. Notice an interesting event or trend? Send me a message! I treat all such messages as anonymous.
Today's topics are:
Industry pulse. A new, experimental column, featuring a roundup of events relevant for software engineers, engineering managers and those working in tech.
The current market, as seen by engineering leaders at Big Tech. I talked to a couple of senior engineering managers and directors at Big Tech, and sensed disillusionment, coupled with few options to leave and do something more exciting than KTLO (keep the lights on). Exclusive.
Why is Google Cloud causing outages for customers by lowering quotas? Railway and Levels.fyi both got hit by the same type of outage on Google Cloud: their quotas were silently reduced, which throttled their services. Founders of each company turned to social media for help. I talked with both to discover their experiences of Google Cloud. Exclusive.
What were performance reviews like at Meta, this time? There’s reports that Meta’s performance reviews have been stricter than usual. I asked engineering managers there for their takes. Exclusive.
Are two Google Cloud zones hosted in the same building? Powering down a building in Paris, France, during a Google Cloud outage fully brought down one Google Cloud Platform (GCP) availability zone, and also another one, partially. I looked into whether the two zones are in the same building, and asked Google. Exclusive.
The Pulse sometimes delivers first hand, original reporting. I’m adding an ‘Exclusive’ label to news that features original reporting direct from my sources, as distinct from analysis, opinion, and reaction to events.
1. Industry pulse
This is an experimental column, where I summarize and comment on recent events relevant for software engineers, engineering managers, and tech workers.
Mass layoff in Google’s recruitment function
The tech giant confirmed it’s laying off hundreds of recruiters. Recruiters being let go confirms what we suspected: Google will hire far fewer people in 2023 and 2024, than in previous years.
I talked with recruiters – unfortunately – impacted and heard that the final number is more than 500 recruiters: which is likely to be 20-25%+ of the recruitment staff. A deep cut!
In June, CEO Sundar Pichai was clear about a hiring slowdown, saying: “We continue to slow our expense growth and pace of hiring and ensure our teams are aligned to our highest priorities [...] We are combining various engineering efforts across core infrastructure and cloud.”
Google slowing hiring also means revenue growth is expected to slow, which means online advertising market growth is slowing. Meta has done a similar scale layoff already, but across all functions – letting go 25% of staff between November 2022 and May 2023. Sadly, it seems logical that other larger companies in the ads business will follow a similar approach by shedding recruitment like Google has, or slowing hiring. (Link)
2021 was an incredibly hot job market for the industry, and so was 2022, until around the middle of the year. We previously covered that 2023 has seen a drop in software engineer jobs, globally (March,) The state of the tech market in 2023 (June,) and How hiring managers see the 2023 market (August.)
Flexport’s founder is back, and started by rescinding 75 offers
Flexport is a logistics company last valued at $8B in February 2022, when it raised $965M. Shortly after, in June 2022, the founder and CEO of the company, Ryan Petersen, stepped down and hired Dave Clark to run it. Clark had worked at Amazon for 23 years, and was the CEO of its worldwide consumer division, before joining Flexport.
Well, after just over a year, Petersen is back after Dave Clark abruptly resigned – well, was effectively fired – and several ex-Amazon executives whom Clark brought in, were also let go. One of the first moves by returning CEO Petersen was to rescind 75 already signed offers, writing: “We have had a hiring freeze for months. I have no idea why more than 75 people were signed to join.”
What is going on? When Ryan stepped down as CEO in 2022, Flexport was profitable. Now it is not. A lot of this is likely to do with how the cost of shipping containers has collapsed in the past year. The cost of shipping a container from China to the US West coast was:
~$15,000 in February 2022, when Flexport raised funding at a $8B valuation
~$9,000 when Dave Clark was named as CEO, and Flexport claimed to be profitable
~$2,500 today (source: Bloomberg)
Flexport was profitable thanks to high container shipping prices, and is likely losing money thanks to the price collapse. A drastic change in the global shipping market means a change in strategy is needed, which is why Petersen is back.
Flexport rescinding offers is a reminder that in a cutthroat market, employers won’t hesitate to rescind offers (the company later said it’s offering some stipend for rescinded offers.) Rescinding offers could well mean that more cuts are ahead at the company. These are clear signs that at Flexport, it’s back to wartime mode, after the good times. (Link)
AWS shuts down EC2 classic after 17 years
AWS is known for reliability when a service reaches general availability, which is a deliberate choice by the company. It allows customers to confidently build on top of services, knowing that even if they are to be retired one day, there will be years of largely hassle-free usage, until then.
EC2 classic (EC2 standing for Elastic Compute) has been the prime example of Amazon’s support for old services since it launched in 2006. The EC2 service is still around. However, since 2013, new AWS customers could no longer start EC2 classic instances, unless explicitly requesting it. The product replacing EC2 classic was Amazon’s virtual private cloud one. So, EC2 classic was “deprecated,” and customers have been heavily incentivised to not use it for 10 years. Yet, Amazon kept supporting it for a decade!
Two years ago, in 2021, AWS announced that it will retire ‘classic’ instances, and the last EC2 instance was shut down in August. As CTO Werner Vogels summarized in the blog post titled “Farewell EC2-Classic, it’s been swell:”
“Retiring services isn’t something we do at AWS. It’s quite rare.”
(Link)
Unity wants to charge per game install, game developers furious
In the recent article Building a simple game, we used Unity, one of the most popular game engines. At the time of publication, Unity’s terms were pretty clear: pay a pretty high fee per month, per developer, to build your game, which you can then publish. The price was $185/month/seat for Unity Pro, and $250/month/seat for Unity Enterprise. This made Unity one of the most expensive SaaS offerings within game development.
Unity is a publicly traded company, and it’s losing money. The business already had two layoffs in the past year. Now, it’s done three things at the same time yesterday, 13 September:
Announced a ‘Runtime Fee’ that will be charged per game install, on top of the existing fees, of up to $0.20 per install for game developers and developer studios crossing $200,000 in revenue per year
Removed a key part of their terms and conditions, which would have allowed customers to keep using older versions of Unity, without opting into the install tax. The deleted part included the section “you may elect to continue any current-year versions of the Unity software (…) The Updated Terms will then not apply to your use of those current-year versions unless and until you update to a subsequent year version”
Quietly deleted the GitHub repository that tracked its terms and service changes
Predictably, the gaming community is furious. Video game sales are down from a peak of 2020-2022, and so raising prices this way was never going to land well. Unity made it worse by trying to unveil this price hike quietly, and ignoring internal pushback coming from its own employees, who accurately predicted the reception this move would get.
The biggest issue, however, is that the “install tax” is forced on game developers who would have never started developing on Unity, had they suspected such a tax would be levied on them. The promise of Unity was simple: pay upfront while you develop; then ship your game. The “install tax” is planned to kick in from 1 Jan 2024, and the only way to “opt out” is to leave the platform. This is not feasible for existing projects with large investments made into the engine.
Clearly, Unity is in a bad financial state: for every $2 that the company makes, it costs the company about $3 to do so (in the trailing 12 months the company made $1.8B in revenue, and recorded a nearly $1B loss). This per-install payment policy is a desperate attempt to quickly generate revenue, the medium and long-term consequences be damned. Moving game engines is time-consuming: so developers currently on Unity will have little choice but to grind their teeth, and pay more for the usage of Unity. However, I’d expect that for new projects: many teams will choose a competing engine without this “install tax.”
Unity’s desperate attempt to suddenly increase revenue is a reminder that publicly traded companies face investor pressure to generate profits and show growth, and this can push management to do seemingly insensible things which damage brands.
Unity is now in damage control mode, but the damage is already done, with the brand linked with the “Game tax” tag. Alternatives to Unity are being shared widely (this one by fellow Substack writer
from his publication ), some Unity staffers are quitting over it, and the winner will surely be other game engines which are currently enjoying free advertising. (Link)VanMoof (finally!) got acquired
In July, e-bike maker VanMoof filed for bankruptcy protection, as covered in The Pulse #56. Back then, I wrote the only question was whether the company could be sold before going bankrupt, because if the company entered liquidation, its bikes would likely become bricked. Fortunately for owners of the e-bike, a buyer has emerged.
Lavoie makes high-end scooters and acquiring VanMoof will add an e-bike to its range. With a new owner, VanMoof owners are likely to keep getting support and, most importantly, the Vanmoof servers which store encryption keys for each bike will not be shut down.
VanMoof raised $197M in total funding, as per Dealroom, but it’s likely it was sold for relative pennies, with investors losing the majority of their investment, and employees not realizing any monetary value in options. Unfortunately, this is the reality of a company that is massively loss-making. Lavoie now plans to spend tens of millions just to get their new business in order. It’s likely Lavioe won’t offer the same kind of loss-making service options that VanMoof did. The new owner wants bike shops to be able to fix VanMoof machines, which wasn’t the case before, and perhaps for retailers to sell the bikes, too. (Link)
2. The current market, as seen by engineering leaders at Big Tech
I recently talked with a couple of senior engineering managers, and director of engineering folks at large, publicly traded tech companies. Almost everyone was looking around, on the side, because they were not really happy where they were. At all of these companies, the common themes I heard were: