The Pulse #67: Okta Schooled on Its Security Practices
Also: the largest DDoS attack to date mitigated; AWS quickly reversed the banning of re:Invent community apps; and advice for engineers considering joining a bootstrapped company.
The Pulse is a series covering insights, patterns, and trends within Big Tech and startups. Notice an interesting event or trend? Send me a message.
Today's topics are:
Okta schooled on security practices – by its own customers. One of Okta’s customers reported a security breach. Okta did nothing. Then, another reported it. Okta still did nothing. 20 days after the first report, Cloudflare – another Okta customer – detected the exact same security breach. How do Okta’s customers have better security processes than Okta, a security-focused company?
Industry pulse. A roundup of recent events, with commentary. The largest DDoS to date mitigated; a one-year-old AI company worth $500M; a potential L8-L9 slowdown at Google and more.
AWS’s swiftly unbans re:Invent scheduling apps. Developers maintaining community websites to help schedule talks to attend on re:Invent (AWS’s annual event) received an unfriendly message, telling them they must remove all AWS content. While AWS leadership reversed this decision in a day: one of the scheduling apps had already complied with the demand, and deleted its production database.
Advice for software engineers considering joining a bootstrapped company. Former Lyft software engineer – now the cofounder of the bootstrapped company Friendly Captcha – Guido Zuidhof shares what characteristics he looks for in early hires, and traits that help with thriving in this position.
1. Okta schooled on security practices – by its own customers
If I had to summarize what Okta sells in one phrase, I would say it’s security as a service. The company calls itself the world’s #1 identity management platform, offering a secure single-sign-on system. They are the SaaS solution thousands of businesses choose to take care of authentication and authorization – and keep sessions secure.
The company, however, has suffered a second, embarrassing data breach. Even worse: several of Okta’s customers discovered the breach before Okta did, disclosed it to Okta, and Okta did nothing, for weeks.
In 2022, Okta suffered a security breach through its customer support function, and only acknowledged it after a third party disclosed the breach happening. On 22 March 2022, a Twitter account shared evidence that suggested that Okta was compromised January 2022. The next day, Okta’s Chief Security officer acknowledged that Okta was, indeed, breached in January, confirming that it has taken Okta more than 2 months to investigate this incident, writing:
“On March 22, 2022, nearly 24 hours ago, a number of screenshots were published online that were taken from a computer used by one of Okta’s third-party customer support engineers. The sharing of these screenshots is embarrassing for myself and the whole Okta team.”
In this report, the company estimated that up to 366 of its customers – companies – could have had their environments accessed by a threat actor, for a period of 5 days. A month later, in April, Okta concluded its investigation stating that 2 customers were compromised, and the threat actor had access for 25 minutes.
Even back then, Okta drew serious criticism both for allowing this kind of compromise to happen, as well as for why they took 3 months to investigate and figure out what actually happened. Just as bad: Okta seemed to have no precise understanding of how the threat played out, two months into the investigation! If Okta has so much trouble figuring out how badly they were compromised: this doesn’t exactly offer reassurance to customers.
A few weeks ago, Okta suffered yet another security breach, again through its customer support channel. To make things worse, this time it was not Okta who detected the breach: but their own customers had alerted the identity management company of the breach. Okta then sat on this information, idle for weeks allowing more of its customers to be breached:
29 September: Okta notified of what looks like a breach. 1Passsword detected suspicious activity on their Okta instance. After a member of their IT support team uploaded a HAR file at the request of Okta – HAR files including sensitive information like session cookies – 1Password detected a bad actor using the same session contained in the HAR file to try to log onto 1Password systems. 1Password terminated the activity, investigated, and notified Okta about the breach coming from Okta.
2 October: another customer notifies of the same breach. A few days after 1Password reported on what looked like a breach, BeyondTrust detected the same type of breach using a HAR file. Just like 1Passsword, BeyondTrust also reported the breach to Okta.
Aside: in what is an amusing detail: BeyondReach kept escalating the issue, as Okta didn’t give them any form of acknowledgment until 19 October! That’s 17 days of ongoing escalation.
18 October: yet another customer notifies Okta of the same breach. 19 days after the 1Password breach, an attacker attempted to breach Cloudflare using this same HAR file. Cloudflare detected and blocked the attempt. The amusing part? Despite Okta knowing about the beach for a long time, they have not notified their customers until this point! From Cloudflare: “In fact, we contacted Okta about the breach of their systems before they had notified us.”
19 October: Okta notifies customers about the breach. The first time customers get acknowledgement from Okta.
20 October: Okta publicly announces it suffered a security breach; doesn’t share details. The Chief Security Officer of the company confirms that attackers were indeed able to access Okta’s support system and access HAR files.
In what is yet another amusing detail in the report, Okta seems to completely evade how it was their customer support system that was breached, or how the HAR file they requested from customers contained the sensitive data that attackers can use. Instead, it felt to me that Okta was pushing responsibility on their customers, writing things like:
“Okta recommends sanitizing all credentials and cookies/session tokens within a HAR file before sharing it. Attacks such as this highlight the importance of remaining vigilant and being on the lookout for suspicious activity.”
Reminding customers of the importance of remaining vigilant right when Okta failed to do so? Is Okta truly in the position to give this advice, when the company failed to do exactly this?
Okta has not given little reason for its customers to trust that it can keep customer data secure. How this breach played out has been deeply disappointing for anyone paying attention to security:
An Okta system that was breached a year ago – customer support – was again breached.
The duration of the breach was at least 20 days. This is unacceptably long.
Okta did not notify customers for at least 19 days after it was aware of the exact details of the breach
When publicly acknowledging the issue, Okta was not transparent. The details we know about the breach came not from Okta: but from 1Password, BeyondTrust, and CloudFlare.
Customers are, understandably, very annoyed. One customer is particularly so: Cloudflare. In what is a rare move, Cloudflare published a summary on the incident where they helpfully remind Okta about security practices that any security provider should follow – and Okta has not done so. I have yet to see this level of schooling of a security-first company. Here is what Cloudflare writes:
“We urge Okta to consider implementing the following best practices, including:
Take any report of compromise seriously and act immediately to limit damage; in this case Okta was first notified on October 2, 2023 by BeyondTrust but the attacker still had access to their support systems at least until October 18, 2023.
Provide timely, responsible disclosures to your customers when you identify that a breach of your systems has affected them.
Require hardware keys to protect all systems, including third-party support providers.
For a critical security service provider like Okta, we believe following these best practices is table stakes.”
I would not be surprised to see Okta lose lots of business over the damning details of this security incident. Like it or not: security breaches are challenging to detect. This is why it’s sensible for many companies to buy security solutions rather than build them: buying solutions like Okta.
But what happens if you realize thatyour security vendor doesn’t follow basic security hygiene? And when you see that the same vendor subtly shifts blame on customers, in their public report?
Well, if you are a customer who was considering moving to Okta: you simply go with another vendor. Enterprise salespeople at Okta will have a hard time defending the indefensible: that Okta knew about a breach for closer to 3 weeks, and yet it did nothing about it. The company didn’t even nofify customers like Cloudflare!
What about Okta’s current customers: could they churn? Moving off of an identity provider is so expensive and time-consuming: so it is likely that only a minority of customers would leave. Security-conscious companies might move off, though: for example, Basecamp migrated off of Okta after their January 2022 breach and poor follow-up.
Existing Okta customers should assume that if Okta could be breached again, unless they see major changes happen a the company. Already twice, Okta was compromised through the same customer support system, and the company offered little transparency on what really happened. Without major changes in how Okta approaches security, Okta’s customers need to invest more in “remaining vigilant and being on the lookout for suspicious activity” – from their Okta account as well!
Okta’s stock price has taken a warranted beating since the disclosure of this breach: the company’s market cap fell from $14B to $11B in one day. Investors are right to be pricing in what will likely be a struggle to sign new customers – and to hold on to existing ones.
Will Cloudflare launch a competitor product to Okta? It is rare to see such a direct response to a breach than what Cloudflare has done. In the response, Cloudflare also highlighted how their Zero Trust architecture was key in ensuring that the breach was caught quickly:
“How Cloudflare mitigated yet another Okta compromise (...)
The attacker used an open session from Okta, with Administrative privileges, and accessed our Okta instance. We were able to use our Cloudflare Zero Trust Access, Gateway, and Data Loss Prevention and our Cloudforce One threat research to validate the scope of the incident and contain it before the attacker could gain access to customer data, customer systems, or our production network. With this confidence, we were able to quickly mitigate the incident before the threat-actors were able to establish persistence.(...) Upon detection, our SIRT was able to engage quickly to identify the complete scope of compromise and contain the security incident. Cloudflare’s Zero Trust architecture protects our production environment, which helped prevent any impact to our customers.”
Clodflare’s market cap is $20B, and Okta’s market cap used to be $14B, before the breach. Cloudflare seems to have far better security controls than Okta, and it keeps establishing itself as a responsible vendor. Cloudflare has more than 100,000 paying customers, while Okta has just over 17,000 – a customer referring to a business. If Cloudflare were to launch an identity provider: on day one, this product would have a better security track record than what Okta has, and such a product could be an obvious growth path.
The more I think about it, the more surprised I would be to see Cloudflare not launch some kind of offering to directly compete with Okta: built on their Zero Trust platform. If and when this happens, it will be more bad news for Okta. Then again, the identity provider was the one who has invited serious competition by under-investing in their security hygiene and practices.
If you work at a vendor: learn from the mistake that Okta has made. Ask yourself: how does your company ensure that security basics – like immediately acting on breach reports – are always followed, without exception? What security processes are in place, and are they sufficient to detect and mitigate a breach like the one Okta has experienced?
Related to Okta: we previously covered how the company had the highest median compensation across all tech companies in 2022 at a whopping $397,000 per year – much of this likely thanks to the Auth0 acquisition.