Crowdstrike blames defect in content update for massive global IT crash

By Katrina Manson and Ryan Gallagher
 

CrowdStrike Holdings Inc., the cybersecurity company at the center of massive global IT outages, said that a bug in a safety mechanism allowed flawed data to go out to customers in a botched update, causing last week’s meltdown. 
 

The US company is trying to piece together the series of events that led to one of the most spectacular rolling IT failures the world has ever seen. The incident crashed Microsoft Windows computer systems around the world on Friday, taking down airline, banking and stock exchange operations from Australia and Japan to the UK.

Microsoft and CrowdStrike rolled out fixes last week, and many systems have been restored. But for several hours, bankers in Hong Kong, doctors in the UK and emergency responders in New Hampshire found themselves locked out of programs critical to keeping their operations afloat. More than 8.5 million Windows users were affected, according to Microsoft. 

In the report, the company said it regularly makes what are known as security content configuration updates, intended to help the company observe, detect or prevent malicious activity, depending on the customer’s policy configuration. A “problematic Rapid Response Content configuration update” carried an undetected error and crashed Windows systems, the company said in a preliminary post-incident review, published about five days after the incident.

CrowdStrike said it would improve testing of Rapid Response Content in future, in a variety of ways. It said a new check “is in process” in order to fix the faulty Content Validator that failed to vet the problematic content. CrowdStrike also plans to stagger future deployments of updates so they are tested piecemeal - known as a canary deployment - before rolling it out at large. 

Finally, the company said it would allow customers greater control over the delivery of such content, so they can select when and where updates are deployed. 

CrowdStrike’s shares dropped nearly 30 per cent in the aftermath of the outage, slashing billions of dollars from its market value. The US House Committee on Homeland Security requested Chief Executive Officer George Kurtz’s appearance and lawmakers called on him to explain how the company will mitigate risks of a similar incident in the future. 

Shawn Henry, CrowdStrike’s chief security officer, apologized in a post on LinkedIn on Monday, saying that the company had “failed” its customers. 

“The confidence we built in drips over the years was lost in buckets within hours, and it was a gut punch,” he said.