Known vulnerability disrupts log delivery.
Event logs at Cloudflare were discarded due to a vulnerability in the logpush service.
According to Bleeping Computer, the vulnerability affected 55 percent of users, and logs within a three and a half hour timeframe on November 14th.
The company explained in a blog post that the logs it normally sends to customers were not sent and were lost. “We’re very sorry this happened, and we are working to ensure that a similar issue doesn't happen again,” it said.
Individual Event Logs
It explained that on a typical day, Cloudflare sends about 4.5 trillion individual event logs to customers, however a faulty Logfwdr configuration update mistakenly provided a 'blank configuration' notice, prompting the removal of logs as the failsafe system began forwarding all logs to the distributed buffering system Buftee.
“The bug in the Logfwdr configuration system was easy to fix, but it’s the type of bug that was likely to happen at some point,” the company said.
“We had planned for it by designing the original ‘fail open’ behavior. However, we neglected to regularly test that the broader system was capable of handling a fail open event.”
The delivery of 40 times more logs than intended then resulted in Buftee shutting down within five minutes, further exacerbating log data loss and hindering recovery processes.
Dedicated Misconfiguration Detection
Such a significant log data loss due to the botched update has led Cloudflare to not only unveil a dedicated misconfiguration detection and alerting system but also strengthen Buftee's tolerance for excessive log volumes.
“We’re creating alerts to ensure that these particular misconfigurations will be impossible to miss, and we are also addressing the specific bug and the associated tests that triggered this incident,” Cloudflare’s blog concluded.
“Just as importantly, we accept that mistakes and misconfigurations are inevitable. All our systems at Cloudflare need to respond to these predictably and gracefully.”
Written by
Dan Raywood
Senior Editor
SC Media UK
Dan Raywood is a B2B journalist with more than 20 years of experience, including covering cybersecurity for the past 16 years. He has extensively covered topics from Advanced Persistent Threats and nation-state hackers to major data breaches and regulatory changes.
He has spoken at events including 44CON, Infosecurity Europe, RANT Conference, BSides Scotland, Steelcon and ESET Security Days.
Outside work, Dan enjoys supporting Tottenham Hotspur, managing mischievous cats, and sampling craft beers.
