Either of two things has happened at Facebook. It has either been the target of a massive hack, or someone somewhere has screwed up in a major fashion. We are getting reports that the depth of the outage has various moving parts and is not simply a basic DNS error. This is something that will take hours at a bare minimum to fix and could even take days.
Facebook outage: massive hack or the biggest human error in social network history?
Briankrebs, whose self describes as “Independent investigative journalist. Covers cybercrime, security, privacy. Author of ‘Spam Nation,’ a NYT bestseller. Former Washington Post reporter, ’95-’09” quickly confirmed that it was indeed the DNS records that were causing the issue. DNS errors for Fortune500 are usually resolved within minutes but it looks like Facebook has become the victim of either one of the biggest hacks in history or a massive human error because Facebook’s authoritative BGP records have been withdrawn (seemingly by the company):
In what seems to define a central point of failure, a person that claims to be an engineer at FB reported that it would take some time for Facebook to come back up again as the outage has caused sprawling logistical nightmares across the company.
To quote, “There are people now trying to gain access to the peering routers to implement fixes, but the people with physical access is seperate from the people with knowledge of how to actually authenticate to the systems and people who know what to actually do, so there is now a logistical challenge with getting all that knowledge unified. Part of this is also due to lower staffing in data centers due to pandemic measures.”
And things seem to keep getting worse for the company because in an almost Hollywood-esque, because of the massive cascade failure of Facebook domains, and the loss of internal tools, Facebook employees were allegedly unable to even enter the building as their ID cards stopped working.
Sheera Frenkel reported that she was “just on phone with someone who works for FB who describer employees unable to enter buildings this morning to begin to evaluate extent of outage because their badges werent working to access doors.” This is something that could happen if Facebook has also lost their LAN/WAN due to networking issues.
The Facebook.com domain is also available for sale, although this is just a symptom of the DNS and BGP records being withdrawn and no one will actually be able to buy the domain even if they tried to but shows just how big of an outage this is.
According to Netblocks, the estimated cost of the outage for Facebook at the time of writing (roughly 4 hours) has already surpassed 600 Million USD and could easily cross a Billion USD soon.
This is a developing story and we will certainly add more details as they become known – especially whether this was human error or an actual hack – so stay tuned!