Why Facebook, WhatsApp and Instagram Went Down For Hours

Two days ago, social media pulled a reverse Uno card on itself by trending on social media.

Facebook and its affiliated apps (WhatsApp, Instagram, Messenger and Oculus) went offline for more than six hours in one of the largest global internet outages.

Here’s what happened? Well, in techno-speak, the DNS (Domain Name System) names of all these sites “stopped resolving” (we’ll explain this shortly) and their IP (Internet Protocol) infrastructure was unreachable. It was as if somebody had “pulled the cables” from their data centres all around the world at once and disconnected them from the internet!

While outages like this aren’t uncommon, this came at a particularly bad time for Facebook. The social media giant is facing increasing criticism following the whistleblower revelations that present some pretty damning evidence about the company’s questionable policies and “moral bankruptcy”.

Conspiracy theorists are already at work trying to connect the dots between the two events!

Regardless, this extended downtime cost the company nearly $100m in revenues. By the end of the day, Mark Zuckerberg had lost approximately $6.11bn of his wealth and Facebook shares had plunged by 4.9% (biggest daily drop in 11 months).

So, let’s try and understand what caused the latest outage and what implications it carries for the company’s governance at large.

The Outage

At the core of this outage lies a technology called BGP or Border Gateway Protocol. Now, we know that the internet is a network of networks. But what enables one network to know, recognise and identify the availability of another is the BGP. Facebook uses this mechanism of BGP-enabled “peering” to advertise its presence to other networks.

On October 4th, Facebook’s link to the BGP was broken. As to why this happened, the company’s engineers said that its backbone connection between data centres shut down during a “routine maintenance”. This is quite vague because a maintenance update could arise due to reasons ranging from a simple software configuration to a server attack.

But what followed the link breakage was a “route withdrawal” meaning that Facebook’s DNS servers went offline. DNS is what tells a computer the IP address of a domain name (like facebook.com). Once that fails, people can’t access that website.

When the Facebook apps and website failed to open, more and more people kept checking the service in an effort to login. This led to a jump in site traffic which in turn pushed up the number of queries on DNS resolver forums. Cloudflare, a web infrastructure firm, reported a 30x jump in the DNS resolver queries worldwide following the outage. (Here’s an interesting infographic that explains the outage.)

There are a few workarounds (aka fallback mechanisms) for situations like this which usually involve re-establishment of contact with the DNS servers manually. But those mechanisms failed too because even if the DNS servers were up and running, Facebook’s connection with the BGP was broken. It’s almost like Facebook had locked itself out of its own car!

The Fallout

Facebook and WhatsApp each have more than 2 billion active monthly users, making these apps essential communication tools around the world. In some regions, these apps are almost synonymous with an internet connection for users due to their cheaper and popular availability vis-a-vis other voice and messaging media platforms. Advertising revenue, which is central to Facebook’s earnings, was affected enormously during the outage (about $545,000 per hour as per some estimates). Businesses worldwide, especially small and medium enterprises which rely on the apps to operate (take orders, etc.) came to a grinding halt, particularly in countries like India which is one of the largest user bases for the Facebook apps.

In fact, it wasn’t just Facebook but a bunch of other apps which faced noticeable slowdowns as users from Facebook apps started flocking elsewhere. Twitter, Signal, Telegram etc. received the maximum increase in site traffic. Meanwhile Facebook, with its considerable size and resources, continued to dabble with such a massive error for hours, indicating that there was no easy fix to it.

And it’s not like Facebook is any stranger to outages. Here’s a timeline of its outage outrage over the years:

“I Link, Therefore I Am”

With an increasingly internet-driven and -dependent world, disruptions due to outages have become commonplace. Within the past year alone, large and popular networks like Slack, Google, Gmail, YouTube and Fastly have faced service disruptions for several hours.

Which begs the question, how dangerously profound is the impact of a social media blackout of this magnitude? Nearly 3.5 billion people around the world, that is, half of all human beings on this planet, use the Facebook suite of apps to communicate, outreach, advertise, distribute messages and expand businesses. So, the apps dropping from the map essentially correspond to livelihoods falling through the cracks, however momentarily.

Facebook has also spent years building its network infrastructure and helping subsidise data plans and phones to make its presence known in the remotest corners of the world. But when it comes to public accountability, placing such outsized dependence on a corporation that is under scrutiny for its questionable ethics and governance is somewhat discouraging.

Especially when this “social media black hole” followed on the heels of reports involving user data manipulation and privacy concerns. Reports in the media have led to internal documents of Facebook becoming public. Documents which paint an incriminating picture of the company when it comes to combating issues like hate speech, teenage body image and privacy safeguards (link to “The Facebook Files”). With each passing day, Facebook becomes prone to more scrutiny because its use as a tool of empowerment is overshadowed by its alleged misuse and “prioritisation of profits” over ethics.

The times are evidently not on Facebook’s side. With inadequate contingencies in place to minimise the damages of an outage like this, there’s no telling in whose hands the leviathan repository of user data held by Facebook is going to end up.

Perhaps, rethinking the ways in which Big Tech companies are allowed to regulate themselves is not such a bad idea after all.

(Originally published October 6th 2021 in transfin.in)