Google accident 'breaks' the Internet in Japan

[WMGroomAK]

In a moment of carelessness, Google accidentally took down a large part of the Internet in Japan temporarily and caused a widespread slowing of internet traffic.  According to an Engadget article, Google had set the wrong network information which made it appear as if large chunks of IP Addresses were to be routed through Google instead of the Japanese ISPs they belonged to.  This resulted in the traffic getting redirected to nowhere and caused large delays in service even after the problem was fixed within 8 minutes of occurring.

 

https://www.engadget.com/2017/08/28/google-accidentally-broke-internet-japan/

Quote

Last week, nearly half of Japan briefly lost access to the internet because of an error made by Google. The mistake was noticed and corrected within just a few minutes, but its effects led to hours of slowed internet connections. The impact was so large that Japan's Internal Affairs and Communications Ministry initiated an investigation into the issue.

 

The problem occurred around noon on Friday. A mistake on Google's end made it appear as if a large chunk of IP addresses were available for internet traffic to be routed through Google. However, those IP addresses actually belonged to Japanese internet service providers. So a large amount of traffic that was meant for Japan was suddenly sent towards Google by major internet service providers like Verizon. But Google isn't a service provider and isn't meant to route traffic, so all of that traffic destined for Japan was basically just sent nowhere.

 

Connectivity was restored within the hour, but persistently slow connection speeds resulting from the mistake affected industries like finance, where online trading was halted, and transportation -- East Japan Railway Co. riders weren't able to buy tickets or board trains. The internet service providers impacted the most were KDDI Corp. and NTT Communications, the latter of which provides service to well over seven million people.

 

A Google spokesperson told Asahi Shimbun, "We set wrong information for the network and, as a result, problems occurred. We modified the information to the correct one within eight minutes. We apologize for causing inconvenience and anxieties (among Internet users)."

Seems like a major Oops moment that you would hope they had some kind of checks for, but I guess it's good to know about this potential issue now and maybe find a solution for it... 

 

EDIT:  Just got around to the Bleeping Computers article, which has a bit more of the technical details on what happened, if not why, who or how...  Basically, Google 'hijacked' about 135,000 network prefixes from around the world, which redirected all of that traffic to Google.  One interesting bit is that the article mentions that this error led to inadvertently broadcasting these prefixes to Verizon who then passed this on down the network...

 

https://www.bleepingcomputer.com/news/technology/google-error-causes-widespread-internet-outage-in-japan/

Quote

According to an analysis by BGPMon, a company part of OpenDNS, part of Cisco, Google hijacked over 135,000 network prefixes, from all over the world, of which over 24,000 belonged to NTT, by far the most affected ISP.

 

"It’s easy to make configuration mistakes that can lead to incidents like this," says Andree Toonk, one of the BGPMon engineers who has analyzed the outage.

 

"In this case it appears a configuration error or software problem in Google's network led to inadvertently announcing thousands of prefixes to Verizon, who in turn propagated the leak to many of its peers," he added.

 

"Since it is easy to make configurations errors, it clearly is a necessity to have filters on both sides of an EBGP session. In this case it appears Verizon had little or no filters, and accepted most if not all BGP announcements from Google which lead to widespread service disruptions," Toonk also said. "At the minimum Verizon should probably have a maximum-prefix limit on their side and perhaps some as-path filters which would have prevented the wide spread impact."

 

 

Edited August 28, 2017 by WMGroomAK
Additional Article and info

[Lurick]

Yah, there are mechanisms that exist within BGP to prevent this type of thing but that involves actually knowing what you're doing AND how to go about implementing them  

[mynameisjuan]

Ah good old BGP.....This is a lot more common of an issue than people realize. But I dont know why it took 8 mins to fix, its like a 20 sec fix follow by 30 sec for BGP to fail over..

[The Benjamins]

2 minutes ago, mynameisjuan said:

Ah good old BGP.....This is a lot more common of an issue than people realize. But I dont know why it took 8 mins to fix, its like a 20 sec fix follow by 30 sec for BGP to fail over..

The 8min could also be the time it took for them to realize they messed up.

[mynameisjuan]

1 minute ago, The Benjamins said:

The 8min could also be the time it took for them to realize they messed up.

I had to do an emergency DNS cut over that took DNS down for 13 secs. In that time we had 43 calls of no internet. And that is for about 10,000 customers. They would of known really really quick lol

[Lurick]

23 minutes ago, valdyrgramr said:

You all laughed at me when I used Bing!  Look who's laughing now!

It's still us laughing at you for using Bing  

[mynameisjuan]

26 minutes ago, valdyrgramr said:

You all laughed at me when I used Bing!  Look who's laughing now!

Jokes on you, A BGP change means you cannot even reach bing if you are on those IP blocks.

[matrix07012]

27 minutes ago, valdyrgramr said:

You all laughed at me when I used Bing!  Look who's laughing now!

I'm still laughing. Use DuckDuckGo.

[mynameisjuan]

2 minutes ago, matrix07012 said:

I'm still laughing. Use DuckDuckGo.

Doesnt Duckduckgo use bing's engine? Just without the telemetry data of course 

[matrix07012]

3 minutes ago, mynameisjuan said:

Doesnt Duckduckgo use bing's engine? Just without the telemetry data of course 

No, It's just one of it's sources.

https://duck.co/help/results/sources

[Sauron]

Honestly it seems to me verizon is at fault here... there is no reason to rely so heavily on google's network settings...

[Lurick]

1 minute ago, valdyrgramr said:

Here I am with my internetz and my gift cards.

Don't make me disable your internet  

[Lurick]

1 minute ago, valdyrgramr said:

Where's your gift cardz m8?

I don't have any ;-;