Latency & Server Issues - ISP's, Backbone Providers, and Content Delivery Providers

EChondo · May 30, 2015

Source: Personal Findings

Hello all, recently I've been having problems loading certain websites, either my request will time out and 403 or the website will load basic HTML and only display text. I have found that ISP's(AT&T,Comcast, etc.), backbone providers, specifically Level 3/NTT, as well as CDP like Akamai are having massive latency issues right now. This is either due to server issues or a DOS/DDOS attack is happening. I can't verify what the issue is since no one is talking about this and I thought I'd provide my findings and issues just in case others are having similar problems.

Now how am I diagnosing these issues? Well Command Prompt in Windows has a neat command called "tracert domain", you enter this command with a domain and it will report back every IP you go through as well as how much latency is involved.

My first example is Origin.com;

As you can see I have around 20ms ping until I reach Level 3 in Washington that reports back 90ms, then it goes through some IP's until I get to JetSetSecrets.es, which is owned by EA and I believe is their CDP.

Here are other examples;

This one is massive, you can see that once I reach the major POP of Los Angeles my ping increases from ~20ms to 130ms! The request times out at 30, so I can't see how much further it goes, but you get the idea.

With this one you can see that AT&T(my ISP) isn't fond of Netflix. :angry:

Here is a list of other websites that have similar issues as well as others that report back fine.

So I have reason to believe that there are issues going on, right now, with some major players. If any of you have had problems with websites I implore you to do a tracert and screenshot your results(block out your IP, unless you don't care) and post it in this thread.

Obviously this isn't an issue on my end, I can still go to YouTube and watch 4k videos with almost no buffering, Netflix is so-so, and other websites like LTT, Reddit, and Imgur load without major issues(Imgur is kinda slow). But other websites like AMD, USAToday, and Steam refuse to load. I can still play games on Steam and download from their servers, but I can't go to community pages or even my own Steam profile!

There are also huge outages happening;

AT&T: https://downdetector.com/status/att/map/

Comcast: https://downdetector.com/status/comcast-xfinity/map/

Time Warner: https://downdetector.com/status/time-warner-cable/map/

Verizon: https://downdetector.com/status/verizon-communications/map/

Kilerbomb · May 31, 2015

I've been noticing this too... I just thought it was on my end. Good find. Hopefully this blows over soon enough.

Neil · May 31, 2015

Couple notes:

The timeouts after reaching the host for a lot of yours indicate that somewhere inside the network, ICMP traffic is being dropped, which causes the "timeout" -- this is not actually an issue.

Your latency is within acceptable parameters. Not saying something isn't wrong--but your latency isn't abnormal especially when hitting border routers.

using the pathping (http://en.wikipedia.org/wiki/PathPing) tool is a better one for diagnostics.

thebennyboy · May 31, 2015

Current DDOS attacks: http://www.digitalattackmap.com/#anim=1&color=0&country=ALL&list=0&time=16585&view=table

SubTract · May 31, 2015

Trace Rout is a fairly flawed tool. Routers / Servers can simply ignore the request and not respond at all, the data required for the information is dropped by the local network (often to reduce overhead) Or the response could take more than the timeout time.

Its great for things diagnosing this locally when you know what everything inside your network is doing.

It is also common to see these kind of latency increases from 1 point to the next.

Can be caused by physical distance, different protocols, type of interface, whether they encrypt local traffic, condition of line, Data correction As well as the possibility of a busy router (DOS/DDOS) or high traffic throughput.

E.g. I live in Western Australia (So expect High pings to begin with) From my house to perth its 15ms, From Perth to Melbourne its 75ms from Melbourne to Sydney is 15ms. Then from Sydney to the Telstra Global network (sydney connection) it takes 200ms. Then proceeds to the US where the global to local connection takes about 60ms. Then local is all within a couple of ms.

None of your trace routes seem to be abnormal to me.

A DOS/DDOS will tend to cause erratic fluctuations on pings and throughput. In fatal cases no response at all.

Anyhow if a point became more restricted than others data would follow a different route (If available)

Meaning that the route you are getting is still the best available to you.