Unsolvable Network Issue, defeated nearly 7 engineers.

[JamesHewitt]

This issue has been going on for 3 weeks now and so far it's defeated 4 Network Engineers over the phone, two BT site engineers along with multiple Openreach staff(UK's Main Fibreoptic Company) including a site engineer, senior engineer and one of the heads of Openreach.

 

In other words, every possible technician have confirmed the fault but are all dumbfounded in finding a solution.

 

So, what is this incredible fault you may ask.

 

90% of websites work fine, internet working fine HOWEVER. Any website for things such as Flights, Hotels, Steam, Disney+ and now occasionally my solar energy are completely blocked.

 

Some give weird errors, most just say "The server at x website is unreachable".  I have attached photos below.

 

 

So what have we done to attempt to fix this.

 

We have replaced every ethernet cable, Factory reset the router, replaced the router, double checked all settings including turning on and off parent controls incase it bugged out, had the entire line all the way back to the exchanges checked, reset the line, signal boosted, every literal test they can do.

 

It is EVERY device which connects to our internet. If you turn the internet off and use data, works. If you connect to another network, works.

 

 

This is the fun part.

 

If you turn a VPN on, it usually improves the results, getting atleast further or fixing some websites but not all.

 

 

So, if anyone has any suggestions at all and wants a crack at it, I will be forever in your debt.

 

All the best,

 

James

 

[Oshino Shinobu]

If you ping easyjet.com and steampowered.com, what IPs do they resolve to?

 

What DNS servers are you using:? If you have them set automatically, change them to manual on your PC and set them to something like google (8.8.8.8, 8.8.4.4) or Cloudflare (1.1.1.1, 1.0.0.1), then type ipconfig /flushdns into CMD and try accessing the site again

[CWALD]

sounds like a DNS problem. I would try setting your DNS manually to one of the ones @Oshino Shinobu recommended, turn on your VPN, and then flush your DNS. If it works then you can try it without the VPN, and finally with the default DNS server. If you get this far and it still works then something else you did already fixed it and you just need to flush DNS on all your devices.

[Oshino Shinobu]

1 minute ago, CWALD said:

sounds like a DNS problem. I would try setting your DNS manually to one of the ones @Oshino Shinobu recommended, turn on your VPN, and then flush your DNS. If it works then you can try it without the VPN, and finally with the default DNS server. If you get this far and it still works then something else you did already fixed it and you just need to flush DNS on all your devices.

Yep good to do one thing at a time to make sure you know what fixes it.

 

Between these steps, flush you DNS each time or you'll end up with cached DNS entries

[JamesHewitt]

7 minutes ago, Oshino Shinobu said:

If you ping easyjet.com and steampowered.com, what IPs do they resolve to?

 

What DNS servers are you using:? If you have them set automatically, change them to manual on your PC and set them to something like google (8.8.8.8, 8.8.4.4) or Cloudflare (1.1.1.1, 1.0.0.1), then type ipconfig /flushdns into CMD and try accessing the site again

They typically don't work, some do.

 

Also I've tried changing the DNS and secondary to both google and cloudflare both on my PC and my router. Didn't fix the issue.

[JamesHewitt]

I have also flushed the DNS on this PC, however it happens across 4 PC's and 5 phones at the exact same time.

 

Does the command do it on the routers side or local? As it wont fix phones otherwise.

[Oshino Shinobu]

4 minutes ago, JamesHewitt said:

They typically don't work, some do.

 

Also I've tried changing the DNS and secondary to both google and cloudflare both on my PC and my router. Didn't fix the issue.

Remove the www from the pings, just ping the domain

 

Did you flush your DNS after you changed them?

The errors you're seeing are often caused by trying to access a server that's in a different region. Seeing as it's on all devices, most likely issue seemed to be DNS doing something funny

 

1 minute ago, JamesHewitt said:

I have also flushed the DNS on this PC, however it happens across 4 PC's and 5 phones at the exact same time.

 

Does the command do it on the routers side or local? As it wont fix phones otherwise.

The command is only local. It's not a fix, it just clears cached entries so that while testing solutions you can see if it actually makes a difference. If it is DNS and we find servers that fix it, for other devices they'd start fixing once the TTLs expire.

[Oshino Shinobu]

Something else to try is to use the below commands in powershell and see if it connects:

 

test-netconnection store.steampowered.com -port 443

test-netconnection store.steampowered.com -port 80

 

[JamesHewitt]

11 minutes ago, Oshino Shinobu said:

Remove the www from the pings, just ping the domain

 

Did you flush your DNS after you changed them?

The errors you're seeing are often caused by trying to access a server that's in a different region. Seeing as it's on all devices, most likely issue seemed to be DNS doing something funny

 

The command is only local. It's not a fix, it just clears cached entries so that while testing solutions you can see if it actually makes a difference. If it is DNS and we find servers that fix it, for other devices they'd start fixing once the TTLs expire.

They started to work  but also I've just checked on my phone and it's came back. 

 

It's not always broken, it can be off fort wo hours or two days. Annoyingly right after I make this post it seems to have fixed itself.

 

I will have to leave this post till it breaks probably over the weekend then try again.

 

Could you please explain why you think it's the DNS? This way when BT phone up again I can explain what all of you thought.

 

Thank you so much again,

[Kilrah]

Check if you're behind CGNAT:

https://www.purevpn.com/blog/how-to-check-whether-or-not-your-isp-performs-cgnat/#How_to_check_if_the_ISP_performs_CGNAT

 

If so it might be that other people on the same ISP have been doing naughty things and those websites are temporarily blacklisting the source IP, that will be the same for a whole host of this ISP's customers and you might be caught in the crossfire.

[Oshino Shinobu]

1 minute ago, JamesHewitt said:

They started to work  but also I've just checked on my phone and it's came back. 

 

It's not always broken, it can be off fort wo hours or two days. Annoyingly right after I make this post it seems to have fixed itself.

 

I will have to leave this post till it breaks probably over the weekend then try again.

 

Could you please explain why you think it's the DNS? This way when BT phone up again I can explain what all of you thought.

 

Thank you so much again,

The fact that it's on all devices when you connect to your network, then it goes away when you disconnect and use data or a different network rules out it being a browser or device config issue. The fact that you've reset the router to factory makes it unlikely that you've enabled web filtering or a proxy on the router and it's unlikely to have that enabled by default.

 

Most likely thing remaining is the DNS servers being assigned by DHCP, or that the router is configured to forward to, are resolving to IPs outside your region or are taking you down a weird route to get to them.

 

My recommendation here would be to go to your router's config and change the DHCP options to hand out one of the DNS servers I suggested earlier (or a combo, so 8.8.8.8 primary and 1.1.1.1 secondary) and then see if the issue improves. It can take a bit of time as devices will hold on to cached DNS records until the TTL expires, after which they'll grab new records using the configured DNS servers.

[JamesHewitt]

5 minutes ago, Kilrah said:

Check if you're behind CGNAT:

https://www.purevpn.com/blog/how-to-check-whether-or-not-your-isp-performs-cgnat/#How_to_check_if_the_ISP_performs_CGNAT

 

If so it might be that other people on the same ISP have been doing naughty things and those websites are temporarily blacklisting the source IP, that will be the same for a whole host of this ISP's customers and you might be caught in the crossfire.

Checked, it's not in that range of IPs.

[Kilrah]

1 minute ago, JamesHewitt said:

Checked, it's not in that range of IPs.

Read the rest of the article, the range is only one potential pointer, the real one is comparing the router WAN address to what you get from a website that gives the originating IP.

[JamesHewitt]

2 minutes ago, Kilrah said:

Read the rest of the article, the range is only one potential pointer, the real one is comparing the router WAN address to what you get from a website that gives the originating IP.

Checked now, IPV4 and my broadband IP in the router are a perfect match outside of that range.

[kirashi]

1 hour ago, JamesHewitt said:

This issue has been going on for 3 weeks now and so far it's defeated 4 Network Engineers over the phone, two BT site engineers along with multiple Openreach staff(UK's Main Fibreoptic Company) including a site engineer, senior engineer and one of the heads of Openreach.

 

In other words, every possible technician have confirmed the fault but are all dumbfounded in finding a solution.

 

So, what is this incredible fault you may ask.

 

90% of websites work fine, internet working fine HOWEVER. Any website for things such as Flights, Hotels, Steam, Disney+ and now occasionally my solar energy are completely blocked.

 

Some give weird errors, most just say "The server at x website is unreachable".  I have attached photos below.

 

 

So what have we done to attempt to fix this.

 

We have replaced every ethernet cable, Factory reset the router, replaced the router, double checked all settings including turning on and off parent controls incase it bugged out, had the entire line all the way back to the exchanges checked, reset the line, signal boosted, every literal test they can do.

 

It is EVERY device which connects to our internet. If you turn the internet off and use data, works. If you connect to another network, works.

 

 

This is the fun part.

 

If you turn a VPN on, it usually improves the results, getting atleast further or fixing some websites but not all.

 

 

So, if anyone has any suggestions at all and wants a crack at it, I will be forever in your debt.

 

All the best,

 

James

 

That's the standard "Access Denied" message that appears when a service behind Akamai's CDN is blocking your access. You'll need to contact the site administrators to determine why you're being blocked and see if they can come up with a solution.

https://community.akamai.com/customers/s/article/Why-is-Akamai-blocking-me?language=en_US

 

Quote

Description

Why is Akamai blocking me?

Akamai does not block users from accessing our customers’ websites. However, our customers can use tools and policies which may in turn block you (the end user). Our customers use these rules to protect them and you from malicious actors on the internet. Some common reasons could include:

• Explicit IP blocking / blacklisting
• Location-based blacklisting
• Rule-based blocking (i.e. web application firewall protections)
• Reputation-based blocking
• HTTP request rate controls (e.g. DoS protections)

The following activities may trigger application security controls:

• Web application layer attacks such as: SQL Injection, Cross-Site Scripting, Local File Inclusion, Remote Command Execution, Remote File Inclusion, etc.
• Volumetric attacks or similar high rate HTTP traffic
• Web contents scraping, data mining, web content indexing and similar automated web activities
• Web vulnerability scanning using automated tools

Your reputation follows you. If your IP is identified as behaving poorly on one site, you may be blocked on other websites. A first step in troubleshooting may be to determine whether your organization is performing one of the activities listed above that could affect your reputation.

You can use our Client Reputation lookup application at https://www.akamai.com/us/en/clientrep-lookup/ to check whether your connecting IP address received a bad reputation score and submit an investigation request in case you believe it was mistakenly flagged as malicious.

When a page cannot be accessed, whether because of a customer policy blocking access to that resource or a variety of other reasons such as a server error, the error page will typically be presented as follows:

 

Notice the reference number. Akamai customers can use this reference number to identify why this request failed.

If you are unable to access a web site, this may be the result of a policy configured by the site owner you are attempting to access. To make a change to this policy, the site owner (the Akamai customer) would have to change their policy. Akamai is unable to make this change without the explicit direction of the site owner. To obtain the contact information for a site owner, one avenue to explore might be via whois. Please contact the site owner directly and have them in turn contact Akamai if they believe that you should be able to access the resource.

To reiterate, Akamai is unable to make this change without the explicit direction of the site owner.

 

[DarkWaterSong]

Also fun fact, ISPs like to start blocking you if you use a VPN they can smell.  Reason is it lets you get around region locks on streaming services and other fun, and the ISP can end up owing the streaming service if they let that happen on their network.  We actually had a problem with this at one places I worked, because our sales staff used VPNs to reach our domain when out of the office.  And many a time a hotel's network blocked all VPNs, so they had to find other places to work.  We ended up sending them out with cellular routers or having them hot spot off their phones most of the time.