Internet outage hits one quarter of Canada

[williamcll]

 

Canadian networks are slowly recovering after an outage knocked out a quarter of the county's internet.

Quotes

Quote

Canada’s tax collection agency lost telephone service. The country’s largest airline, Air Canada, reported technical problems at call centers. Even the Canadian telecom regulator said its phone lines were affected. In the city of Winnipeg, police asked Rogers users who need to call 911 to try to find landlines or phones on another network in case of emergency. Rogers Communications said late Friday that it had “made meaningful progress” toward restoring service but that it could not say when the networks would fully recover. “Many of our wireless customers are starting to see services return,” the company’s president, Tony Staffieri, said in the statement. “I want to sincerely apologize for this service interruption and the impact it is having on people from coast to coast to coast.” It was the second disruption at the provider in a little over a year. Critics pointed to the outages Friday as evidence that there should be more competition in Canada’s telecom industry. Rogers, which has more than 11 million wireless subscribers, is one of three companies that dominate the market in Canada.

With the failure of government services, including the passport office, Conservative Party lawmaker Michelle Rempel Garner said the latest outage showed the potential “national vulnerability” because of limited competition. Gordon Barton, who was attending a weekend music festival, told the Canadian Broadcasting Corp. he had deposited some cash to spend money with his card but could no longer use it. “But that’s maybe not a bad thing,” he quipped. “I mean, I don’t think a cash-only system is the answer,” he added, “but it’s scary how quickly things can come down.” Vancouver International airport said travelers could not pay for food or use ATMs in its terminals. One domestic airline in the north said it was unable to contact passengers with flight updates. The disruption also prompted a health-care network in Toronto to ask physicians who were on call Friday to “physically come into the hospital for the shift.” Meanwhile, at a venue in the city, disappointed crowds lingered outside after a concert by the Weeknd was postponed at the last minute because of the outage. “I’m crushed & heartbroken,” the Canadian singer tweeted Friday night. “Been at the venue all day but it’s out of our hands because of the Rogers outage.”

My thoughts

Oh dear, maybe consider a fallback system next time and maybe, just maybe not have a monopoly on the network infrastructure? 

 

Sources

https://www.washingtonpost.com/world/2022/07/09/canada-network-outage-rogers-communications-internet/

[dizmo]

They don't have a monopoly, there are other providers.

However, I think that, since this has happened twice, the Canadian government needs to step in and find a different solution, especially in the banking sector.

Having people unable to access their money is unacceptable.

[Fasterthannothing]

48 minutes ago, dizmo said:

... the Canadian government needs to step in 

Ah yes because Government needs one more thing to mess with 

[dizmo]

1 hour ago, Fasterthannothing said:

Ah yes because Government needs one more thing to mess with 

When it comes to ensuring that retail can still function and people can still buy necessities like food, yes, yes they do.

[PeachGr]

Well I m Greek and hearing that that's uncommon and it reaches governments and whatnot makes me smile and cry at the same time

[GhostRoadieBL]

Toxic company pulling yet another temper tantrum outage, if I remember correctly this happened when net neutrality was announced too but less wide spread and much shorter.

 

When we moved in our choice was telus fibre or nothing because they "owned" the neighbourhood area's backbone. Other areas near us are telus fibre or shaw broadband or nothing. It's the same for anywhere within a full day's drive from where we live. Telus fibre or slow 90's tech

[TetraSky]

Now I understand why Louis Rossmann said he was paying for a secondary line at his store (that and the fact US internet ain't reliable depending on the location). Crazy how so many stores are on Rogers network. Do they simply offer the best rates to businesses or what.

[dalekphalm]

37 minutes ago, TetraSky said:

Now I understand why Louis Rossmann said he was paying for a secondary line at his store (that and the fact US internet ain't reliable depending on the location). Crazy how so many stores are on Rogers network. Do they simply offer the best rates to businesses or what.

In most parts of Canada that Rogers operates hardline internet infrastructure, they’re simply the best choice. 
 

Take the region I live in, which is in Southern Ontario with over half a million population, mostly urban and suburban. 
 

Most places in the city, you can get Gigabit internet via Rogers infrastructure (either directly or via a TPIA ISP (Third Party Internet Access) that uses Rogers last mile. 
 

In the same area, most Bell subscribers are capped at 50 Mbps VDSL. Bell does deploy Pure Fibre connections to customers that can get Gigabit, but the vast majority of residents can’t get that service. 
 

Also in most places in Canada there is an effective Duopoloy. Ontario, for example - most areas have Bell via DSL or Rogers via DOCSIS (and all the other competition uses one of those two incumbent connections). 
 

In areas that Shaw or Cogeco operates, Rogers doesn’t. In areas that Telus operates, Bell doesn’t. There are exceptions to this, but not many. 

 

Any new ISP entrant basically has to use Rogers or Bell (or Shaw or Telus) lines, and to actually lay down new infrastructure is incredibly costly and often blocked by one of the existing incumbents. 

[HenrySalayne]

Beaver strikes again?

https://www.bbc.com/news/world-us-canada-56894828

https://www.howtogeek.com/811604/a-single-beaver-caused-internet-outages-in-canada/

 

[wanderingfool2]

7 hours ago, williamcll said:

just maybe not have a monopoly on the network infrastructure

I think it really depends.

 

I suspect this had to do with one of the backbone routers and maybe a terrible config push that broke the bgp.  Similar to how researchers crashed a bunch of backbone routers in America before by testing a new encrypted bgp or something like that (can't remember the specifics).  It nearly took down even more infrastructure as well.  It's actually quite surprising how unresistant the internet can be.  At best it's a better call for having better regulation in regards to require a certain amount of backup equipment (even if it's enough to make things barely functioning)

 

The issue is if you have too much fragmentation you can end up with a worse state of things.  I think just better regulations in terms of contingencies and profit margins would be better suited.

 

e.g. if they were limited to the amount of money they could make per customer ... so when they say they are raising prices by $5 for network "upgrades" they can't get away with just calling it that while only spending an extra $1 on it, while giving the remaining $4 to share holders.

 

Another reason, cell coverage...if you had 10 cell carriers, vs 2 cell carriers the coverage could be a lot better given you can utilize the frequencies a lot more...and instead of having 10 carriers having each having their cell towers covering all of Canada, you have 2 groups of overlapping towers.  The reason why I go with Fido/Rogers is simply because of the coverage, the "competitors" can't compete.

 

Overall, I think smarter regulations is the way to go, instead of creating more competition. 

[LAwLz]

7 hours ago, williamcll said:

 

Good thing he called Roger and told them that fixing the issue quickly was important.

I am sure Roger didn't think it was a major deal until Francois called them.

They were probably just rolling their thumb and thinking that they could deal with the issue on Monday after having had a relaxing weekend. 

[GoodBytes]

You see, this is what happens when you have competition, it creates too many choices for Canadians which causes headaches, and lowers prices of services which kills job and service quality. 

 

As the CRTC mentioned, competition is bad for Canadians, we should have only 1 choice. The right choice, Rogers Communications.

[Kisai]

7 hours ago, dizmo said:

They don't have a monopoly, there are other providers.

The have a de-facto monopoly on wireless service, that an outage with them, sometimes removes 100% of the wireless in some places outside of the major cities. Freedom mobile was impacted because they use Rogers as a their preferred roaming partner (which means on some devices you'd have to manually select telus or bell to get service, even if you were on freedom.)  7-11 speakout, Fido, and various other MVNO's that use them were completely impacted.

 

Wireline access, they have a near-monopoly on in Ontario, with Teksavvy being impacted. 

 

Here's the thing people outside Canada don't realize.

BC and Alberta are served well by Telus. A bit less so by Shaw communications. Rogers wants to buy Shaw (which owns freedom mobile)

Feedom mobile has zero coverage outside major cities. It's pretty much a MVNO like Bell is in Alberta and BC.

 

When we say there is no competition, it's because each part of canada is serviced by a maximum of TWO carriers. Rogers does not offer wireline internet in BC, Shaw doesn't offer wireline internet in Ontario. 

 

Same with television. Telus or Shaw, no other options in BC. If you buy a DTH satellite dish, you get... Bell (formerly ExpressVu) or Shaw (formerly Starchoice), 

 

Rogers is "at most 50%" on paper, but realistically, it has a near-monopoly on cable services in Ontario, and has most of the wireless spectrum in Canada. The fact that it took down the Toronto 9-11 and Canada-wide interac (debit card network) also shows that in some cases it has contracts/customers that have single-point-of-failures. 

 

Like imagine if this was a cyber attack that happened every two weeks. Like over in BC, the most impact I saw, personally, was that no debit card service. Not that I use debit at all. Credit was just fine.

 

7 hours ago, dizmo said:

However, I think that, since this has happened twice, the Canadian government needs to step in and find a different solution, especially in the banking sector.

Having people unable to access their money is unacceptable.

There better be some regulation, be it interac and 9-11 services must be multi-homed, or government run.

 

Or you know, split up all the carriers so that their wireless, wired and backhaul systems are separate businesses. I don't see that happening, but we see the consequences, no redundancy, low to zero wireless coverage outside major cities, unavailability of advanced internet and wireless services to small cities.

 

I've been saying for years, that the ISP's should be separate from the "Content" services. Our ISP's "vertical integration" has instead resulted in having to pay for services in a bundle you don't want, otherwise you're paying for it anyway and getting none of the benefit. 

 

Is that 5$/mo discount worth getting a wireless and wireline bundle, when both of them will go down if the ISP screws up their routing tables?

 

[LAwLz]

It would be interesting to see a post-mortem on this. 

 

I see a lot of people commenting on how "this would not have happened if X" or "we need to do Y to ensure this doesn't happen again" but without even knowing what the root cause is we don't know if any of the solutions would have prevented this. 

 

To me it seems like a lot of people are just using this incident to push their political agenda even though it might be completely unrelated to the issue at hand. 

[wanderingfool2]

31 minutes ago, Kisai said:

There better be some regulation, be it interac and 9-11 services must be multi-homed, or government run.

 

Or you know, split up all the carriers so that their wireless, wired and backhaul systems are separate businesses. I don't see that happening, but we see the consequences, no redundancy, low to zero wireless coverage outside major cities, unavailability of advanced internet and wireless services to small cities.

 

I've been saying for years, that the ISP's should be separate from the "Content" services. Our ISP's "vertical integration" has instead resulted in having to pay for services in a bundle you don't want, otherwise you're paying for it anyway and getting none of the benefit. 

 

Is that 5$/mo discount worth getting a wireless and wireline bundle, when both of them will go down if the ISP screws up their routing tables?

I don't think there is a need to split up the company...in fact I think splitting up a company that is so heavily "vertically integrated" is a bad policy, because now you double or triple the admin work, while likely still creating a partnership where if one fails the other does as well.

 

It should be regulated as how phone companies were regulated back in the day.  If they wanted to increase the price of the service, they had to effectively get permission to by justifying where the extra fees were coming from.  That way the only way they can raise extra funds would be to sign up more customers.  Less of this whole concept, oh the guy who created Shaw retired and is now getting a massive payout package...btw here is an "upgrade" $5/month charge because we need to keep upgrading our infrastructure.

 

To an extent it is like Netflix.  I wish they managed to maintain their monopoly, but alas now try finding disney stuff on Netflix or other shows.  They are being fragmented across other platforms (and so the net sum is more money for less content)

[leadeater]

1 hour ago, LAwLz said:

I see a lot of people commenting on how "this would not have happened if X" or "we need to do Y to ensure this doesn't happen again" but without even knowing what the root cause is we don't know if any of the solutions would have prevented this. 

All of our major backbone outages have been from a famer with a digger, or other contractor with a digger lol

 

The solution is obviously ban all diggers.

[Kisai]

55 minutes ago, wanderingfool2 said:

I don't think there is a need to split up the company...in fact I think splitting up a company that is so heavily "vertically integrated" is a bad policy, because now you double or triple the admin work, while likely still creating a partnership where if one fails the other does as well.

Nah, it's the reverse. Let's say, it's 2030 and all the Canadian ISP's were broken up

Telus Mobility

Rogers Mobility

Bell Mobility (MTS and Aliant are technically the same company)

Freedom Mobility

Videotron Mobility

Sasktel Mobility

Now we have 6 options, not the two that exist in one place. Going further

Koodoo mobility

Fido Mobility

I'm not sure what Bell's discount MVNO is at the moment.

 

Now we have 9 options.

 

Then we have the wireline options:

Telus Fiber

Telus DSL

Telus Cable

Shaw Cable

Shaw Fiber

Rogers Fiber

Rogers Cable

Bell DSL

Bell Fiber

Shaw Cable

Shaw Fiber

Sasktel

That's at least 12 options before even getting into Teksavvy and other ISP's that don't own the fiber/copper

 

Now imagine that all options can be mix and matched. Then you have the backhaul, which is basically Telus, Shaw, Bell, and Rogers. If you have deep enough pockets, be your own ISP and connect to someone's backhaul. Or you know, use foreign backhaul if you live in a city where the carrier hotel is (Vancouver or Toronto.)

 

Like something that happens in Vancouver is that if you live in one of high-rises downtown, is that you can get super-cheap internet from Shaw or Telus because they have competition from Novus. I live like two blocks from these buildings, and still can't get fiber from Telus either.

 

Novus also offers television, despite owning no broadcast assets.

 

The way things are now, if you live in Vancouver or Toronto, you have options. If you live in Montreal, or Calgary, you might have options. Everywhere else, sucks to be you, pick the one or maybe two options. 

 

55 minutes ago, wanderingfool2 said:

It should be regulated as how phone companies were regulated back in the day.  If they wanted to increase the price of the service, they had to effectively get permission to by justifying where the extra fees were coming from. 

I believe that is still the case for for people who have landline phone service.

 

Here's the problem, and I don't ever see this changing, regulation solves one problem (competition) while not necessarily increasing service, all it does is set minimums. When you allow ISP's to consolidate, that minimum bar never get's increased.

 

Like the only condition I'd ever consider a take over or Shaw by Rogers is if all UBB, Throttling, and other garbage tactics were removed. When one ISP does something bad, they all do it.

 

Look at the fine print for the "unlimited internet" on all the ISP's, On the wireless ISP's, your first 25GB is at 4G/5G speed, and then it drops to 2G speed (512 or 256 or 128kbits (EDGE maximum is 384k)) You can not even watch video at that speed.

 

The lack of competition often get's "oh canada is a big place, our extortionately expensive internet is because we have to support small communities" ... BS. These carriers don't offer services to small communities in the first place.

 

 

[wanderingfool2]

6 hours ago, Kisai said:

Nah, it's the reverse. Let's say, it's 2030 and all the Canadian ISP's were broken up

But realistically that's not exactly how technology currently works

 

All phone calls effectively become voip somewhere along the line now and days (analog is a rare breed)..

Cable has transitioned toward utilizing IP

Internet is effectively the backbone of everything.

 

Cable and internet effectively ride together now-and-days because you need to provide a consistent connection for cable.  Actually in this case it would be a failure on Telus or failure on Shaw's end and you would lose your television.  On top of that you introduce any routing issues that exist between Telus and Shaw (yes, it happens.  I've had an ISP vendor who for close to 3 months had to route all traffic to verizon through east coast Canada....so from Vancouver to Washington I was getting like 100ms latency and inconsistent speeds.)

 

Actually a good example with this is all the horrors I've had with Primus over the years; and how it's "telus' fault" or "something on Telus' end"  (and then Primus replaces the modem for a new one and it works).

 

So for decoupling cable from internet now you are very much so adding complexities that wouldn't otherwise exist.

 

Also, the more random carriers you have the less frequency you have to share as well...so that means slower speeds and in busy areas when everyone is on a same carrier you get a lot more interference.

 

Then you get the pole access issues.  Telus has primarily all the fibre lines and Shaw/Rogers all the coaxial lines.  So the majority of other ISP's rent parts of the system.  While I can't talk too much about it, at least one of the third party ISP's that had their own sort of backbone went near bankrupt and has had multiple outages that literally nearly had us drop them as clients.

 

6 hours ago, Kisai said:

Like something that happens in Vancouver is that if you live in one of high-rises downtown, is that you can get super-cheap internet from Shaw or Telus because they have competition from Novus. I live like two blocks from these buildings, and still can't get fiber from Telus either.

And in some of those buildings you can't get Telus fiber because Novus signed a multi-year agreement when it was built (or being refurbished).  It's how Novus manages to compete, they only build out to areas that are dense enough they can actually benefit from having a high-rise wired.  Also, I'm in an area where Telus and Shaw only exist, yet I get those "super-cheap" internet packages from Telus and Shaw as well...comparing to a co-worker who was bragging about it one (I managed to get a rate that was $10/month lower)

 

I don't think splitting up makes sense really at all.  Also, it's important to see the RFO for this.  Let's say it's a bad configuration, well that could happen to smaller companies as well (and I'd argue would happen more frequently)

[leadeater]

22 minutes ago, wanderingfool2 said:

Also, the more random carriers you have the less frequency you have to share as well...so that means slower speeds and in busy areas when everyone is on a same carrier you get a lot more interference.

The solution there is to stop using cable 

 

haha ok, but anyway it really is, but just have to find those billions of dollars to do it. Won't be done by 2030 that's for sure, or even started for that matter.

[wanderingfool2]

2 minutes ago, leadeater said:

The solution there is to stop using cable

Well for that bit I was referring to cell carriers.  If lets say you have 16 carriers who you want to give equal opportunity, then you auction of the cell frequencies evenly...but then that creates the issue that if one town has only Fido and another town has Telus, they now have less bands to operate all that traffic in.

 

Anyways, apparently the RFO for this was

https://www.cp24.com/news/some-ontario-residents-still-left-without-service-after-rogers-outage-1.5981251

Quote

 “We've narrowed the cause to a network system failure following a maintenance update that we did late Thursday evening, early Friday morning,” Staffieri said in a statement.

“That update caused some of the routers in our system to malfunction and that malfunction caused traffic overload. And as a result of that, the whole system just shuts down,” he continued.

So pretty much a miss-managed system for failure to properly check updates maybe.  Which effectively killed their backbone routers.

[leadeater]

35 minutes ago, wanderingfool2 said:

Well for that bit I was referring to cell carriers.  If lets say you have 16 carriers who you want to give equal opportunity, then you auction of the cell frequencies evenly...but then that creates the issue that if one town has only Fido and another town has Telus, they now have less bands to operate all that traffic in.

5G and LTE Advanced I think has solutions for that sort of problem. Carrier Aggregation and Dynamic Spectrum Sharing, two very different things but both can help a lot.

 

Edit:

Also just turn the cellular networks in to a utility like our fiber UFB, have it operated by the private sector but do not allow those companies to also be retail servicers and then retail cellular providers sell their services on top. No more spectrum fighting 

[wanderingfool2]

32 minutes ago, leadeater said:

Also just turn the cellular networks in to a utility like our fiber UFB, have it operated by the private sector but do not allow those companies to also be retail servicers and then retail cellular providers sell their services on top. No more spectrum fighting 

Well that's a similar thought I had years ago, but at least in Canada/US we have auctioned off a large chunk of the bands already...so can't really create an unified backbone with all the bands available.  So we are stuck with what we have; for better or worse (like it was auctioned off for billions)

 

35 minutes ago, leadeater said:

5G and LTE Advanced I think has solutions for that sort of problem. Carrier Aggregation and Dynamic Spectrum Sharing, two very different things but both can help a lot.

On any given frequency though there will still be a limited amount of bandwidth that it's capable of, the best solution is having that shared bandwidth chopped up into as little amount of pieces as possible...which gets me back to why I think monopolies with tight regulation is better (because then you can effectively get the backbone unified, and use regulation to allow the "competitors" to use the lines).  It's actually what already happens here anyways (just not well regulated for cable/cell phone side).  Land-line phones though are quite regulated (unless it's eased in the last few years).

 

Could be wrong in regards to the frequency bands, and such but I don't think I'm too far off the mark (but never know).

[Beskamir]

16 hours ago, dizmo said:

When it comes to ensuring that retail can still function and people can still buy necessities like food, yes, yes they do.

Except that begging the government to "step in and help" completely ignores all the times when it was the government that created the problem in the first place. Our government illegally froze people's bank accounts for dissent just a few months ago. Giving them any more power is not something anyone should be comfortable with let alone in favor of after that incident.

[Ydfhlx]

22 hours ago, dizmo said:

They don't have a monopoly, there are other providers.

However, I think that, since this has happened twice, the Canadian government needs to step in and find a different solution, especially in the banking sector.

Having people unable to access their money is unacceptable.

What if we just, I don't know, have money printed on paper and distributed physically?

[leadeater]

46 minutes ago, Ydfhlx said:

What if we just, I don't know, have money printed on paper and distributed physically?

Eww, how archaic. Outside of extreme rarity aka been given cash I haven't used that for a decade or more.