How do ISPs limit speed?

[Mornincupofhate]

So I've been wondering this question for a while, how do providers limit the speed of the plan that they're on? Most people would just think traffic shaping, but that could be bypassed by an encrypted tunnel. So, anyone know how they do it?

[zMeul]

does your router have QoS? think of it in a similar manner

[Mornincupofhate]

2 minutes ago, zMeul said:

does your router have QoS? think of it in a similar manner

Can't QoS be bypassed with encryption, though?

[zMeul]

Just now, Mornincupofhate said:

Can't QoS be bypassed with encryption, though?

no

not that I know of

[Skiiwee29]

I work for an ISP and how we do it is 1 of 2 ways. If you have a cable modem, we have a specified maximum speed allowed through a .cm file that the modem downloads for its provisioning. if its fiber or a metro ethernet device, then we will do a policing on the bandwidth available to a given port. 

[DarkBlade2117]

10 minutes ago, legacy99 said:

I work for an ISP and how we do it is 1 of 2 ways. If you have a cable modem, we have a specified maximum speed allowed through a .cm file that the modem downloads for its provisioning. if its fiber or a metro ethernet device, then we will do a policing on the bandwidth available to a given port. 

And would so ISP worker guy know how to bypass this <3

lol jk

[Skiiwee29]

2 minutes ago, DarkBlade2117 said:

And would so ISP worker guy know how to bypass this <3

lol jk

You cant unfortunately. When your cable modem reaches out to the CMTS, or Cable Modem Termination System, it has a preconfigured destination to reach out to download the .cm file. Without this .cm file, the modem will come online, but in a denied state and not allowed to pass any traffic. Similarly with the fiber or metro Ethernet circuits, the only way to get around it is if you can log into the provider equipment that you plug into and change the Service Policy set. 

[tt2468]

1 hour ago, legacy99 said:

You cant unfortunately. When your cable modem reaches out to the CMTS, or Cable Modem Termination System, it has a preconfigured destination to reach out to download the .cm file. Without this .cm file, the modem will come online, but in a denied state and not allowed to pass any traffic. Similarly with the fiber or metro Ethernet circuits, the only way to get around it is if you can log into the provider equipment that you plug into and change the Service Policy set. 

Though the .cm file can be bypassed with some hardware hacking. Not saying I get tons of free bandwidth though, as that would be against forum rules.

[RadiatingLight]

encryption doesn't do anything to the max bandwith because no matter how you encrypt that data, it will still be sent out in packets. monitoring the amount of packets = monitoring speed

[brwainer]

At the low level, regardless of the device (modem, ONT, or upstream carrier equipment) a bandwidth limit is applied by dropping packets over a certain rate. It's just like how a switch drops packets if the outgoing buffer for a port is full. Client applications have to detect the dropped packets and adjust their sending rate accordingly.

[leadeater]

4 hours ago, brwainer said:

At the low level, regardless of the device (modem, ONT, or upstream carrier equipment) a bandwidth limit is applied by dropping packets over a certain rate. It's just like how a switch drops packets if the outgoing buffer for a port is full. Client applications have to detect the dropped packets and adjust their sending rate accordingly.

It still surprises me that we still don't have a better way to do bandwidth QoS now days, dropping packets is such a brute force way to do it. There are other tweaks and adjustments that can be made etc but when it comes down to it packets will get dropped to force endpoints to adjust how fast they are sending and TCP Windows etc.

 

In the ye'oldie days of Frame Relay/ISDN/ATM you could actually set signaling rates to control bandwidth along with many other settings at the DCE so when a DTE connects it knows all the limits and parameters of the link and will abide by them. CIR, EIR, BC, BE, DE, FECN, BECN these are all things that are part of Frame Relay that are actually extremely good.

 

Admittedly I was at the very tail end of Frame Relay and was covered in my CCNA course and labs but I've never actually used it out in the wild, seems like some of those ideas should be implemented in Ethernet though.

[brwainer]

4 hours ago, leadeater said:

It still surprises me that we still don't have a better way to do bandwidth QoS now days, dropping packets is such a brute force way to do it. There are other tweaks and adjustments that can be made etc but when it comes down to it packets will get dropped to force endpoints to adjust how fast they are sending and TCP Windows etc.

 

In the ye'oldie days of Frame Relay/ISDN/ATM you could actually set signaling rates to control bandwidth along with many other settings at the DCE so when a DTE connects it knows all the limits and parameters of the link and will abide by them. CIR, EIR, BC, BE, DE, FECN, BECN these are all things that are part of Frame Relay that are actually extremely good.

 

Admittedly I was at the very tail end of Frame Relay and was covered in my CCNA course and labs but I've never actually used it out in the wild, seems like some of those ideas should be implemented in Ethernet though.

While providing client devices information on their max upload rate would help out preventing dropped packets within the ISP's network, as soon as you have to be routed to another ISP you're likely to be seeing dropped packets again due to a congested link. And now you have the modem or possibly the router aware of the upload limit, but the only way it can abide by that is to drop packets on its own when clients are sending too much.

 

I thought I learned in college that there are ICMP messages that backbone routers can send to endpoint devices to inform them of congestion and request they lower their sending rate, but I might be wrong.

[leadeater]

43 minutes ago, brwainer said:

While providing client devices information on their max upload rate would help out preventing dropped packets within the ISP's network, as soon as you have to be routed to another ISP you're likely to be seeing dropped packets again due to a congested link. And now you have the modem or possibly the router aware of the upload limit, but the only way it can abide by that is to drop packets on its own when clients are sending too much.

 

I thought I learned in college that there are ICMP messages that backbone routers can send to endpoint devices to inform them of congestion and request they lower their sending rate, but I might be wrong.

Yea it's called flow control but its no where near as good as Frame Relay, it's just a message saying hey stop sending me stuff for random offset time. In Frame Relay the information follows the frame all the way through the network, you can I would imagine be able to re-encapsulate frames when going between different backbone providers and on how much bandwidth the ISP has purchased but it is a much smarter system and it needed to be, bandwidth back then was just sooo much less.

 

Rather than re-writing it worse than someone else has already done.

Quote

The Frame Relay network uses a simplified protocol at each switching node. It achieves simplicity by omitting link-by-link flow-control. As a result, the offered load has largely determined the performance of Frame Relay networks. When offered load is high, due to the bursts in some services, temporary overload at some Frame Relay nodes causes a collapse in network throughput. Therefore, Frame Relay networks require some effective mechanisms to control the congestion.

Congestion control in Frame Relay networks includes the following elements:

1. Admission Control. This provides the principal mechanism used in Frame Relay to ensure the guarantee of resource requirement once accepted. It also serves generally to achieve high network performance. The network decides whether to accept a new connection request, based on the relation of the requested traffic descriptor and the network's residual capacity. The traffic descriptor consists of a set of parameters communicated to the switching nodes at call set-up time or at service-subscription time, and which characterizes the connection's statistical properties. The traffic descriptor consists of three elements:
2. Committed Information Rate (CIR). The average rate (in bit/s) at which the network guarantees to transfer information units over a measurement interval T. This T interval is defined as: T = Bc/CIR.
3. Committed Burst Size (BC). The maximum number of information units transmittable during the interval T.
4. Excess Burst Size (BE). The maximum number of uncommitted information units (in bits) that the network will attempt to carry during the interval.

Once the network has established a connection, the edge node of the Frame Relay network must monitor the connection's traffic flow to ensure that the actual usage of network resources does not exceed this specification. Frame Relay defines some restrictions on the user's information rate. It allows the network to enforce the end user's information rate and discard information when the subscribed access rate is exceeded.

Explicit congestion notification is proposed as the congestion avoidance policy. It tries to keep the network operating at its desired equilibrium point so that a certain quality of service (QoS) for the network can be met. To do so, special congestion control bits have been incorporated into the address field of the Frame Relay: FECN and BECN. The basic idea is to avoid data accumulation inside the network.

FECN means forward explicit congestion notification. The FECN bit can be set to 1 to indicate that congestion was experienced in the direction of the frame transmission, so it informs the destination that congestion has occurred. BECN means backwards explicit congestion notification. The BECN bit can be set to 1 to indicate that congestion was experienced in the network in the direction opposite of the frame transmission, so it informs the sender that congestion has occurred.

https://en.wikipedia.org/wiki/Frame_Relay#Congestion_control

 

One of the key differences here is the available bandwidth and current bandwidth usage is known and the congestion controls are time controlled and not just frame dropping, but it will also drop frames if it has to. Ethernet can only drop frames or send a flow control pause request, there is no proper way to set CIR/BC. You'll also notice at lot of IT people talk about CIR rates for business internet connections, it isn't actually a thing anymore it's just a throw back to Frame Relay era that has just stuck as an industry term for guaranteed bandwidth.

 

DiffServ/DSCP is sort of similar to Frame Relay congestion controls but all it really does it tag packets with the QoS value and routers/switches use that information to put them in to predefined traffic queues which is used to drop packets based on these queues, lowest priority to highest, but only as much as it needs to so only one queue might actually end up dropping packets.

[GR8-Ride]

I think it's important to note, however, that very little pure "Ethernet" traffic exists on your ISP's network, or the internet in general.   The backbone is primarily MPLS, which has multiple levels of QoS / CoS / ToS capabilities in place.

 

The idea behind Frame Relay was to implement packet-switched data transport capabilities onto a circuit switched network.   The problem is, traditional circuit switched networks are not terribly resilient (ie, a break in the path results in a network-down condition), nor are they terribly adaptable to changing traffic flows.   Despite all of it's QoS capabilities, Frame Relay was never very good at transporting voice or video streams.

 

ATM was an attempt to fix much of what was wrong with Frame Relay, while also introducing network re-routing capabilities (via SVCs and Soft PVCs, for example) and packet based switching (vs circuit based, such as Frame Relay).

 

Frame Relay still relies heavily upon the establishment of PVCs (permanent virtual circuits).  So while the end-user might only get charged for the number of frames actually transported, the underlying link itself was essentially still a permanent circuit. 

 

MPLS VPNs can be configured with very similar QoS parameters similar to those of ATM or Frame Relay (ie, guaranteed pipe method, similar to a PVC).  The difference is, with MPLS VPNs you can have multiple paths to get to the destination, thus allowing consistent throughput even in the event of a network outage affecting a single path.

 

 

 

Patrick

[leadeater]

5 hours ago, GR8-Ride said:

I think it's important to note, however, that very little pure "Ethernet" traffic exists on your ISP's network, or the internet in general.   The backbone is primarily MPLS, which has multiple levels of QoS / CoS / ToS capabilities in place.

 

The idea behind Frame Relay was to implement packet-switched data transport capabilities onto a circuit switched network.   The problem is, traditional circuit switched networks are not terribly resilient (ie, a break in the path results in a network-down condition), nor are they terribly adaptable to changing traffic flows.   Despite all of it's QoS capabilities, Frame Relay was never very good at transporting voice or video streams.

 

ATM was an attempt to fix much of what was wrong with Frame Relay, while also introducing network re-routing capabilities (via SVCs and Soft PVCs, for example) and packet based switching (vs circuit based, such as Frame Relay).

 

Frame Relay still relies heavily upon the establishment of PVCs (permanent virtual circuits).  So while the end-user might only get charged for the number of frames actually transported, the underlying link itself was essentially still a permanent circuit. 

 

MPLS VPNs can be configured with very similar QoS parameters similar to those of ATM or Frame Relay (ie, guaranteed pipe method, similar to a PVC).  The difference is, with MPLS VPNs you can have multiple paths to get to the destination, thus allowing consistent throughput even in the event of a network outage affecting a single path.

 

 

 

Patrick

Sort of forgot about MPLS when writing all of that . It's also something I haven't really had much hands on experience with, only as a customer being provided an endpoint to one. It was also one of those puzzles when I was much earlier in my career about why internet route paths weren't much longer in hops, learned about MPLS and that was like oh right that's why.

 

Far as Ethernet goes or general IP traffic it needs a little bit more love, even with our 10G TOR and 40G distribution we get congestion and while DSCP QoS rules help we also run iSCSI across the common network which I personally believe is stupid but that's something I can't control.

[Lurick]

1 hour ago, leadeater said:

Sort of forgot about MPLS when writing all of that . It's also something I haven't really had much hands on experience with, only as a customer being provided an endpoint to one. It was also one of those puzzles when I was much earlier in my career about why internet route paths weren't much longer in hops, learned about MPLS and that was like oh right that's why.

 

Far as Ethernet goes or general IP traffic it needs a little bit more love, even with our 10G TOR and 40G distribution we get congestion and while DSCP QoS rules help we also run iSCSI across the common network which I personally believe is stupid but that's something I can't control.

Get you some 100Gig ToRs and distribution switches stat!

[leadeater]

34 minutes ago, Lurick said:

Get you some 100Gig ToRs and distribution switches stat!

It's funny since our WAN links are in the middle of being upgraded to 100G but TOR uplinks and datacenter routing/distribution is still only going to be 40G, it's a nice problem to have though.

[Lurick]

Just now, leadeater said:

It's funny since our WAN links are in the middle of being upgraded to 100G but TOR uplinks and datacenter routing/distribution is still only going to be 40G, it's a nice problem to have though.

Yah, and in a few years you'll look back and the front panel ports will all be 40Gig or 100Gig and you'll have 100Tb uplinks

[Wombo]

22 hours ago, Mornincupofhate said:

So I've been wondering this question for a while, how do providers limit the speed of the plan that they're on? Most people would just think traffic shaping, but that could be bypassed by an encrypted tunnel. So, anyone know how they do it?

See Rate Limiting, primarily what is often referred to as Policing.

 

As per wiki;

"In computer networks, rate limiting is used to control the rate of traffic sent or received by a network interface controller. It can be induced by the network protocol stack of the sender due to a received ECN-marked packet and also by the network scheduler of any router along the way."

 

"The recipient of traffic that has been policed will observe packet loss distributed throughout periods when incoming traffic exceeded the contract."