Why doesn't LMG use iperf3 to test local network performance?

[Zipdox]

This seems like a no-brainer. Using SMB is a terrible way to measure network performance as it has many bottlenecks such as drive speed and filesystem limitations.

[Kilrah]

10 minutes ago, Zipdox said:

Using SMB is a terrible way to measure network performance as it has many bottlenecks such as drive speed and filesystem limitations.

But it's what they're going to be using the network for, so it's what matters.

[heimdali]

Maybe they don't need to test it.

 

[heimdali]

1 minute ago, Kilrah said:

But it's what they're going to be using the network for, so it's what matters.

Using something else to test could reveal if the network works well to begin with.  Then when a particular protocol doesn't, it gives a pointer to where to look at.

 

[seanondemand]

They usually caveat their tests saying that filesystem limitations are usually what they run into, and drive performance these days is so fast that's it's rarely going to be the bottleneck. Then there's the part where 9 times out of 10 they're testing a storage server, so again, raw network speed isn't what they're looking for - it's how fast their data is actually going, and filesystem and drive limitations are part of that.

 

What drives me crazy is when LTT/anyone uses speedtest.net to test their network/internet speed, because it is wildly inconsistent on internet speed and nearly useless for network speed. 

[heimdali]

8 hours ago, seanondemand said:

They usually caveat their tests saying that filesystem limitations are usually what they run into, and drive performance these days is so fast that's it's rarely going to be the bottleneck. Then there's the part where 9 times out of 10 they're testing a storage server, so again, raw network speed isn't what they're looking for - it's how fast their data is actually going, and filesystem and drive limitations are part of that.

Ok but what does it help them to test only file server access and to wonder why they get maybe 1/20 of the expected performance, searching for days for possible causes, when the underlying network has an issue, like a malfunctioning switch or cable or connector or whater.  Even if the test results meet expectations, they still won't know if it could be better when they can't tell if the network performs as it should or not.  Ultimately, it remains a possibly limiting factor.

 

It's kinda what scients do when making experiments with particle accelerators.  Since the time they get is limited, they sometimes base what they do on results from previous tests instead of doing these pre-experiments themselves, and that is scientifically not valid because they are only assuming that previous results are still valid.  So they don't actually know if their results are true/valid or not because something could have changed in the meantime and things behave differently than they expected.

8 hours ago, seanondemand said:

 

What drives me crazy is when LTT/anyone uses speedtest.net to test their network/internet speed, because it is wildly inconsistent on internet speed and nearly useless for network speed. 

Speedtest.net works fine here and is the only test I have found that yields results which are sufficiently consistent and make sense, letting aside downloading test files.  Besides, "speed" is meaningless, it's bandwidth, latency and jitter.

[Zipdox]

14 hours ago, seanondemand said:

They usually caveat their tests saying that filesystem limitations are usually what they run into, and drive performance these days is so fast that's it's rarely going to be the bottleneck. Then there's the part where 9 times out of 10 they're testing a storage server, so again, raw network speed isn't what they're looking for - it's how fast their data is actually going, and filesystem and drive limitations are part of that.

 

What drives me crazy is when LTT/anyone uses speedtest.net to test their network/internet speed, because it is wildly inconsistent on internet speed and nearly useless for network speed. 

I'm referring mainly to when they test out networking equipment, like the fiber run to the lab and the dish antennas before that.

[Alex Atkin UK]

On 11/21/2022 at 1:39 AM, seanondemand said:

What drives me crazy is when LTT/anyone uses speedtest.net to test their network/internet speed, because it is wildly inconsistent on internet speed and nearly useless for network speed. 

My favourite are the many router reviews where they test the WiFi performance using speedtest.net.  Also reviewing a high-end router on 300Mbit broadband, so you have no clue where it actually tops-out for NAT performance.

 

Likewise testing a high-end GPU with a mid-range CPU.  There are so many "reviews" out there that can't be trusted because its just a random configuration of hardware where any part of it could be bottlenecking because they don't even understand how it all works.

[LAwLz]

4 hours ago, Alex Atkin UK said:

My favourite are the many router reviews where they test the WiFi performance using speedtest.net.  Also reviewing a high-end router on 300Mbit broadband, so you have no clue where it actually tops-out for NAT performance.

 

Likewise testing a high-end GPU with a mid-range CPU.  There are so many "reviews" out there that can't be trusted because its just a random configuration of hardware where any part of it could be bottlenecking because they don't even understand how it all works.

I don't really think either one of those are an issue. You have to be mindful of what is being tested and how when reading reviews, and that goes for all reviews.

For example a high-end GPU paired with a mid-range CPU is a very relevant test for someone who has a mid-range CPU and is thinking of buying a high end GPU. They might be testing that not because they don't understand "how it all works", but rather because it is more relevant to some group of people.

 

It's a bit harder to justify using speedtest.net with 300Mbps upload speed though.

[Alex Atkin UK]

10 hours ago, LAwLz said:

I don't really think either one of those are an issue. You have to be mindful of what is being tested and how when reading reviews, and that goes for all reviews.

For example a high-end GPU paired with a mid-range CPU is a very relevant test for someone who has a mid-range CPU and is thinking of buying a high end GPU. They might be testing that not because they don't understand "how it all works", but rather because it is more relevant to some group of people.

 

It's a bit harder to justify using speedtest.net with 300Mbps upload speed though.

I disagree, because they do not mention that their CPU might be limiting performance.  Yes its useful to know at which point upgrading your GPU wont be useful because of your CPU, but to do that you really need to test across multiple CPUs rather than "were testing this one specific combination" which is useful to very few people unless you have that EXACT combination.  It doesn't tell you if you might be better of getting a new CPU instead, or get a slightly lower GPU, its just plain not useful.

[heimdali]

1 minute ago, Alex Atkin UK said:

It doesn't tell you if you might be better of getting a new CPU instead, or get a slightly lower GPU, its just plain not useful.

How would switching to a worse GPU increase "performance" with the same CPU?

 

[Alex Atkin UK]

2 minutes ago, heimdali said:

How would switching to a worse GPU increase "performance" with the same CPU?

 

You misunderstand, reviews are to decide if its worth upgrading something.  If they review a GPU that is being bottlenecked that can lead to someone buying a GPU more expensive than they needed to.

 

Yes its buyer beware, but that's my point about those sorts of reviews being worthless.

[heimdali]

1 hour ago, Alex Atkin UK said:

You misunderstand, reviews are to decide if its worth upgrading something.  If they review a GPU that is being bottlenecked that can lead to someone buying a GPU more expensive than they needed to.

 

Yes its buyer beware, but that's my point about those sorts of reviews being worthless.

If they review a CPU that doesn't bottleneck a GPU, that may also lead to someone buying a GPU more expensive than they needed to.

 

When the reviewer discloses what has been reviewed (tested) and how, anyone can make the same experiment and will either get the same the results or different ones.  Someone may watch a review for other reasons than the ones you have and the things you find relevant may not be the same things relevant for someone else.

 

There's nothing worthless about the review.  It's your perspective that makes it appear to you as if it is.  Everyone may have a different perspective and that doesn't have anything to do with the review.

 

[Alex Atkin UK]

4 minutes ago, heimdali said:

If they review a CPU that doesn't bottleneck a GPU, that may also lead to someone buying a GPU more expensive than they needed to.

 

When the reviewer discloses what has been reviewed (tested) and how, anyone can make the same experiment and will either get the same the results or different ones.  Someone may watch a review for other reasons than the ones you have and the things you find relevant may not be the same things relevant for someone else.

 

There's nothing worthless about the review.  It's your perspective that makes it appear to you as if it is.  Everyone may have a different perspective and that doesn't have anything to do with the review.

 

That's my point, you can't test just a single configuration and get a meaningful result.  The only way to know if its worth upgrading your GPU is if someone with a similar setup tests multiple GPUs.  The only way to know if the GPU is bottlenecking on the CPU is to test multiple CPUs.  Testing just a single configuration with nothing to compare to is worthless.

 

Same thing as testing high-end routers on 300Mbit is silly, and nobody seems to test on PPPoE which is still widely used in many countries.

[heimdali]

21 minutes ago, Alex Atkin UK said:

That's my point, you can't test just a single configuration and get a meaningful result.  The only way to know if its worth upgrading your GPU is if someone with a similar setup tests multiple GPUs.  The only way to know if the GPU is bottlenecking on the CPU is to test multiple CPUs.  Testing just a single configuration with nothing to compare to is worthless.

If a result is meaningful and how meaningful it is lies solely in the eye of the beholder.  The beholder can always do their own tests.  That doesn't make any of the tests worthless.

21 minutes ago, Alex Atkin UK said:

Same thing as testing high-end routers on 300Mbit is silly, and nobody seems to test on PPPoE which is still widely used in many countries.

That depends on what you're testing.

 

What else is there other than PPPoE?  I don't understand why it's being used in the first place since an ISP has bascially a dedicated line to each customer and doesn't need to identify them because there's noone else who could use it.  Even if someone did it's not the ISPs problem because the line is payed for.  And what are they using instead?

 

[Zipdox]

On 11/25/2022 at 3:15 AM, heimdali said:

What else is there other than PPPoE?  I don't understand why it's being used in the first place since an ISP has bascially a dedicated line to each customer and doesn't need to identify them because there's noone else who could use it.  Even if someone did it's not the ISPs problem because the line is payed for.  And what are they using instead?

 

Yeah kinda weird. My ISP doesn't require specific credentials for PPPoE, they just require you to put something in, anything will work. Why not just use regular DHCP?

[heimdali]

44 minutes ago, Zipdox said:

Yeah kinda weird. My ISP doesn't require specific credentials or PPPoE, they just require you to put something in, anything will work. Why not just use regular DHCP?

Well, what are they using?  Putting something in could mean sending a DHCP request.

 

[Zipdox]

9 hours ago, heimdali said:

Well, what are they using?  Putting something in could mean sending a DHCP request.

 

My bad, I forgot the letter f. They use PPPoE.

[heimdali]

1 hour ago, Zipdox said:

My bad, I forgot the letter f. They use PPPoE.

Lol, ok, so, darn, I still don't know what else there is

 

Having the MTU reduced because of PPPoE somehow annoys me ...  Then there are other tunnels going through that and it's being reduced even further and sooner or later you can't send anything anymore because it's all swallowed up by some tunnel

[mynameisjuan]

On 11/24/2022 at 9:15 PM, heimdali said:

I don't understand why it's being used in the first place since an ISP has bascially a dedicated line to each customer and doesn't need to identify them because there's noone else who could use it.  Even if someone did it's not the ISPs problem because the line is payed for.  And what are they using instead?

It's commonly used because it's simple to tie into billing systems, gives detailed accounting, easy to manage and saves on address space. It's popular because of this simplicity and flexibility. In the BNG space, DHCP is much more difficult from an accounting standpoint.

 

15 hours ago, heimdali said:

Having the MTU reduced because of PPPoE somehow annoys me ...  Then there are other tunnels going through that and it's being reduced even further and sooner or later you can't send anything anymore because it's all swallowed up by some tunnel

It's standard in the provider space to increase L2 MTU on all core interfaces to the max MTU supported by hardware, typically you'll see a minimum of 9000B. This eliminates the impact to customer traffic due to overhead from tunneling mechanisms. Ex. MPLS, QinQ, PPPoE, etc.

 

You should still expect a L3 MTU of 1500B over PPPoE and if not, then the provider is not following the basic designs.

 

On 11/24/2022 at 9:15 PM, heimdali said:

That depends on what you're testing.

Their main purpose for pointing out test with PPPoE is it is not always done in hardware. This means you may get 950/950mbps on a speed test over a non-PPPoE connection, but on a PPPoE connection you may only see 500/500mbps because it's all being done in software (CPU). Even "high end" consumer routers tend to not have the ability to do PPPoE in hardware.

 

On 11/20/2022 at 12:45 PM, Zipdox said:

This seems like a no-brainer. Using SMB is a terrible way to measure network performance as it has many bottlenecks such as drive speed and filesystem limitations.

I agree that iPerf3 should be tested along with SMB because of hardware reliance, but testing SMB alone can still give decent results. SMB has very little tolerance and if you can achieve certain rates with SMB, then there is a very good chance that any other synthetic or real world test will be on par or exceed SMB's rate.

 

iPerf3 can give you raw throughput, but it doesn't always translate into real world performance because the amount of variables in play. Basic iPerf3 test do not provide much value outside high level testing. It needs to be adjusted to have much value.

[heimdali]

13 hours ago, mynameisjuan said:

It's commonly used because it's simple to tie into billing systems, gives detailed accounting, easy to manage and saves on address space. It's popular because of this simplicity and flexibility. In the BNG space, DHCP is much more difficult from an accounting standpoint.

BNG space?  Why would they need accounting?  I pay so much every month anyway.

13 hours ago, mynameisjuan said:

It's standard in the provider space to increase L2 MTU on all core interfaces to the max MTU supported by hardware, typically you'll see a minimum of 9000B. This eliminates the impact to customer traffic due to overhead from tunneling mechanisms. Ex. MPLS, QinQ, PPPoE, etc.

 

You should still expect a L3 MTU of 1500B over PPPoE and if not, then the provider is not following the basic designs.

Perhaps they increase MTU internally.  For PPPoE I'v never seen anything other than 1492.

13 hours ago, mynameisjuan said:

Their main purpose for pointing out test with PPPoE is it is not always done in hardware. This means you may get 950/950mbps on a speed test over a non-PPPoE connection, but on a PPPoE connection you may only see 500/500mbps because it's all being done in software (CPU). Even "high end" consumer routers tend to not have the ability to do PPPoE in hardware.

Ok but there are a lot more things that need to be tested.  And of course I expect a router that's advertised as 1GB internet capable to be able to do that over PPPoE since there is no other way than that.

 

[Alex Atkin UK]

On 11/25/2022 at 2:15 AM, heimdali said:

What else is there other than PPPoE?  I don't understand why it's being used in the first place since an ISP has bascially a dedicated line to each customer and doesn't need to identify them because there's noone else who could use it.  Even if someone did it's not the ISPs problem because the line is payed for.  And what are they using instead?

 

Because your broadband doesn't terminate at the other end of the line, it gets aggregated at the street cabinet or exchange, sent over the backhaul to your ISP and THEN its terminated using PPPoE.

This is particularly relevant in countries that have ISP competition where the ISP and the physical network are not the same company.  The ISP leases a connection on the backhaul and there needs to be a way to then authenticate back to their network.   PPPoE tends to be used because these ISPs have migrated from the dialup days, over to DSL, maybe then to fibre, so it was easier to use the same method.

[mynameisjuan]

3 hours ago, heimdali said:

BNG space?

Broadband Network Gateway (BNG). In the service provider space, BNG is the access portion on the network. It consist of management and operations for customer delivery. Ex. delivery of addresses, traffic rates, tunneling, etc, via PPPoE or DHCP,  Also it's name does not mean it's just related to the common term of broadband.

 

3 hours ago, heimdali said:

Why would they need accounting?  I pay so much every month anyway.

Accounting consist of far more than just billing. It includes customer traffic usage, logins and uptimes along with logging of assigned IPs which in many regions is required by law.

 

Outside of billing, accounting is a critical component when troubleshooting on the support side.

 

3 hours ago, heimdali said:

Perhaps they increase MTU internally.  For PPPoE I'v never seen anything other than 1492

Yes this is done from the core to the edge. Bare minimum standard design is to increase both L2 MTU at the concentrator to account for the 8 byte PPP header. Because we are an all Juniper shop, all our devices are set to a L2 MTU of 9216 and our customers that are served via PPPoE can use it with L3 MTU of 1500B if their hardware supports it.

 

3 hours ago, heimdali said:

And of course I expect a router that's advertised as 1GB internet capable to be able to do that over PPPoE since there is no other way than that.

That's not how consumer specs are advertised. They will advertise the best throughput that the device can handle and this is typically basic IP routing, NAT and 1500B packets. If you're lucky, they may provide some other benchmarks including any additional packages the device may support (IPS or UTM for example).

 

Many providers still provide access via DHCP or sometimes what may appear to be DHCP but the modem/ONT is handling the PPPoE. There are multiple designs, many of which include custom software for handling their access. But to say "internet capable" must include PPPoE, that's just not the case.

[heimdali]

Even over 20 years ago when internet access was though telephone lines and modems, there was PPPoE.  And I have never seen anything else being used for it.  If they just advertise theoretical throughput that may occur in the best case without saying so, the advertising is misleading.

 

[wseaton]

I learned a hard lesson on this decades ago when it comes to benchmark tests. None of it means squat unless you are using the specific protocol / application load that the gear will be running to pay the bills. RDP sessions vs H264/H265 streams vs SQL clients vs large file copy or VM backups all have different limitations and data streams. 

 

It's like testing your SAN to see how well your core switch handles jumbo frames when you won't be running jumbo frames.

 

SMB is fat, bloated protocol, and it's great to test network performance because it's a worse case scenario. I've seen this too many times. MSP comes in, throws down a WiFi extension, their survey tool says everything is great, and it's crap. I then come in with my laptop and my smartphone and do SMB copies while walking all over the venue. Phones and laptop each having their own hardware based strengths and weaknesses in regards to wifi connections. If I drop packets it's broke. Don't care what your survey tool says. 

 

Lost count at the number of times I've used a basic ping command with the -l extension to send the max packet size and seen transmit fails while the 'approved' survey tool sez everything is ok because it's sending data in nice, small packets.

 

You really can't compare internet transmission vs local LANs. The former has too much packet shaping going on to squeeze every bit out of pier network connections you have no control of while a local LAN is usually pretty open and will swallow anything thrown at it.