10gbps server infrastructure

[gauthiermaxime]

Hi everyone,

 

I am considering adding 10gbps networking to our server infrastructure in order to be able to store virtual machines on a NAS.  I am planning to have everything on the same VLAN.

 

We currently have 1gbps switches and router.  I am being told by my reseller that I would need a 10gbps router to allow communication between the 10gbps and 1gbps interfaces.

 

Since a router would normally bind two different networks together and that the UniFi switch is backwards compatible, I am quite confused as why I would need such additional equipment.

 

Thanks in advance

[Lurick]

The reseller is trying to sell you more expensive gear

Unless you need 10Gbps routed between VLANs (which as you said, you don't) then just 10Gbps on the switch is all you need.

 

Edit:

To add some more detail, if you had an L3 capable switch then you would only need the router for getting out of the network(s) on the switch that aren't connected to that switch as well.

[TechBruhE]

why would you consider a 10gbps network card for the host if you aren't going to put 10gbps distributed in your network and the other access points? its just going to slow down to 1gbps...

[Lurick]

1 minute ago, shawnhovde said:

why would you consider a 10gbps network card for the host if you aren't going to put 10gbps distributed in your network and the other access points? its just going to slow down to 1gbps...

Multiple host access

[Guest]

Yeah I don't see any reason to implement 10Gbit/s at the gateway, given that you're on the same VLAN and not planning on using this outside of the subnet.

[TechBruhE]

2 minutes ago, Windspeed36 said:

Yeah I don't see any reason to implement 10Gbit/s at the gateway, given that you're on the same VLAN and not planning on using this outside of the subnet.

yeah I don't understand  he wants to put a 10g card in the server but not anywhere else? wouldn't the connection have to be 10g everywhere to actually communicate at 10g?

[Lurick]

1 minute ago, shawnhovde said:

yeah I don't understand  he wants to put a 10g card in the server but not anywhere else? wouldn't the connection have to be 10g everywhere to actually communicate at 10g?

Multiple 1Gig clients can access the server at the same time and not be bottlenecked by a 1gig server link.

[Guest]

1 minute ago, shawnhovde said:

yeah I don't understand  he wants to put a 10g card in the server but not anywhere else? wouldn't the connection have to be 10g everywhere to actually communicate at 10g?

If he's got a single or dual port 10Gig switch then it makes sense which I believe he does.

 

He can then have multiple hosts talking to the server while not contending for limited bandwidth.

[TechBruhE]

1 minute ago, Windspeed36 said:

If he's got a single or dual port 10Gig switch then it makes sense which I believe he does.

 

He can then have multiple hosts talking to the server while not contending for limited bandwidth.

he said his switch and routers are 1g connections, so wouldn't that limit it?

[Lurick]

5 minutes ago, shawnhovde said:

he said his switch and routers are 1g connections, so wouldn't that limit it?

I assumed he was going to upgrade the switch as well and the reseller was trying to push a 10gig router on them as well.

If that's not the case and the OP was just going to connect a 10Gb interface to a 1Gb port then, indeed, it would be limited by that port's 1Gb speed.

[TechBruhE]

1 minute ago, Lurick said:

I assumed he was going to upgrade the switch as well and the reseller was trying to push a 10gig router on them as well.

If that's not the case and the OP was just going to connect a 10Gb interface to a 1Gb port then, indeed, it would be limited by that port's 1Gb speed.

ok, I was confused, that's what I thought, hes asking if he should upgrade the switch/router or not too. the switch would be required to have 10g to see any improvement right?

[Lurick]

Just now, shawnhovde said:

ok, I was confused, that's what I thought, hes asking if he should upgrade the switch/router or not too. the switch would be required to have 10g to see any improvement right?

Yes, correct, otherwise there won't be any benefit

[gauthiermaxime]

Thanks for all the very quick replies

 

Here are some precisions on my question.

 

In fact, I just want to be able to use VM on a NAS that provides a connection to the host just as performant as if the VM storage was inside the host (more than SATA 3).  This is to to be able to swap hosts in minutes in the event where the main host would go down (power down host 1, boot up standby host 2 that would share the same VM storage location with host 1). 

 

I don't plan yet to provide 10gbps to the clients, since we're not having bottlenecks on that part of the network.

 

In the proof of concept of my reseller, he tried to connect his 10gbps switch using sfp+ to the rest of his 1gbps network and nothing worked until he connected a 10gbps router.  Note that I have made him swear that both switches were configured on the same subnet and trunked with SFP+.

 

Many thanks for shedding light on that confusing problem.

[Lurick]

1 minute ago, gauthiermaxime said:

Thanks for all the very quick replies

 

Here are some precisions on my question.

 

In fact, I just want to be able to use VM on a NAS that provides a connection to the host just as performant as if the VM storage was inside the host (more than SATA 3).  This is to to be able to swap hosts in minutes in the event where the main host would go down (power down host 1, boot up standby host 2 that would share the same VM storage location with host 1). 

 

I don't plan yet to provide 10gbps to the clients, since we're not having bottlenecks on that part of the network.

 

In the proof of concept of my reseller, he tried to connect his 10gbps switch using sfp+ to the rest of his 1gbps network and nothing worked until he connected a 10gbps router.  Note that I have made him swear that both switches were configured on the same subnet and trunked with SFP+.

 

Many thanks for shedding light on that confusing problem.

What model switches and router if you don't mind my asking?

[gauthiermaxime]

On 06/02/2018 at 7:22 AM, Lurick said:

What model switches and router if you don't mind my asking?

Sorry, for the delay.

 

He made his proof of concept using X540-T2 network adapters, HP JE009A#ABA and JL386A#ABA switches and MicroTik RB3011UiAS-RM router.

 

He tested connecting the adapters in both switches and was not able to get traffic to go through.

 

While looking at the specs, I realized that both are 1Gbps swtiches with SFP (not SFP +) and the adapters are SFP+.  My guess would be that it was not possible to auto-negociate a working speed between the adapters and the switches.

[Lurick]

33 minutes ago, gauthiermaxime said:

Sorry, for the delay.

 

He made his proof of concept using X540-T2 network adapters, HP JE009A#ABA and JL386A#ABA switches and MicroTik RB3011UiAS-RM router.

 

He tested connecting the adapters in both switches and was not able to get traffic to go through.

 

While looking at the specs, I realized that both are 1Gbps swtiches with SFP (not SFP +) and the adapters are SFP+.  My guess would be that it was not possible to auto-negociate a working speed between the adapters and the switches.

That would be correct

[Jay Deah]

What are the specs on the NAS? I really doubt you’ll be pushing 1gbps+ and need this. I’ve run 1gbit iscsi on midrange storage a few times and have never found it to be a practical bottleneck as the mechanical disks don’t have that sort of throughput.

 

Some of my production esxi hosts running 50+ VMs each are only connected via 4gbps fibre channel with no signs of any saturation.

 

just because SATA bus has a 6gbps max rating doesn’t mean anything comes close to hitting it other than pukka SSDs

 

[tscanausa]

My Setup is a bit different but similar enough to hopefully help you out. This works perfectly fine for me

Server:  1x 1GB Nic, 2x 10GB nic
NAS: 4x 1GB Nic, 2x 10GB nic
Switch: unifi-switch-16-150

Server and NAS 10GB are connected via cross over
Server is connected to switch via 1GB

NAS is connected to switch via 4x 1GB

No router needed for 10GB.