High speed link between 2 computers

[RTX 3090]

Hi, I have a NVME SSD in my server and I need a high speed link between it and my PC. I am on a budget (around £30)

I was thinking of fibre optics but 10gb will cost around £100 in total for 2 PC's

Then I remembered USB-C and USB3.0 is 10-20gbps, so is there a cable I could use to link the USB-c from my PC's together.

I was thinking of getting a card like this for both computers https://www.ebay.co.uk/itm/SIIG-PCIe-Express-To-USB-3-0-USB-C-Type-C-Dual-Port-Expansion-Card-Adapter/303821164881

Would this work

 

 

Also is there any USB switch like a network switch which can split my USB 10gbps into 10 1gb ports for more PC's. Thanks

 

[WereCatf]

16 minutes ago, RTX 3090 said:

Then I remembered USB-C and USB3.0 is 10-20gbps

USB 3.0 is 5Gbps. USB-C is just the connector, it does NOT say anything about the speed of the port, so that's irrelevant. You also can't use just plain, old male-to-male USB-cables; you need specifically a bridge-cable. Connecting USB from one PC to another PC directly with a regular cable would most likely burn the USB-ports on both PCs.

 

From a quick googling around it seems USB 3.0 <-> USB 3.0 network only achieves around 1Gbps - 2Gbps speeds in practice, so it'd barely be any better than just regular gigabit Ethernet. I can not find any USB 3.2 bridge-cables at all.

[Eigenvektor]

32 minutes ago, RTX 3090 said:

Also is there any USB switch like a network switch which can split my USB 10gbps into 10 1gb ports for more PC's. Thanks

USB isn't really made for this, especially if you want multiple PCs to be able to communicate with each other at once. Just use regular ethernet and a switch.

 

You don't necessarily need fiber for 10 Gbit, Cat 6 is fast enough, provided you don't need to bridge large distances.

https://en.wikipedia.org/wiki/Category_6_cable

[RobinD]

Removed due to inaccuracy

 

[RobinD]

5 minutes ago, TimsTips said:

Yeah, but you gotta buy the cards for each end. 

Right you are the OP clearly does not have 10GB NIC card or they wouldnt be looking at the card suggested. 

On a budget of £30 i dont see Fibre working from Ebay, most cards im seeing are going at £30 each + and fibre connectors cost abit. 
IMO 2nd hand market in UK is not as vibrant as some countries as business sellers must certify their goods for a year even if selling on ebay ect. Many businesses therefore sell to distro who sell the kit abroad. 

ive removed my comment as it would not come in budget but i have no other suggestions sorry 

 

[RTX 3090]

1 hour ago, James Evens said:

Picked up the five 8 GB/s FibreChannel HBA (if you want shared storage using FC there is a little bit more work involved) cards with cable for 30€.

 

57 minutes ago, TimsTips said:

10gb fiber channel off of ebay is your best bet. 

 

A 10GB card is £20 on eBay each, so cards alone without the SFP module would cost £40

https://www.ebay.co.uk/itm/IBM-Emulex-10Gb-PCIe-Dual-Port-Network-Card-49Y7942/393043446024

 

I would like to try this card but the connector looks different. Can anybody guide me on the module and cable to use

[NineEyeRon]

Can’t you just put the drive in the PC? 

[RTX 3090]

1 hour ago, NineEyeRon said:

Can’t you just put the drive in the PC? 

I have a SSD RAID array and want to link it to my other PC's for video editing

[mariushm]

So basically usb is out .... you can't link computers directly with USB, as it's a  master-slave design ... computer is master, the devices (usb hub, usb device) are "slaves".  You can't have a master-master connection.

 

You may see "usb transfer cables" out there but those are basically two usb devices in the middle of the cable exchanging data between them, there's a chip in the middle of the cable doing the work.

 

The cheapest option would be 10gbps fiber cards and a DAC cable  (or two transceivers + cheap fiber cable) ... you can find 10g fiber cards for cheap (but in the case of the cheapest you may have difficulty finding drivers for Windows 10 or other popular operating systems)

ex 29$ each for mellanox connectx-2 :  https://www.unixplus.com/collections/network-adapters/products/refurbished-mellanox-connectx-2-single-port-10gbe-sfp-mnpa19-xtr

 

example of dac cable (there's sites selling cheaper than this) : https://www.unixplus.com/collections/network-cable/products/refurbished-10g-direct-attach-cable-1-meter-2-meter-3-meter-5-meter

 

example of transceivers (you need two if you plan to use cheaper fiber cable) : https://www.fs.com/de-en/products/11552.html

example of fiber that would work with transceivers above (lc duplex mmf) https://www.fs.com/de-en/products/41730.html

 

 

 

 

[RTX 3090]

56 minutes ago, mariushm said:

So basically usb is out .... you can't link computers directly with USB, as it's a  master-slave design ... computer is master, the devices (usb hub, usb device) are "slaves".  You can't have a master-master connection.

 

You may see "usb transfer cables" out there but those are basically two usb devices in the middle of the cable exchanging data between them, there's a chip in the middle of the cable doing the work.

 

The cheapest option would be 10gbps fiber cards and a DAC cable  (or two transceivers + cheap fiber cable) ... you can find 10g fiber cards for cheap (but in the case of the cheapest you may have difficulty finding drivers for Windows 10 or other popular operating systems)

ex 29$ each for mellanox connectx-2 :  https://www.unixplus.com/collections/network-adapters/products/refurbished-mellanox-connectx-2-single-port-10gbe-sfp-mnpa19-xtr

 

example of dac cable (there's sites selling cheaper than this) : https://www.unixplus.com/collections/network-cable/products/refurbished-10g-direct-attach-cable-1-meter-2-meter-3-meter-5-meter

 

example of transceivers (you need two if you plan to use cheaper fiber cable) : https://www.fs.com/de-en/products/11552.html

example of fiber that would work with transceivers above (lc duplex mmf) https://www.fs.com/de-en/products/41730.html

Hi, I have managed to source a 10GB card for £20 cable for £5 and each SFP module for £3. 

 

Just a bit curious about the HP infiniband thing, will it work out cheaper. 

 

Also could I feed the fibre into a switch and split it into ethernet ports as I have a switch with a SFP slot inside

[RTX 3090]

This is the cable I found : https://www.ebay.co.uk/itm/5m-OM4-Aqua-Fibre-Optic-LC-SC-Duplex-MM-50-125-Patch-LSZH-Cable-007262/372755179454

This is the card I found https://www.ebay.co.uk/itm/IBM-Emulex-10Gb-PCIe-Dual-Port-Network-Card-49Y7942/393043446024

[mariushm]

SFP is good for 1gbps

SFP+ is for 10gbps

IF your switch only has a SFP port, then it will do maximum 1 gbps.

It it has 10gbps, yeah, you can connect your NAS or whatever to switch and have 10gbps between pc and switch, from there each device connected to switch will transfer at up to 1 gbps speed.

 

Not sure you can use Infinityband card directly with a switch. Not familiar with them but I assume you may have to flash a "ethernet mode" firmward on the cards, or configure them in Windows to switch to ethernet mode.

 

[RTX 3090]

8 hours ago, mariushm said:

SFP is good for 1gbps

SFP+ is for 10gbps

IF your switch only has a SFP port, then it will do maximum 1 gbps.

It it has 10gbps, yeah, you can connect your NAS or whatever to switch and have 10gbps between pc and switch, from there each device connected to switch will transfer at up to 1 gbps speed.

 

Hi, thanks for info. Because the cards have two ports each, could I transfer 20gbps if I use both ports at the same time.

 

Also 8gb hardware is significantly cheaper then the 10gb and there is only two gb difference, would it be better to use 8gb instead

 

https://www.ebay.co.uk/itm/HP-8GB-Dual-Port-PCI-E-Fibre-Channel-Card-489191-001/223964587495

https://www.ebay.co.uk/itm/HP-AJ718A-8Gb-SFP-Transceiver-8-Gbps-FC-468508-001/222993065603

https://www.ebay.co.uk/itm/5m-OM4-Aqua-Fibre-Optic-LC-LC-Duplex-MM-50-125-Patch-LSZH-Cable/

[mariushm]

3 hours ago, RTX 3090 said:

Hi, thanks for info. Because the cards have two ports each, could I transfer 20gbps if I use both ports at the same time.

 

Also 8gb hardware is significantly cheaper then the 10gb and there is only two gb difference, would it be better to use 8gb instead

 

https://www.ebay.co.uk/itm/HP-8GB-Dual-Port-PCI-E-Fibre-Channel-Card-489191-001/223964587495

https://www.ebay.co.uk/itm/HP-AJ718A-8Gb-SFP-Transceiver-8-Gbps-FC-468508-001/222993065603

https://www.ebay.co.uk/itm/5m-OM4-Aqua-Fibre-Optic-LC-LC-Duplex-MM-50-125-Patch-LSZH-Cable/

 

You will have to do some research and determine if you can configure that 8 GB Fiber Channel card into "ethernet mode", because by default they're not "ethernet cards" ... think of them more like a SATA controller but which uses Fiber Channel instead of SATA. 

If you can configure them in ethernet mode, you'll have 2 ports which will probably run at 3.2 6.4 gbps or something like that ... because the card is probably gonna use 8:10 encoding (for every 8 bits, 2 bits are error correction information) ... that's why 40 gbps fiber channel cards will only show 32 gbps in ethernet mode (there's an older LTT video about using such fiber channel card)

 

Yes, you can probably run both ports at the same time, but by default they're not both used at same time - I mean if you start a file copy, only one port will be used, and if your drives are fast enough, only one port will be saturated while the other idles. 

There are some standards/technology names for port pairing/teaming but I don't remember their names now... and not all work in regular windows, you may need Windows 2016 or whatever server version is that latest for them to work. 

Even then, some of these technologies keep one thread per port, so you'll only see both ports used if you copy two files in parallel, each file using a separate port.

 

I'd say it's not worth the hassle, for just 10 euro or so extra.  Stick with a card that's more popular and has much wider support and for which you can find guides and tutorial out there. 

 

edit :  my math was bad, not 3.2 gbps, 6.4 gbps.  8 gbps / 10 * 8 = 6.4 for 8:10 encoding

[RTX 3090]

22 minutes ago, mariushm said:

You will have to do some research and determine if you can configure that 8 GB Fiber Channel card into "ethernet mode", because by default they're not "ethernet cards" ... think of them more like a SATA controller but which uses Fiber Channel instead of SATA. 

If you can configure them in ethernet mode, you'll have 2 ports which will probably run at 3.2 gbps or something like that ... because the card is probably gonna use 8:10 encoding (for every 8 bits, 2 bits are error correction information) ... that's why 40 gbps fiber channel cards will only show 32 gbps in ethernet mode (there's an older LTT video about using such fiber channel card)

 

Yes, you can probably run both ports at the same time, but by default they're not both used at same time - I mean if you start a file copy, only one port will be used, and if your drives are fast enough, only one port will be saturated while the other idles. 

There are some standards/technology names for port pairing/teaming but I don't remember their names now... and not all work in regular windows, you may need Windows 2016 or whatever server version is that latest for them to work. 

Even then, some of these technologies keep one thread per port, so you'll only see both ports used if you copy two files in parallel, each file using a separate port.

 

I'd say it's not worth the hassle, for just 10 euro or so extra.  Stick with a card that's more popular and has much wider support and for which you can find guides and tutorial out there. 

Hi, thanks for the info, greatly appreciated

 

Also wanted to ask one more question. I am helping a non-profit with their networking, right now they have 100mb switches and they are using it to share a SMB drive across the network too. I have bought them new 1GB switches and it is a lot faster, but the problem is there is a 1GB link between the server, which means if 1 PC is using the drive, it slows down for the next person. So I wanted to ask what fibre link would be best for that scenario. They have a HP server, with 5 250GB SSD's in RAID. So how can I provide more bandwidth to the switch. Should I add extra ethernet cables as the server has 4 ports ethernet, or should I get them a SFP switch and then use a fibre line between the PC and the server. Also you were saying it uses a SATA like mode by default, and it would need to be reconfigured to be a ethernet like card. I found this info 

Quote

Ethernet over Infiniband

Ethernet over InfiniBand, abbreviated to EoIB, is an Ethernet implementation over the InfiniBand protocol and connector technology. EoIB enables multiple Ethernet bandwidths varying on the InfiniBand (IB) version

Quote

QSFP connectors are used. The InfiniBand Association also specified the CXP connector system for speeds up to 120 Gbit/s over copper, active optical cables, and optical transceivers using parallel multi-mode fiber cables with 24-fiber MPO connectors.

I have found a cheap 40gbps InfiniBand card, and this cable which convert InfiniBand to SFP, so can I plug it into the switch to provide it a 40gbps link. 

 

I found this info about InfiniBand and ethernet 

Quote

How is InfiniBand different than 10 Gigabit Ethernet?

10 Gigabit Ethernet is another incremental upgrade to the Ethernet protocol, and it works with TCP/IP in exactly the manner you’d expect. You can run IP over InfiniBand, but that isn’t what it is designed for—you lose the advantages of InfiniBand’s Remote Direct Memory Access (RDMA). I knew before ordering my InfiniBand cards that this might cost me some performance, but I didn’t know how much.

I can run iSCSI over RDMA, and these InfiniBand cards should be fast enough that I wouldn’t be able to tell if an SSD were plugged into the local machine or the server on the other side of the InfiniBand cable.

Even though I don’t currently have a need for iSCSI, I did initially plan to test its performance. However, you need to install all sorts of third-party OFED packages to make use of iSCSI. I don’t need any of those packages to use IP over InfiniBand (IPoIB), so I didn’t think it was worthwhile to pollute my desktop and homelab server with extra cruft.

Quote

IPoIB (IP-over-InfiniBand) is a protocol that defines how to send IP packets over IB; and for example Linux has an "ib_ipoib" driver that implements this protocol. This driver creates a network interface for each InfiniBand port on the system, which makes an HCA act like an ordinary NIC.

IPoIB does not make full use of the HCAs capabilities; network traffic goes through the normal IP stack, which means a system call is required for every message and the host CPU must handle breaking data up into packets, etc. However it does mean that applications that use normal IP sockets will work on top of the full speed of the IB link (although the CPU will probably not be able to run the IP stack fast enough to use a 32 Gb/sec QDR IB link).

Does this mean I can use ipoib but lose a bit of performance. I don't mind losing a bit of performance as even 20gb  is better then what we have rn. 

[mariushm]

59 minutes ago, RTX 3090 said:

Hi, thanks for the info, greatly appreciated

 

Also wanted to ask one more question. I am helping a non-profit with their networking, right now they have 100mb switches and they are using it to share a SMB drive across the network too. I have bought them new 1GB switches and it is a lot faster, but the problem is there is a 1GB link between the server, which means if 1 PC is using the drive, it slows down for the next person.

So I wanted to ask what fibre link would be best for that scenario. They have a HP server, with 5 250GB SSD's in RAID. So how can I provide more bandwidth to the switch. Should I add extra ethernet cables as the server has 4 ports ethernet, or should I get them a SFP switch and then use a fibre line between the PC and the server.

 

If two users copy files from the server, the speed of both should even out to half over time (a few seconds of transfer) - it won't be one user hugging 900 mbps and the other 100 mbps... both users would get around 500 mbps. 

 

You should have bought a switch with gigabit ports and a few SFP+ ports, which allow up to 10gbps.  Then, connect the NAS/server to the switch using a 10gbps connection (either using a DAC cable or a 2 transceivers + fiber  or optical-rj45 transceiver and cat6 cable to 10g rj45 on server) 

The server has 10 gbps connection to switch, and from there, each user has 1 gbps to switch, so each user can transfer at up to around 120 MB/s or 1 gbps. 

 

 

Quote

 

Also you were saying it uses a SATA like mode by default, and it would need to be reconfigured to be a ethernet like card. I found this info 

I have found a cheap 40gbps InfiniBand card, and this cable which convert InfiniBand to SFP, so can I plug it into the switch to provide it a 40gbps link. 

I found this info about InfiniBand and ethernet 

Does this mean I can use ipoib but lose a bit of performance. I don't mind losing a bit of performance as even 20gb  is better then what we have rn. 

Again ... it's SFP  for 1 gbps, and it's SFP+  for 10 gbps  - they're DIFFERENT.  A switch with SFP ports won't work at 10gbps, it will be limited to 1 gbps. 

 

That cable is probably Infiniband to 4 x SFP+ ... it would probably show up as 4  x 10 gbps or 4 x 8 gbps connections.  IF the switch supports QSFP , then the switch may group the 4 SFP+ ports into a single 32/40 gbps connection. 

 

EDIT :  The cable doesn't CONVERT anything ... the card has to format the signals into the QSFP format or whatever format is required, and then the cable just splits the packets into the 4 separate SFP+ transceivers. You get cheap card but expensive cable and compatibility issues. For what... I doubt you need more than 10g (1.2 GB/s transfer speed)

 

 

You're trying to save a few euro. Do you realize how stupid this sounds?  You're buying hardware that will be in use for YEARS, so we're talking about CENTS per month.

Buy a regular 10g SFP+ ethernet card, or spend 70-100$ on a RJ45 ethernet card - you can use rj45 to SFP+ transceivers if you HAVE TO, if the switch has only SFP+.

 

 

149$ gives you a 48 x 1 g + 4 x SFP+ 10g ports switch - it's 99$ without the 4 x 10g ports : https://www.unixplus.com/collections/network-switches/products/force-10-48-port-gigabit-ethernet-switch-dual-port-10gb-uplink-s60-44t-ac-r

 

Can probably find a NEW switch with 1 or 2 10g SFP+ ports at around $150-200 - see Microtik switches or qnap or ubiquity

 

180$  8x1g + 4 sfp+ : https://www.amazon.com/QNAP-QSW-M408S-Managed-Switch-Gigabit/dp/B08DJWZ6VJ/

178$ 24x1g + 2 sfp+ : https://www.amazon.com/Mikrotik-CSS326-24G-2S-RM-Gigabit-Ethernet/dp/B0723DT6MN/