SSD/NVME MultiHost

[latot]

Hi all, I'm looking over some multihost storage, by multihost means the storage can be connected to several motherboards and all of them have access to the data, so I can have one pc with linux, other with windows, and both have access to the same devices.

 

Ideal SSD mode than NVME, to avoid PCIe transfer with GPU.

 

Why multihost? Well, I usually uses a lot of data (maybe not that much, lets put 7TB), but..., is common that I need to access to it from several devices, and with high speeds, so is not ideal do this via web/NAS. But SSD/NVME to replicate the data in all the devices is not great...

 

Someone knows how to do this? If there is the case, this only can be done via PCIe, well that would be the solution.

 

Thx!

[Electronics Wizardy]

You probablay want a nas. Then mount the network share and all the systems can access storage at the same time. Depending on how much performance you want, you can get some pretty high speed network connections for very high speeds.

[latot]

I would like to avoid it...., Well I can build a NAS, but..., before say anything more, what would be the difference in speed? In my check, Wifi 6 handles 75MB/s, while NVME can reach at least 1GB/s, while Ethernet with Cat6 can reach at max 1GB/s. Seems no fast enough.

[latot]

One of the main reasons why multihost, is that I need to be able to access to the data from windows and linux, to be able to backup my data I need windows for the cloud service and linux to work in the data.

 

Actually Sync.com does not have linux support, so I need the data to be able to be accessed all the time, if by accident the NAS is turn off and the windows pc is on, can be a mess with the data, in the worse case I can lost all the data. The othercase would be a NAS with windows....

 

In a similar context, I want to try split workstations, like one pc for server and other for data analysis, this process reads a lot of data from the disks, so I need high speed for read/write, and from more than one pc, because the data can be used by the server and the pc that analyze it....

 

This is Why I try to avoid NAS, multihost seems to be the ideal for this type of implementation.

[Electronics Wizardy]

25 minutes ago, latot said:

I would like to avoid it...., Well I can build a NAS, but..., before say anything more, what would be the difference in speed? In my check, Wifi 6 handles 75MB/s, while NVME can reach at least 1GB/s, while Ethernet with Cat6 can reach at max 1GB/s. Seems no fast enough.

You can get extremly fast ethernet connections. You can get network cards up to 400gbe, or about 10x what a normal nvme ssd can do. If you want more pratical, you can get a nas with 2x 100gbe connection, cables and 2 100gbit nics to put in 2 pcs for sub $1000 dollars. 

 

13 minutes ago, latot said:

One of the main reasons why multihost, is that I need to be able to access to the data from windows and linux, to be able to backup my data I need windows for the cloud service and linux to work in the data.

 

Actually Sync.com does not have linux support, so I need the data to be able to be accessed all the time, if by accident the NAS is turn off and the windows pc is on, can be a mess with the data, in the worse case I can lost all the data. The othercase would be a NAS with windows....

 

In a similar context, I want to try split workstations, like one pc for server and other for data analysis, this process reads a lot of data from the disks, so I need high speed for read/write, and from more than one pc, because the data can be used by the server and the pc that analyze it....

 

This is Why I try to avoid NAS, multihost seems to be the ideal for this type of implementation.

If your using multiple oses you really want to have a nas. While there are multihost nvme enclosures, they need special drives, and need the same os running in a cluster with the other system. Since both systems see one block device, they need to make sure they don't write on the same data used by the other system. 

 

A nas should be able to provide extremely fast performance here and allow multiple users if configured correctly.

[latot]

30 minutes ago, Electronics Wizardy said:

If your using multiple oses you really want to have a nas. While there are multihost nvme enclosures, they need special drives, and need the same os running in a cluster with the other system. Since both systems see one block device, they need to make sure they don't write on the same data used by the other system. 

 

A nas should be able to provide extremely fast performance here and allow multiple users if configured correctly.

Good point.

 

31 minutes ago, Electronics Wizardy said:

You can get extremly fast ethernet connections. You can get network cards up to 400gbe, or about 10x what a normal nvme ssd can do. If you want more pratical, you can get a nas with 2x 100gbe connection, cables and 2 100gbit nics to put in 2 pcs for sub $1000 dollars. 

Mmm, that is great!, Now I'm thinking how to do that, actually for one access, the bottleneck would be... the cable, the fastest cable I found is the Cat8, has 5GB/s, so even if the card can get 400GB/s the cable will limit this, but with 5GB/s is a lot more decent now. In some years can be a bottleneck again, but maybe there is a new ethernet cable.

 

The last issue, would be the software access to the NAS, a lot of software starts with the system, the NAS should be mounted before anything, and in case the connection is lost to the NAS..., still can be difficult to handle.... is not like we design the software, so they don't necessarily support this type of configurations....

[Electronics Wizardy]

10 minutes ago, latot said:

Mmm, that is great!, Now I'm thinking how to do that, actually for one access, the bottleneck would be... the cable, the fastest cable I found is the Cat8, has 5GB/s, so even if the card can get 400GB/s the cable will limit this, but with 5GB/s is a lot more decent now. In some years can be a bottleneck again, but maybe there is a new ethernet cable.

With this networking, your not using CAT cabling, there is fiber cables and dacs that you will be using.  Here is a 5m DAC cable that will work with 100GBE(check with the card your uisng though) https://www.fs.com/products/50483.html?attribute=10068&id=196685

 

I have configured many systems running linux and windows to mount nfs/smb shares automatically and programs start after. If there is issues of programs starting before the network share is up, you can configure a delay, or have it check for the network. Normally these connections are pretty reliable, so I woudln't worry too much about network link failure.

[latot]

Ufff, I feel so old talking about RJ45... I forgot to check new tech for network, maybe is not that new anyway, I'm checking the cables, seems the ideal solution to this, is great!, Where can I found cards of 400GB? I'll need to learn more about networking, seems to be very important for what I does, is not the first time I need to do this type of things.

 

3 hours ago, Electronics Wizardy said:

I have configured many systems running linux and windows to mount nfs/smb shares automatically and programs start after. If there is issues of programs starting before the network share is up, you can configure a delay, or have it check for the network. Normally these connections are pretty reliable, so I woudln't worry too much about network link failure.

That is great!, just to confirm, I'll probable ask to Sync.com about it.

 

I was thinking, is possible..., use a NAS with DAC for high speeds, but also LVM with lines for several drives, LVM has a performance prince, NAS too, is there any tips for an ideal mix of this two things?

[Electronics Wizardy]

8 hours ago, latot said:

 

I was thinking, is possible..., use a NAS with DAC for high speeds, but also LVM with lines for several drives, LVM has a performance prince, NAS too, is there any tips for an ideal mix of this two things?

Id personally run zfs here, cause I'm a fan of zfs. 

 

LVM has a pretty small performance penalty. Your gonna get slower speed that local nvme drives, but really depends on how much your putting into this server.

[latot]

Where can we found the network cards? I'm checking cards that uses DAC or Fiber...

 

18 hours ago, Electronics Wizardy said:

You can get extremly fast ethernet connections. You can get network cards up to 400gbe, or about 10x what a normal nvme ssd can do. If you want more pratical, you can get a nas with 2x 100gbe connection, cables and 2 100gbit nics to put in 2 pcs for sub $1000 dollars. 

I'm checking cards, but are pretty expensive when we want over 100gbe... the total with other pc components can go easily over $1000 dollars.

[Electronics Wizardy]

6 hours ago, latot said:

Where can we found the network cards? I'm checking cards that uses DAC or Fiber...

 

I'm checking cards, but are pretty expensive when we want over 100gbe... the total with other pc components can go easily over $1000 dollars.

I'm checking ebay for the cards. I'd probably stick with 100gbe here as it gets much more pricy beyond that.

[latot]

Like... going back, by the price, maybe multihost is still a good option? but I can't found hardware for that....

[LIGISTX]

10 minutes ago, latot said:

Like... going back, by the price, maybe multihost is still a good option? but I can't found hardware for that....

Your not going to want to do 100 gigabit… that’s 12GB per second, no NAS you’re going to build will be able to read or write that fast anyways. If the price of the NIC is what scares you away, the cost of the machine to move data that quickly is out of the question. 
 

You can very easily and cheaply do 10 gigabit, conectX3 or 4’s are pretty cheap, and the fiber transceivers are also cheap. You can buy 2 cards and 2 transceivers for easily under 100 bucks. 10 gigabit is 1.2 GB/s which is already going to be faster then your system will be able to continually send data at, unless it’s nvme based storage which gets expensive quickly.

 

But, you can likely do this for a couple grand. 

[latot]

It is possible, run two network cards as one?

[LIGISTX]

1 hour ago, latot said:

It is possible, run two network cards as one?

Technically yes there are ways to do this. But… what is your use case here? What is the data you’re trying to edit? Why can’t you just use a nas and some NVME storage in each machine?

 

Use syncthing, instal that on all devices and the NAS, it will keep data the same across all devices, and since the NAS will always be on, even if the other devices are off it will always be available to push the newest data to devices that were off while edits were made. If you truly need NVME speeds… you are not going to get that on an “economically priced” server and network setup. You basically need to just put nvme devices in each system you need that performance on. 

[kokosnh]

8 hours ago, latot said:

It is possible, run two network cards as one?

There is smb 3.0 multipath, but it’s experimental

there the iscsi mpio but as I run it daily on Windows 10, the Windows implementation is just bad, multipath works only on the setup, later when i look at it, it’s broken, and the performance is not even, sometimes I get 1200/900 MBps, sometimes i get 600/600MBps, but well that could be my qnap.    
 

and in iscsi, only one client can be connected to share,  so everyone would have to have their own share.    

  

 

 

   

 

 

 

[latot]

On 2/28/2023 at 6:22 PM, LIGISTX said:

Technically yes there are ways to do this. But… what is your use case here? What is the data you’re trying to edit? Why can’t you just use a nas and some NVME storage in each machine?

Well.. the point is.... The data I use, can be used bu several types of programs, so I need access across several devices, and the data is heavy, maybe instead of heavy dense would be better, as an example, a pc that handles a web server, other for data analysis, and both need fast access to exactly the same data, so... a way to handle the sync from several devices seems... complicated, I want to avoid that, while if the SSD has multihost, all the devices can have exactly the same access, if something changes will change for everyone... Well if two devices try to write in the same file at the same time, I excpect or at least, the result to be closer to two apps doing the same from the same device. For me is fine while the hardware/software has a filesystem uses a lock system? just avoid the sync problem over several edits.

[LIGISTX]

19 minutes ago, latot said:

Well.. the point is.... The data I use, can be used bu several types of programs, so I need access across several devices, and the data is heavy, maybe instead of heavy dense would be better, as an example, a pc that handles a web server, other for data analysis, and both need fast access to exactly the same data, so... a way to handle the sync from several devices seems... complicated, I want to avoid that, while if the SSD has multihost, all the devices can have exactly the same access, if something changes will change for everyone... Well if two devices try to write in the same file at the same time, I excpect or at least, the result to be closer to two apps doing the same from the same device. For me is fine while the hardware/software has a filesystem uses a lock system? just avoid the sync problem over several edits.

Two devices writing the same data at the same time will almost always cause corruption and issues like that. 
 

Can whatever you are running not just all be VM’s on a single machine, and mount the storage via local, virtual network mounts? If you need fast networking and don’t want to pay for fast networking, you can always virtualize everything and allow the VM’s communicate with the storage appliance via virtual NIC’s (that’s how VM’s work unless you provide them with physical PCIe NIC’s), and virtio virtual NIC’s more or less transfer data at the kernel level; data would transfer from VM to VM as fast as RAM can move data… which will instantly make your harddrives or SSD’s the bottleneck. You just need to run everything virtually under a single hypervisor. 

[latot]

19 hours ago, LIGISTX said:

Can whatever you are running not just all be VM’s on a single machine, and mount the storage via local, virtual network mounts? If you need fast networking and don’t want to pay for fast networking, you can always virtualize everything and allow the VM’s communicate with the storage appliance via virtual NIC’s (that’s how VM’s work unless you provide them with physical PCIe NIC’s), and virtio virtual NIC’s more or less transfer data at the kernel level; data would transfer from VM to VM as fast as RAM can move data… which will instantly make your harddrives or SSD’s the bottleneck. You just need to run everything virtually under a single hypervisor. 

Personally, that sounds great, there is just a few hard points.

To run all of VM's, if we want to get more cpu/ram the only option would be at least Threadripper, and go over 128gb of ram, for that the price go over... a lot...

 

But...., the option to can help to get other organization, like two devices with 16 cores, connect them with network cards, and in each of them use VM's, that would be the ideal mix in this case to minimize costs...

[LIGISTX]

21 minutes ago, latot said:

Personally, that sounds great, there is just a few hard points.

To run all of VM's, if we want to get more cpu/ram the only option would be at least Threadripper, and go over 128gb of ram, for that the price go over... a lot...

 

But...., the option to can help to get other organization, like two devices with 16 cores, connect them with network cards, and in each of them use VM's, that would be the ideal mix in this case to minimize costs...

What are the workloads..? What do you need these servers for?

[latot]

Well..., I usually plays analyzing data, and trying to upload to the cloud (which only works on windows).

 

Is hard to me tell the workload, I usually handles too different project, and is common to reach the limit of read/write speeds of disks or ram or use of the CPU.

 

So I try to keep the things most open/flexible possible, but with lowing cost or they will go to the moon.

 

Due to this, and limitated resources, if I try for example, be able to have more space from a NAS, I would try to get at least the bandwith I already have locally, close to it, and i is possible more than that.