10 Gbit Network + SSD Sever for Scientific Computing

[neff89]

Hello everyone,

I am a PhD student in structural biology and we are setting up the computing environment of our lab right now. We do not have a great IT support here and have to setup things by our own, so I hope you guys can help me with that a bit

 

First I would shorty like to describe our setup and computational needs:

 

We are dealing with large datasets. One of these datasets is roughly 150-500 GB in size and is composed of 5000 - 20.000 individual files (32 mb each). These files need to be loaded, processed and written in a repetitive manner, using Matlab. The processing can be heavily parallelized and is using both CPU and GPU (CUDA).

 

Currently we have 4 identical workstations, which are setup in a Matlab Cluster to process one dataset in parallel. This processing can take up to one week. In order to use the 4 workstations in parallel, the files need to be stored on a server where all workstations can access the data in parallel. Right now we are using a 1 Gbit network, and actually the loading and writing of the processed files right now is the most time consuming step.

 

This is the reason why we are thinking about setting up a small local server and connect all workstations to this central server by 10 Gbit network. All workstations are already equipped with a 10 Gbit network card.

 

We have these 10 Git network cards: Intel CNA X710-DA2, which are SFP+ Direct Attached Twinaxial Cabling.

 

Here are the other specs of the workstations:

Mainboard: Gigabyte X299 UD4-CF

CPU: i9-7980XE

GPU: Nvidia GTX 1080 Ti

RAM: 128 GB Crucial CT16G4DFD824A

LAN: Intel CNA X710-DA2

PSU: Supermicro PWS-903-PQ

 

I have no idea about 10 Gbit networking and I would really be happy if you guys could give me any suggestions on a 10 Gbit switch and 10 Gbit server for our needs. I guess a total to 2 TB storage  (8x 512GB SSD in raid10) would be enough, because we would only load one active project on the server.

 

We have some budget to invest to this, but the price should be somewhat reasonable. 

 

Thanks a lot 

Nef

[Acedia]

 

1 hour ago, neff89 said:

One of these datasets is roughly 150-500 GB in size and is composed of 5000 - 20.000 individual files (32 mb each).

I am actually not sure if you'd benefit from 10GBit/s at all since the files are so small that your Window size won't increase before they are transfered...At least not as a network file share.

iSCSI might be better tho.

[unijab]

Do you have any data of disk/CPU/memory/network I/O utilization per workstation?

 

 

[mynameisjuan]

1 hour ago, Acedia said:

I am actually not sure if you'd benefit from 10GBit/s at all since the files are so small

Latency. Small burst of data benefit more from low latency. Especially with render farms/folding

[leadeater]

I'll comment later about 10Gb switch recommendations but for lots of smaller files SMB Direct helps a lot. All you need is RDMA support which those Intel X710's have, make sure the file server does too.

https://docs.microsoft.com/en-us/windows-server/storage/file-server/smb-direct

 

SAMBA also has SMB Direct Support if you're using Linux, which I guess is probably the first question I should have asked? Which OS are you using?

[neff89]

Hey, sorry for the late response. Thanks a lot for all your answers. Regarding the small file sizes, I am not sure it matters that much, because in the workflow usually a large portion of the dataset (let's say 1000 x 32 mb file, so roughly 32GB) is loaded at the same time and then processed.

 

We are using centOS7 and the network server is connected by SMB. The server we are connected to is a "Dell EMC Isilon".

 

I will look into SMB Direct and iSCSI.

 

@leadeater It would be really great if you could recommend an 10Gb switch.

 

Thanks a lot 

Steffen

[mynameisjuan]

1 hour ago, neff89 said:

if you could recommend an 10Gb switch.

Whats your price range?

[leadeater]

8 hours ago, neff89 said:

The server we are connected to is a "Dell EMC Isilon"

Nice 

 

8 hours ago, neff89 said:

a large portion of the dataset (let's say 1000 x 32 mb file, so roughly 32GB) is loaded at the same time and then processed.

That's typically where you'd want SMB direct if you cared about getting the files over as fast as possible, not that standard 10Gb over NFS or SMB is that slow anyway. It's not going to effect your processing anyway since you aren't pulling files on the fly. Isilon should support it anyway, unless it's an older one or something.

 

8 hours ago, neff89 said:

@leadeater It would be really great if you could recommend an 10Gb switch.

Where is the switch going to be located? Most of them are loud and designed to be put in a rack far away from people that would get annoyed. There are quieter options just need to know if that's what you need.