Fast AF Networking Not Need Just Because

[Bmoney]

Server 

Hp xw8600

2x x5430 Xeon

28GB ddr2 ECC ram

Dual port 10Gbe network card

LSI 16e 6gpbs SAS/Sata card (HBA)

Nvidia 710GT (Basically headless server)

Boot Drive WD 250GB SSD

Storage Drives 7x 4TB Drive WD/HGST in raidz2 (estimated transfer speed 380MB/s)

(Looking to add 13x more drives in a full rack mount case which will make estimated write of 1.0GB/s)

 

I want to explore Infiniband 40Gbps but know the limits are in my storage drives. I like getting second hand equipment so all new SSD array is not an option. Yes Yes I know why do you need to go that fast? Because it is fun. How do you break the HDD bottlenecks. 

[Stormseeker9]

You break the HDD bottleneck, By second hand SSD’s

[mariushm]

Buy some more DDR2 memory sticks

Set up a  RAM DISK, use that for testing.

 

Use unraid with a ssd for write caching maybe? ...

 

You'll probably get 32 gbps (up to 4GB/s0 throughput through that infiniband but your CPU - CHIPSET link is only 4GB/s or less, so you'll be limited by chipset and other things anyway.

[Bmoney]

Wow insightful. Was thinking more along the line of zfs caching on a few SSDs. 

 

[Bmoney]

1 minute ago, mariushm said:

Buy some more DDR2 memory sticks

Set up a  RAM DISK, use that for testing.

Not looking for short testing phase looking for a permanent solution for my SAMBA server.

[FloRolf]

Your raidz2 is pretty slow. I'm getting 700/500 on my 6x4Tb WD Red drives. 

 

Honestly though, anything other than nvme on both ends and 40gbit is useless. Or at least 3 SSDs in Raid0, on both ends again. 

[Electronics Wizardy]

1 hour ago, Bmoney said:

Not looking for short testing phase looking for a permanent solution for my SAMBA server.

your system will automatically use ram as a disk cache, so adding more will speed it up

 

What filesystem are you using?

 

How are you using the data? 

 

For zfs, the ssd will only help with random io.

[Mikensan]

ZFS is not designed for speed by any stretch of the imagination. They have workarounds to help with speed, but their focus is data integrity. I would reflash your HBA, throw in a BBU and some RAM, and use it for RAID if you want speed.

 

Well right off the bat your HBA is probably plugged into a x4 slot (or will be if you buy an infiniband) which means you're capped out around 8Gbt/s. So no configuration of disks (even with an SSD on the SATA 2 port on your motherboard) is going to break 10Gbit/s.

 

Your next solution as already mentioned would be RAM disks. You may want to do some performance tests, as I "feel" like you might not hit 40gbit/s with DDR2 / your CPU.

 

Final note is SMB - it is horrible. It will definitely put some strain on your CPU which is already choking. It's certainly a very supported protocol, but it is not easy on your CPU. 

 

*not trying to be a debbie downer, just hoping you don't buy these infiniband cards only to see 0 gain.

[Bmoney]

On 2/4/2019 at 5:38 AM, Mikensan said:

ZFS is not designed for speed by any stretch of the imagination. They have workarounds to help with speed, but their focus is data integrity. I would reflash your HBA, throw in a BBU and some RAM, and use it for RAID if you want speed.

 

Well right off the bat your HBA is probably plugged into a x4 slot (or will be if you buy an infiniband) which means you're capped out around 8Gbt/s. So no configuration of disks (even with an SSD on the SATA 2 port on your motherboard) is going to break 10Gbit/s.

 

Your next solution as already mentioned would be RAM disks. You may want to do some performance tests, as I "feel" like you might not hit 40gbit/s with DDR2 / your CPU.

 

Final note is SMB - it is horrible. It will definitely put some strain on your CPU which is already choking. It's certainly a very supported protocol, but it is not easy on your CPU. 

 

*not trying to be a debbie downer, just hoping you don't buy these infiniband cards only to see 0 gain.

Thanks I thought the CPU and Board might be the problem. Strong for a 1080p plex server but not high speed file transfers or 4k video. End goal to switch to a 1st gen TR now that the prices are falling massively. (50% of retail). Which 32 cores at 4,0GHz will handle no problem. Mean time found some compatible ram on ebay for now going to go from 28g to 64g for $16 no real risk. 

 

Side note no kernel panics on 1g but only on 10 gig for now which does sound like a ram issue.