Expanding storage without spending much

[porina]

Who would have thought video content creation generates a lot of data? Currently I run two storage servers, one for "live" data, and one for offline backups.

 

The offline backup server is as follows - this contains backups from systems, as well as a copy of the live data server:

Some dual core Intel CPU with ECC ram

Unraid: 7x 3TB drives arranged as 2 parity, 5 data (15GB usable). It had 4 data drives until it almost filled up yesterday, when I used my cold spare as a data drive to give some breathing space. There is enough room for maybe 2 more data drives if I want to go there, but I'm not sure 3TB drives are worth getting new now and at most I'd add one more.

 

The live data server I just partially rebuilt, currently running Win10 off a small SSD, and has a pair of 8TB drives in mirrored configuration (8TB usable). It was previously in a microserver running Server 2008R2 which has been unsupported for a long time, and I was overdue replacing it. I couldn't figure an easy way to install Win10 on it, so just moved it to a different system totally.

 

So to recap, I currently have:

Live data server: 4TB used, 8TB capacity

Backup server: 12TB used, 15GB capacity.

 

Live data server contents are also duplicated onto the backup server. I'm thinking, one easy way I can get more capacity without spending more is simply to break the mirror and use the two 8TB drives as normal drives. Instantly I get 8TB more storage, but I'd have to also upgrade the backup server as that will fill soon. My thinking is, I think I'm happy to accept the risk from not being mirrored, as I can mitigate against drive failure with more frequent backups to the other server.

 

Any other bulk storage solutions I should consider? I notice there's a 14TB USB drive on sale on Amazon for £190, which I'm informed is not SMR so it might not totally suck. I'm just not sure how I'd add a USB drive into my arrangement, or if I want to shuck it. But this is certainly into the spending money territory.

 

Further thought: the microserver used ECC ram, but now I've moved it into a consumer system, it does not. How paranoid should I be? I'm not running ZFS where I think it is more recommended. Are there any low cost ECC solutions? This would at the least contain CPU+mobo+ECC ram. Since it is a file server, it doesn't really care about CPU (dual core would suffice) or ram quantity (4GB+ ok).

[Bombastinator]

35 minutes ago, porina said:

Who would have thought video content creation generates a lot of data? Currently I run two storage servers, one for "live" data, and one for offline backups.

 

The offline backup server is as follows - this contains backups from systems, as well as a copy of the live data server:

Some dual core Intel CPU with ECC ram

Unraid: 7x 3TB drives arranged as 2 parity, 5 data (15GB usable). It had 4 data drives until it almost filled up yesterday, when I used my cold spare as a data drive to give some breathing space. There is enough room for maybe 2 more data drives if I want to go there, but I'm not sure 3TB drives are worth getting new now and at most I'd add one more.

 

The live data server I just partially rebuilt, currently running Win10 off a small SSD, and has a pair of 8TB drives in mirrored configuration (8TB usable). It was previously in a microserver running Server 2008R2 which has been unsupported for a long time, and I was overdue replacing it. I couldn't figure an easy way to install Win10 on it, so just moved it to a different system totally.

 

So to recap, I currently have:

Live data server: 4TB used, 8TB capacity

Backup server: 12TB used, 15GB capacity.

 

Live data server contents are also duplicated onto the backup server. I'm thinking, one easy way I can get more capacity without spending more is simply to break the mirror and use the two 8TB drives as normal drives. Instantly I get 8TB more storage, but I'd have to also upgrade the backup server as that will fill soon. My thinking is, I think I'm happy to accept the risk from not being mirrored, as I can mitigate against drive failure with more frequent backups to the other server.

 

Any other bulk storage solutions I should consider? I notice there's a 14TB USB drive on sale on Amazon for £190, which I'm informed is not SMR so it might not totally suck. I'm just not sure how I'd add a USB drive into my arrangement, or if I want to shuck it. But this is certainly into the spending money territory.

 

Further thought: the microserver used ECC ram, but now I've moved it into a consumer system, it does not. How paranoid should I be? I'm not running ZFS where I think it is more recommended. Are there any low cost ECC solutions? This would at the least contain CPU+mobo+ECC ram. Since it is a file server, it doesn't really care about CPU (dual core would suffice) or ram quantity (4GB+ ok).

TLDNR: vague possibly useless stuff
My first zero cost thought is compression for bulk storage but I don’t know how much that would net you and it may already be in place. 
I’m not up on hardware anymore.  I know a new tape system just came out so there may be old tape solutions floating about cheap.  You don’t need a lot of space as far as tape is concerned.  Some thing obsolete might work fine for you.  Old AMD consumer stuff used to run ecc no problem.  I don’t know about the new stuff. 

[leadeater]

1 hour ago, porina said:

Any other bulk storage solutions I should consider? I notice there's a 14TB USB drive on sale on Amazon for £190, which I'm informed is not SMR so it might not totally suck. I'm just not sure how I'd add a USB drive into my arrangement, or if I want to shuck it. But this is certainly into the spending money territory.

What about 8TB Toshiba or WD Gold enterprise disks off ebay used? They go for about $90-$100 USD and there are thousands of them. Even though they are used odds are they will run for years just fine and you can always buy spares to cover yourself if this is cheap enough for you. Could also look at 6TB which are slightly cheaper.

 

A gradual swap/replacement of the 3TB disks to 6TB/8TB should give you a nice upgrade.

 

1 hour ago, porina said:

Further thought: the microserver used ECC ram, but now I've moved it into a consumer system, it does not. How paranoid should I be? I'm not running ZFS where I think it is more recommended. Are there any low cost ECC solutions? This would at the least contain CPU+mobo+ECC ram. Since it is a file server, it doesn't really care about CPU (dual core would suffice) or ram quantity (4GB+ ok).

No don't bother, you're more likely to win Lotto than to have a ram issue that would cause data corruption that cannot be fixed and would have been prevent by ECC.

[leadeater]

45 minutes ago, Bombastinator said:

My first zero cost thought is compression for bulk storage but I don’t know how much that would net you and it may already be in place. 

Video files don't compress much at all (they are themselves already a compressed data type), unless it's direct raw with zero compression which nobody really uses. Even so if you are wanting to keep raw footage files the last thing you want to do is compress it as it will no longer be raw anymore even when it gets uncompressed again. What is lost during compression cannot be gained back, nature of the beast when it comes to video and pictures unlike general data/document files.

[Bombastinator]

10 minutes ago, leadeater said:

Video files don't compress much at all (they are themselves already a compressed data type), unless it's direct raw with zero compression which nobody really uses. Even so if you are wanting to keep raw footage files the last thing you want to do is compress it as it will no longer be raw anymore even when it gets uncompressed again. What is lost during compression cannot be gained back, nature of the beast when it comes to video and pictures unlike general data/document files.

So already in place. 

[Bombastinator]

Looking at times for this I apparently replied 2 minutes after posting.  This is not my habit.  I was under the impression this was up for a long time before I posted on it.  Something is messed up.  That something is usually me, but there could possibly be another issue. 

[porina]

1 hour ago, Bombastinator said:

TLDNR: vague possibly useless stuff

 

 

1 hour ago, Bombastinator said:

My first zero cost thought is compression for bulk storage but I don’t know how much that would net you and it may already be in place. 

Leadeater's reply sums it up, but out of interest I thought I'd try it anyway on a representative file. 600MB uncompressed on disk. 600MB with Windows built in filesystem compression applied. It actually saved 100k, so totally insignificant.

 

Other forms of compression aren't an option. I doubt a Zip would do much better and it'll take far too much CPU time, and means files wont be directly accessible. Recompression of the data isn't really an option either. It is already at a compromise bitrate to balance quality with size.

 

55 minutes ago, leadeater said:

What about 8TB Toshiba or WD Gold enterprise disks off ebay used? They go for about $90-$100 USD and there are thousands of them.

I just had a look for 6 to 8 TB drives on ebay UK... no such luck here. The lowest priced ones are around £130 and they're one-offs so I can't get in quantity even if I did want to pay for it. Maybe you have someone who just upgrade a datacenter, but the market here is usually small and limited. I might poke about other capacities later but not going to get my hopes up.

 

Still makes me wonder about the 14TB USB drives at £190 new. I think the price-per-TB is going to be hard to beat, especially as they're understood NOT to be SMR.

 

55 minutes ago, leadeater said:

No don't bother, you're more likely to win Lotto than to have a ram issue that would cause data corruption that cannot be fixed and would have been prevent by ECC.

I suppose we should consider "normal" desktops as an example of that. While it is near-impossible to prove any unexpected behaviour would have been prevented by ECC, we just don't get that much unexpected behaviour.

 

In the past I've used HPE (micro)servers which have that as standard, but the server functions are a pain from a consumer viewpoint and I don't want to get more.

[leadeater]

5 hours ago, porina said:

Still makes me wonder about the 14TB USB drives at £190 new. I think the price-per-TB is going to be hard to beat, especially as they're understood NOT to be SMR.

Yep and you are also getting a new drive too so that is a plus, very close cost wise so new is probably better unless you find a killer deal.

[Jarsky]

8 hours ago, porina said:

Unraid: 7x 3TB drives arranged as 2 parity, 5 data (15GB usable). It had 4 data drives until it almost filled up yesterday, when I used my cold spare as a data drive to give some breathing space. There is enough room for maybe 2 more data drives if I want to go there, but I'm not sure 3TB drives are worth getting new now and at most I'd add one more.

Because you have 2 Parity disks, it means the minimum you can start an upgrade with is 3 disks. Upgrade both parity disks (1 at a time) to larger disks, and then add/upgrade one of your existing data drives. The best value to $ right now is 8TB drives. If you're in the US, I believe BestBuy has a special right now for WD Easystore 8TB's which you could shuck drives from. There are UltraDC's & Exo's for a decent price on eBay, or of course theres a range of various drives you can go for brand new, but yeah 8TB is the best value right now. 

 

Quote

Live data server contents are also duplicated onto the backup server. I'm thinking, one easy way I can get more capacity without spending more is simply to break the mirror and use the two 8TB drives as normal drives. Instantly I get 8TB more storage, but I'd have to also upgrade the backup server as that will fill soon. My thinking is, I think I'm happy to accept the risk from not being mirrored, as I can mitigate against drive failure with more frequent backups to the other server.

Personally I wouldn't if its your only backup of the data besides your primary UnRAID. But your data, your choice of risk. 

 

Quote

Any other bulk storage solutions I should consider? I notice there's a 14TB USB drive on sale on Amazon for £190, which I'm informed is not SMR so it might not totally suck. I'm just not sure how I'd add a USB drive into my arrangement, or if I want to shuck it. But this is certainly into the spending money territory.

https://www.bestbuy.com/site/wd-easystore-8tb-external-usb-3-0-hard-drive-black/5792401.p?skuId=5792401&intl=nosplash

This is the drive everybody shucks. It has basically the equivelant to a white labelled HGST He8 drive with TLER disabled. 

 

Quote

Further thought: the microserver used ECC ram, but now I've moved it into a consumer system, it does not. How paranoid should I be? I'm not running ZFS where I think it is more recommended. Are there any low cost ECC solutions? This would at the least contain CPU+mobo+ECC ram. Since it is a file server, it doesn't really care about CPU (dual core would suffice) or ram quantity (4GB+ ok).

You have more chance of a naked homeless man breaking into your house, taking a piss on your server until he fries it. I forget the calculation for a set amount of ram, but the numbers are astronomically low compared to other problems you can face. 

[porina]

1 hour ago, Jarsky said:

Because you have 2 Parity disks, it means the minimum you can start an upgrade with is 3 disks.

I don't want to buy new disks unless I really run out of capacity, or I need to replace existing. I'm aware of the process that unraid has to upgrade drives, and TBH it seems enough of a pain if I ever wanted to have a bigger capacity unraid store, it would be much less pain to pay for another licence and just build a new one from scratch. The endless juggling of drives and resyncs would take forever.

 

One further option I'm thinking of is dropping down to one parity, and transfer the current 2nd parity drive over to data. Just how much backup does my backup need? Again, the unraid only stores a (backup) copy of data that is live elsewhere. Unraid also has the nice feature that if a disk dies, you only lose the data on that disk, not the whole system.

 

1 hour ago, Jarsky said:

Personally I wouldn't if its your only backup of the data besides your primary UnRAID. But your data, your choice of risk. 

To make sure you correctly understand the situation:

1, typical desktop systems: no drive redundancy. Backup to backup server.

2, live file server: formerly mirrored drives. Backup to backup server.

3, backup server only holds backups. Currently two parity drives.

 

I've already gone ahead and broke the mirror. I accept the risk that on desktop systems, there is a window of potential data loss if the storage drive fails before new data is copied to the backup server. I now have the same risk level on the live file server. The mirror only buys me protection from mechanical failure of one of the drives, not other potential sources of data loss.

 

I know best practice is to have a 3rd copy off site, but that's simply not practical in any affordable way for me.

 

1 hour ago, Jarsky said:

I forget the calculation for a set amount of ram, but the numbers are astronomically low compared to other problems you can face. 

That kinda goes against numbers I saw, which I also don't recall the exact numbers to. Now, I am cheating a little here because the basis was that we are cramming ever more ram into our systems. What was rare while we had smaller amounts is starting to grow into something we are more likely to encounter than ever. DDR5 having die level ECC built in may be more for yield/performance reasons, but it could also help out here, so something to look forward to.

[Loki0111]

One of these (LSI 9207-8i):

https://www.ebay.ca/itm/LSI-9207-8i-HP-9205-8i-w-2-CABLES-IT-MODE-PCI-E-3-0-SAS2308-ZFS-UNRAID-TRUENAS/313273295453?hash=item48f08b225d:g:M7sAAOSwk2Je3Sxb

 

With one of these (Intel RES2SV240 SAS Expander)

https://www.ebay.ca/itm/Intel-RES2SV240-24port-6G-6Gbps-SATA-SAS-Expander-Server-Adapter-Gebraucht/263931828166?hash=item3d739017c6:g:M4IAAOSwle1dZ4ik

 

Gives you 16 SATA 3 drive connections at 6Gbps, the Intel SAS Expander does not need to go into a PCIe slot as it can independently powered off a 4 pin connector and just bolted to the side of a case. The PCIe connection only provide power. Do not use the 4 pin power connector if you put in a card slot, its one or the other.

 

Then just use any group of 16 drives, plus whatever the motherboard will take for a large storage cluster.

 

Unraid is okay if we are talking about high speed data access I'd probably go with TrueNAS, OMV, Linux or Windows Server 2016/2019. You can also use Windows 10 Pro but you lose tiered storage, you can cheat around that with Primocache though which will allow you to add L1 (RAM), L2 (NVME/SSD) read/write cache.

 

The issue with Unraid is because its not using any form of raid its largely limited to single disk access speeds. NVMe/SSD caching helps in some situations but everything else also supports caching these days and almost everything else also has faster access speeds to backstop any caching.

[porina]

2 hours ago, Loki0111 said:

Gives you 16 SATA 3 drive connections at 6Gbps, the Intel SAS Expander does not need to go into a PCIe slot as it can independently powered off a 4 pin connector and just bolted to the side of a case. The PCIe connection only provide power. Do not use the 4 pin power connector if you put in a card slot, its one or the other.

Ooh, you might be onto something here. The main reason being, used SAS drives seem to be fairly cheap, presumably as it is difficult for consumers to use them. Similar situation to ECC registered ram.
 

2 hours ago, Loki0111 said:

Then just use any group of 16 drives, plus whatever the motherboard will take for a large storage cluster.

Also I have another experimental use where this could apply but I'm struggling on how the expander works. Skimming Intel's document on it, it is still not clear what you can or can't do with it, perhaps due to my lack of understanding of SAS in general. Would you connect one or two cables from the LSI card to the expander? I understand each connector can connect up to 4 drives, so since you mention 16, that assumes 4 on the expander going to drives, with two going to the LSI card? Main reason for asking this is to understand the potential bandwidth. Very roughly, each 4-port connector could transfer just over 2GB/s, for just over 4GB/s for the whole card. Does the expander essentially share this bandwidth with attached drives? Not sure exactly what typical bandwidth of a 3TB 7200rpm drive is, but if we were generous and called it 200MB/s, 16 drives still wouldn't saturate the adapter...

 

If applicable, once I'm done with that side task, I can then reconfigure it for general storage. Although I don't need 16x3GB of potential storage... yet.

 

2 hours ago, Loki0111 said:

Unraid is okay if we are talking about high speed data access I'd probably go with TrueNAS, OMV, Linux or Windows Server 2016/2019. You can also use Windows 10 Pro but you lose tiered storage, you can cheat around that with Primocache though which will allow you to add L1 (RAM), L2 (NVME/SSD) read/write cache.

I have two use cases. The unraid is the backup server, so performance is not critical on it. With the drives I currently have in it, I typically get 50MB/s writes. Reads in theory would be whatever the native single drive performance is. For that use case, I want simple, affordable. Set it up and largely forget about it. I keep thinking ZFS would be the best solution but despite looking it at many times over the years I really don't understand it's limitations when it comes to modifying capacity. It fails on the simplicity which is where the more adaptive nature of unraid wins.

 

The other system with live data is now multiple single drives. Drives in there sustain just over 200MB/s, which is closely matched to the 2.5g network throughput. Going faster there wont make a big difference, unless random seeks are separately optimised for.

[Loki0111]

Quote

Ooh, you might be onto something here. The main reason being, used SAS drives seem to be fairly cheap, presumably as it is difficult for consumers to use them. Similar situation to ECC registered ram.

Just to be clear the HBA and the SAS Expander are compatible with both SATA and SAS drives, you just use a different cable depending on the type of drives you have.

 

Quote

Also I have another experimental use where this could apply but I'm struggling on how the expander works. Skimming Intel's document on it, it is still not clear what you can or can't do with it, perhaps due to my lack of understanding of SAS in general. Would you connect one or two cables from the LSI card to the expander?

You can use one or two cables between the HBA and SAS Expander. Depending on the expander you may lost ports on the card if you only use one cable and even if you retain all the ports obviously using two cables in almost all situations will give you better speed.

 

Quote

I understand each connector can connect up to 4 drives, so since you mention 16, that assumes 4 on the expander going to drives, with two going to the LSI card?

You'd want two SFF-8087 to SFF-8087 cables going between the HBA and the SAS Expander. Just use the 4 pin power connector on the Expander and mount it to the side of the case. Then you'd use the ports on the SAS Expander to fan out to the drives, either SFF-8087 to SATA or to SAS fan out cables. You'd have a total of 16 drive connections available off the SAS Expander.

 

Your overall bandwidth available between the 16 drives is limited to 6Gbps. The SAS Expander essentially operates sort of like a SAS/SATA hub and pipes everything though that HBA card which handles all the communication with the system. If you need more speed you can bump up to 12Gbps with newer cards but that is going to jump the price.

 

In almost any real world scenario 16 spinning drives won't saturate a 6Gbps connection. If you are mixing in a lot of SSD's it may, in which case you may need to look at a 12Gbps card.

 

Quote

For that use case, I want simple, affordable. Set it up and largely forget about it. I keep thinking ZFS would be the best solution but despite looking it at many times over the years I really don't understand it's limitations when it comes to modifying capacity. It fails on the simplicity which is where the more adaptive nature of unraid wins.

 

So zfs is the technically better solution and you can modify the capacity of the fly (depending on your drive setup), with mirrored drives I've found it extremely easy to expand or reduce as need. It does require a little bit of command line knowledge on FreeBSD at times. I am also using Mellanox 40G cards though so I have close to NVMe transfer speeds between my server and desktop. Now that I am used to that going back to single disk access speeds would unbearable in my case for any extended period of time.

 

That being said Unraid is *a lot* easier to use for beginners. In terms of simplicity Unraid is almost always going to win because it just has fewer moving parts and options on the drive config side. It was designed with that intention in mind.

[porina]

4 hours ago, Loki0111 said:

Just to be clear the HBA and the SAS Expander are compatible with both SATA and SAS drives, you just use a different cable depending on the type of drives you have.

Got it. I already have a SAS controller to play with some high rpm enterprise drives in the past. However if I'm looking to buy storage, used SAS drives could be interesting over new SATA drives.

 

Quote

Your overall bandwidth available between the 16 drives is limited to 6Gbps. The SAS Expander essentially operates sort of like a SAS/SATA hub and pipes everything though that HBA card which handles all the communication with the system. If you need more speed you can bump up to 12Gbps with newer cards but that is going to jump the price.

To be clear, I'm not so interested in the drive-controller bandwidth, which I know will be fine. I'm curious how the controller handles the aggregate bandwidth between all the drives and the host. So for example, if I have 16 HDs, will I have 16x one drive worth of peak transfer rate? I want to experiment with HDs as poor person's substitute for huge ram compute applications, but cost has put me off in the past. Capacity or latency wont be a problem, but bandwidth will be. 

 

Quote

It does require a little bit of command line knowledge on FreeBSD at times.

That's where it loses me. I will make a mistake sooner or later and don't want to lose data when I do.

 

Quote

I am also using Mellanox 40G cards though so I have close to NVMe transfer speeds between my server and desktop. Now that I am used to that going back to single disk access speeds would unbearable in my case for any extended period of time.

That would be nice, to have remote storage as fast as local (in bandwidth if not in latency). My primary application is video editing, so even HD rates are not so bad as I'm compute limited most of the time.

 

[Loki0111]

Quote

To be clear, I'm not so interested in the drive-controller bandwidth, which I know will be fine. I'm curious how the controller handles the aggregate bandwidth between all the drives and the host. So for example, if I have 16 HDs, will I have 16x one drive worth of peak transfer rate? I want to experiment with HDs as poor person's substitute for huge ram compute applications, but cost has put me off in the past. Capacity or latency wont be a problem, but bandwidth will be. 

 

It entirely depends on the number of drives, type and how your data is layed out. Ultimately you are limited to 6Gbps on that controller.

 

So spinning drives under perfect conditions might get 100-200 megabytes a second worth of transfer speed. SATA SSD's might get 500-600 megabytes a second.

 

Keep in mind spinning drives are a lot quicker in sequential reads and writes, as soon as they have to start seeking for data their transfer speeds drop to shit.

 

So you'd need about 40 spinning drives being fully accessed at the same time to totally saturate out those 6Gbps connections. If you were using SSD's then 10 SSD's being fully accessed would in theory saturate out the 6Gbps links.

[porina]

2 minutes ago, Loki0111 said:

It entirely depends on the number of drives, type and how your data is layed out. Ultimately you are limited to 6Gbps on that controller.

Per link right? So with two of those cables thats 8 links between controller and expander.

 

2 minutes ago, Loki0111 said:

So spinning drives under perfect conditions might get 100-200 megabytes a second worth of transfer speed. SATA SSD's might get 500-600 megabytes a second.

I can't afford that many SSDs  It will have to be HDs.

 

2 minutes ago, Loki0111 said:

Keep in mind spinning drives are a lot quicker in sequential reads and write, as soon as they have to start seeking for data their transfer speeds drop to shit.

Fortunately the application I have in mind has been designed with HD seek latency in mind and can mitigate its effects, so the effective limit is bandwidth. This is also a reason SSDs might not be faster in practice. Most affordable SSDs will use a small write cache, and the sustained performance could be well below HD levels. I did look at used enterprise SSDs which are cacheless but they were still too expensive. Also, due to the high write rate of the workload I'd probably kill them quickly. Workload can be approximated as 50/50 read/writes sustained for days.

[Loki0111]

Quote

Per link right? So with two of those cables thats 8 links between controller and expander.

Okay, I see what you are asking.

 

The HBA supports 8 lanes full duplex at 6Gbps per lane and up to 256 SATA or SAS end devices. When connected to both ports the SAS Expander optimizes the device traffic across those 8 lanes.

[porina]

3 minutes ago, Loki0111 said:

Okay, I see what you are asking.

 

The HBA supports 8 lanes full duplex at 6Gbps per lane and up to 256 SATA or SAS end devices. When connected to both ports the SAS Expander optimizes the device traffic across those 8 lanes.

Thanks, that is what I was looking to understand. It makes sense, but I didn't want to assume. I'll have to see how rich I feel, because with the quantity of drives involved it'll still be costly even if I re-purpose it for general storage later on.

 

Can you stack expanders? Or put two expanders on one controller? Need to think about potential bandwidth optimisations.

[Loki0111]

Quote

Can you stack expanders? Or put two expanders on one controller? Need to think about potential bandwidth optimisations.

In theory yes, but I don't recommend stacking them behind one another. That is asking for an issue.

 

You could feed one cable with 4 lanes to a single SAS Expander and the other cable with its 4 lanes to another SAS Expander. It would just cut the number of lanes they have available for traffic down, but also allow you to connect more devices.

 

The other alternative is to just get an HBA card with more ports and lanes available, but it'll cost more.

[porina]

1 minute ago, Loki0111 said:

In theory yes, but I don't recommend stacking them behind one another. That is asking for an issue.

 

You could feed one cable with 4 lanes to a single SAS Expander and the other cable with its 4 lanes to another SAS Expander. It would just cut the number of lanes they have available for traffic down, but also allow you to connect more devices.

 

The other alternative is to just get an HBA card with more ports and lanes available, but it'll cost more.

I was thinking stacking them was unlikely to be a good idea as soon as I wrote it. Anyway, thanks again, given me much to think about here.

[Bombastinator]

7 hours ago, porina said:

Got it. I already have a SAS controller to play with some high rpm enterprise drives in the past. However if I'm looking to buy storage, used SAS drives could be interesting over new SATA drives.

 

To be clear, I'm not so interested in the drive-controller bandwidth, which I know will be fine. I'm curious how the controller handles the aggregate bandwidth between all the drives and the host. So for example, if I have 16 HDs, will I have 16x one drive worth of peak transfer rate? I want to experiment with HDs as poor person's substitute for huge ram compute applications, but cost has put me off in the past. Capacity or latency wont be a problem, but bandwidth will be. 

 

That's where it loses me. I will make a mistake sooner or later and don't want to lose data when I do.

 

That would be nice, to have remote storage as fast as local (in bandwidth if not in latency). My primary application is video editing, so even HD rates are not so bad as I'm compute limited most of the time.

 

Re: swap.
Using even SSDs as memory let alone HHDs is super slow.  You’ll be drinking a lot of coffee.  It does work, but really really slow.  I used to make poster sized stuff with QUARK that were about 8 times the size of the amount of ram available.  I’d punch one up in the evening and they were generally ready by lunch the next day.  Not breakfast. 

Edited December 21, 2020 by Bombastinator

[porina]

15 minutes ago, Bombastinator said:

Re: swap.
Using even SSDs as memory let alone HHDs is super slow.  You’ll be drinking a lot of coffee.  It does work, but really really slow.  I used to make poster sized stuff with QUARK that were about 8 times the size of the amount of ram available.  I’d punch one up in the evening and they were generally ready by lunch the next day.  Not breakfast. 

You don't need to tell me that, but last time I checked my back account, buying a system with 8TB of ram it is was a little out of my reach. That's why I'd need a crazy number of HDs, which means I'd need to get them cheap. As an illustration, 32 HDs assuming a good case of 200MB/s each, would translate to 6.4GB/s, or roughly equivalent to the upper end of single channel DDR2. The workload I have in mind has been designed to work like this, since no one can afford enough ram to run it, and it also takes into consideration HD seek times and makes the access as sequential as possible. As I hinted at, the sort of tasks I'd like to try on it can run days or longer. I know others who have run similar for months, setting world records in the process for how fast they can do it.

 

Edit: in case it was not obvious, I would not be sitting there watching it run. It'll be set and forget.