Plans to build a 12 drive SSD array

[Windows7ge]

I have questions and concerns about this project including issues such as:

• TRIM
• Using multiple SATA controllers & how that will impact single network client performance
• write cache/buffer
• under-provisioning

Server Hardware:

• CPU: Intel Xeon E5-2670's
• RAM: 16x8GB Kingston DDR3 UDIMM ECC 1600MHz
• Motherboard: ASRock EP2C602-4L/D16
• SATA Controller: 3xSATAIII SAS 9207-8i HBA
• CASE: NORCO RPC-4224

Server Software:

• FreeNAS
  • ZFS raidz2 (RAID6)

I plan to use the Intel DC S4500 960GB (subject to change).

 

Questions/Concerns:

• TRIM
  • In the early days of consumer SATA SSDs it was known that hardware RAID controllers disabled the TRIM function.
    • What about software RAID?
    • What about FreeNAS ZFS RAID specifically? The only difference I know of between software and hardware RAID is that hardware RAID uses a dedicated controller while software uses system CPU/memory.
  • Does the OS need to support TRIM to use it or is it solely a feature of the SSD itself?
    • Would the SSD run TRIM itself on its own schedule or does the OS have to tell it to?
• Multiple SATAIII Controllers
  • When drives are spanned evenly across multiple controllers is it possible for a single client to see speeds that exceed the limit of a single controller?
    • In theory it should be possible but I won't be surprised if it doesn't workout that way. With multiple clients though there should be perfect scaling.
• Write Cache/Buffer
  • In the past putting an SSD in the server I've seen abnormal write operations where it will start out strong and drop radically. After a short time it would shoot up again and repeat this cycle until the move completed. It acted as if a high speed cache or buffer were being filled, then emptied to a slower permanent media then started refilling the buffer with more data, etc. Can anyone explain this activity?
• Provisioning
  • Would any form of provisioning/under-provisioning be recommended or just setup all the usable space in one giant raidz2/RAID6 volume and not worry about degradation?

 

@Electronics Wizardy @leadeater @Lurick I believe all of you could have valuable input among others who's usernames I don't remember so I can't link them.

 

If anybody has any other input or things I should be wary of don't be afraid to mention it.

[GDRRiley]

you should be fine with provisioning as these drives are rated for almost 2PB written to them so they have lots of unused nand on them.

write cache these will have for sure. Almost all SSDs have them its just something to make small reads and writes faster.

It should exceed 1 controller speeds. 

Trim is a OS/controller thing. 

Note all raid is software. somewhere a chip is doing raid calculations. 

[Electronics Wizardy]

What Are you using this storage for?

 

What OS are you on? freeBSD, Linux? Other?

EDIT SAW FREENAS, TRIM WILL WORK

 

IGNORE:Trim depends on os, its only enabled on freeBSD currently, but really won't make a big difference, esp with enterprise ssds.

 

Are you using sync? That makes a big difference in performance here.

 

1 hour ago, Windows7ge said:

When drives are spanned evenly across multiple controllers is it possible for a single client to see speeds that exceed the limit of a single controller?

Yep you can see speeds higher than a single controller can provide.

 

1 hour ago, Windows7ge said:

. Can anyone explain this activity?

Thats write caching. Lots of setting to tweak here, like how big, how often to dump. If you have sync on its all disabled.

 

1 hour ago, Windows7ge said:

Does the OS need to support TRIM to use it or is it solely a feature of the SSD itself?

You need OS, controller, filesystem, and ssd support.

 

For case, Id get something like this https://www.ebay.com/itm/Robust-Dell-PowerEdge-R720xd-Server-2x-2-00Ghz-E5-2650-8C-32GB/323079681518?hash=item4b390cb5ee:g:U94AAOSwS6ZavEXz

 

There much better built, and its 2u instead of 4u and you get better software support. And 24dimms(go r820 if you want 48 dimms and quad socket).

 

1 hour ago, Windows7ge said:

Would any form of provisioning/under-provisioning be recommended or just setup all the usable space in one giant raidz2/RAID6 volume and not worry about degradation?

Depends on use. Also zfs normally isn't the fastest solution, so usecase?

 

1 hour ago, Windows7ge said:

I plan to use the Intel DC S4500 960GB (subject to change).

Big depends on use case.

 

If you want Iops, go pcie.

 

If you want capicity, not a bad choice. If you doing lots of writes, get a different drive, these have pretty slow writes.

[leadeater]

1 hour ago, Windows7ge said:

TRIM

• In the early days of consumer SATA SSDs it was known that hardware RAID controllers disabled the TRIM function.
  • What about software RAID?
  • What about FreeNAS ZFS RAID specifically? The only difference I know of between software and hardware RAID is that hardware RAID uses a dedicated controller while software uses system CPU/memory.
• Does the OS need to support TRIM to use it or is it solely a feature of the SSD itself?
  • Would the SSD run TRIM itself on its own schedule or does the OS have to tell it to?

Yes, on BSD only

 

Quote

illumos	not yet ported
FreeBSD	Sep 2012
ZFS on Linux	not yet ported
OpenZFS on OS X	not yet ported

http://open-zfs.org/wiki/Features#TRIM_Support

https://www.phoronix.com/scan.php?page=news_item&px=ZFS-On-Linux-TRIM-Closer

 

1 hour ago, Windows7ge said:

Multiple SATAIII Controllers

• When drives are spanned evenly across multiple controllers is it possible for a single client to see speeds that exceed the limit of a single controller?
  • In theory it should be possible but I won't be surprised if it doesn't workout that way. With multiple clients though there should be perfect scaling.

Yes, this is one of the reasons to span across controllers although it's rather unlikely you'll exceed the performance of a single SAS2 controller. Only a concern when using expanders but it's often easier and cheaper to just add another controller to get the number of required ports when using software RAIDs.

 

This does depend on how the data is being distributed across disks, if you're going to use RAIDz2 then not for writes but it will for reads.

 

1 hour ago, Windows7ge said:

Write Cache/Buffer

• In the past putting an SSD in the server I've seen abnormal write operations where it will start out strong and drop radically. After a short time it would shoot up again and repeat this cycle until the move completed. It acted as if a high speed cache or buffer were being filled, then emptied to a slower permanent media then started refilling the buffer with more data, etc. Can anyone explain this activity?

That's why you are buying server SSDs . You see that behavior more often when TRIM is not supported but it happens on all consumer SSDs if you have sustained I/O for long enough. There isn't enough OP NAND, physically/firmware wise, and the firmware isn't optimized for working well without TRIM and doesn't have good background garbage collection.

 

1 hour ago, Windows7ge said:

Provisioning

• Would any form of provisioning/under-provisioning be recommended or just setup all the usable space in one giant raidz2/RAID6 volume and not worry about degradation?

Up to you, shouldn't be a big issue with the SSDs you are buying but the more you set aside the longer you can sustain higher peak performance before it starts to drop off. Server SSDs do have much high low point than consumer SSDs, 15000-50000 IOPs is about standard for a SATA SSD. I think consumer gets as low as 1000-3000 but that's a rather bad one.

 

http://www.storagereview.com/samsung_ssd_845dc_pro_review

 

http://www.thessdreview.com/our-reviews/samsung-845dc-pro-review-800gb-class-leading-speed-endurance/3/

[Windows7ge]

25 minutes ago, Electronics Wizardy said:

What Are you using this storage for?

Primarily storage for all my excess data. File sizes range anywhere as low as bytes all the way up to 10GB+ files. With the excess storage though I can put a few zvol on the array and it can double as a boot drive for a series of VMs.

 

29 minutes ago, Electronics Wizardy said:

Are you using sync? That makes a big difference in performance here.

A question I cannot answer because I haven't bought the drives yet. I recall a sync setting when I was experimenting with L2ARC devices. The sync you're speaking of is likely different. If you could clarify where I might find it and what I should change it to would be appreciated.

 

36 minutes ago, Electronics Wizardy said:

Thats write caching. Lots of setting to tweak here, like how big, how often to dump. If you have sync on its all disabled.

So that's what that was. I was experimenting with a single desktop SSD purely for the sake of the experiment that day and you've just explained the performance I saw when writing to the drive over the network.

 

47 minutes ago, Electronics Wizardy said:

There much better built, and its 2u instead of 4u and you get better software support. And 24dimms(go r820 if you want 48 dimms and quad socket).

Too bad the whole server is built and has been in operation since January, 2016. This topic is just to discuss upgrading from mechanical media to pure solid state (except I'll likely backup the solid state array on the mechanical array).

 

52 minutes ago, Electronics Wizardy said:

Depends on use. Also zfs normally isn't the fastest solution, so usecase?

Primarily a file server. Will likely share the array with a few zvols for VMs.

 

54 minutes ago, Electronics Wizardy said:

If you want Iops, go pcie.

 

If you want capicity, not a bad choice. If you doing lots of writes, get a different drive, these have pretty slow writes.

As I've been told in the past even 2500 IOPS can run many, many VMs. So I don't plan to go full PCI_e (at this capacity, I technically have a PCI_e SSD in the server already) any time soon nor do I need that many IOPS (though this will equivalently have way more than I'll ever need).

 

Capacity does have to do with it but the desire is to also exceed 10Gbit writes. Even if I'm limited by 6Gbit (both you and GDRRiley say that shouldn't be the case) then multiple clients can access it and exceed 10Gbit. I know you know more than me in this subject but writes as high as 490MB/s x10 then factoring in 2 parity bits there's no way to say how fast or slow that would be without testing it but I would well expect beyond 10Gbit performance hopefully as high as the max theoretical 18Gbit of the 3 controllers. Could I find faster drives? Yes but I don't think it'd make that big a difference when working with 12 drives. You'd hit the max speed of the controller and then you'd have to use denser disks to get the same capacity. At the same time even though throughput is bottlenecked by the controllers bandwidth IOPS will effectively scale even though they're so high they're useless for my application.

[Electronics Wizardy]

26 minutes ago, Windows7ge said:

Primarily storage for all my excess data. File sizes range anywhere as low as bytes all the way up to 10GB+ files. With the excess storage though I can put a few zvol on the array and it can double as a boot drive for a series of VMs.

So this is just your home file server? Reason why all ssd?

 

Id personally make a fast a fast pool with pcie ssds for vms and a slow pool with hdds with a large l2arc for less used data, but if you got the money for all ssd go for it.

 

26 minutes ago, Windows7ge said:

A question I cannot answer because I haven't bought the drives yet. I recall a sync setting when I was experimenting with L2ARC devices. The sync you're speaking of is likely different. If you could clarify where I might find it and what I should change it to would be appreciated.

Sync depends on the workload, not drives.

 

Vms are normally sync, so esxi is all sync. NFS can be either, CIFS/samba is async, so a slog won't help with async. Getting a 900p as a  slog may help with vms a good amount.

 

26 minutes ago, Windows7ge said:

So that's what that was. I was experimenting with a single desktop SSD purely for the sake of the experiment that day and you've just explained the performance I saw when writing to the drive over the network.

Whats the network speed? Write caching looks normally like a burst, then steady. If its changing lots, esp on 1gbe its probably different file sizes of loads from other things.

 

26 minutes ago, Windows7ge said:

Primarily a file server. Will likely share the array with a few zvols for VMs.

What are the clients using it doing? video editing? databases? VMs? Fileserver can be many things.

 

26 minutes ago, Windows7ge said:

Capacity does have to do with it but the desire is to also exceed 10Gbit writes.

You should be able to do this with hdds easily. A 10 hdd drive array should do this. ZFS can be a bit slow here though with large arrays compared to therotical speeds.

 

26 minutes ago, Windows7ge said:

Even if I'm limited by 6Gbit (both you and GDRRiley say that shouldn't be the case)

Thats 6gbit per drive, 48gbit per controller, that is not a limit at all here.

 

26 minutes ago, Windows7ge said:

As I've been told in the past even 2500 IOPS can run many, many VMs.

Big thing is depends on the vm. I can run 100 vms on a single hdd, but its going to suck, so depends on the vm workload. Im currently running my vms on 3x 4 drive raidz with a slog and a l2arc and its io limited a good amount, I should really get a ssd for vms. I am personally looking at these. and you might want to aswell. You can get bigger ones as well. There basically 4 sata ssds on a lsi controller.  https://www.ebay.com/itm/Sun-Oracle-800GB-Flash-Accelerator-F80-Card-LSI-Nytro-WarpDrive-PCIe-SSD/183037763174?hash=item2a9de69e66:g:pZEAAOSwqvNaaww

 

 

[leadeater]

27 minutes ago, Electronics Wizardy said:

There basically 4 sata ssds on a lsi controller.  https://www.ebay.com/itm/Sun-Oracle-800GB-Flash-Accelerator-F80-Card-LSI-Nytro-WarpDrive-PCIe-SSD/183037763174?hash=item2a9de69e66:g:pZEAAOSwqvNaaww

Hmm that's actually really nice and great price too. Might get a few of these in a couple of weeks . Any more gems like these?

[Windows7ge]

1 hour ago, leadeater said:

Yes, this is one of the reasons to span across controllers although it's rather unlikely you'll exceed the performance of a single SAS2 controller. Only a concern when using expanders but it's often easier and cheaper to just add another controller to get the number of required ports when using software RAIDs.

 

This does depend on how the data is being distributed across disks, if you're going to use RAIDz2 then not for writes but it will for reads.

So you don't predict beyond 6Gbit (SAS2) for writes but it's probable to see beyond 6Gbit reads?

 

1 hour ago, leadeater said:

That's why you are buying server SSDs . You see that behavior more often when TRIM is not supported but it happens on all consumer SSDs if you have sustained I/O for long enough. There isn't enough OP NAND, physically/firmware wise, and the firmware isn't optimized for working well without TRIM and doesn't have good background garbage collection.

The very unenvironmentally controlled test where I discovered this was with just any desktop SSD. It was either a Samsung or Adata. I can't remember which but yes it didn't like what I was doing. It was only for experimental purposes I didn't keep using it in that application.

 

1 hour ago, leadeater said:

Up to you, shouldn't be a big issue with the SSDs you are buying but the more you set aside the longer you can sustain higher peak performance before it starts to drop off. Server SSDs do have much high low point than consumer SSDs, 15000-50000 IOPs is about standard for a SATA SSD. I think consumer gets as low as 1000-3000 but that's a rather bad one.

The size and number of files I would plan to write to the array would probably rarely exceed 20GB at a time so divide that by 10 and add 16.6% for parity worse case scenario I'd be looking at a sustained write of 2~3GB to each drive. I don't think that would need much or any unallocated storage in order to handle but if I'm wrong I can always rebuild the array leaving a little free space and see if it improves sustained writes.

[Electronics Wizardy]

Just now, leadeater said:

Hmm that's actually really nice and great price too. Might get a few of these in a couple of weeks . Any more gems like these?

The 400gb version is about 70 bucks with shipping. 

 

 

The problem with most used pcie ssds is they need special drivers and driver support is ended. Fusion io drives are cheap, but drivers can be a pain as not included in the linux kernel. Thats why I liked these drives. The intel ones are starting to get cheap(ish). You see p3500's for about .4 dollars a gig, its not horrible for a newish intel enterprise pcie ssd.

 

 

 

 

[Electronics Wizardy]

3 minutes ago, Windows7ge said:

So you don't predict beyond 6Gbit (SAS2) for writes but it's probable to see beyond 6Gbit reads?

The 6gbit limit is per drive, not total. SAS/sata won't be a limit here at all.

 

[leadeater]

5 minutes ago, Electronics Wizardy said:

The problem with most used pcie ssds is they need special drivers and driver support is ended. Fusion io drives are cheap, but drivers can be a pain as not included in the linux kernel. Thats why I liked these drives. The intel ones are starting to get cheap(ish). You see p3500's for about .4 dollars a gig, its not horrible for a newish intel enterprise pcie ssd.

These seem a little different to the Fusion io drives I've used, those ones showed up as a single disk where this looks like it comes up as 4 individual drives.

https://docs.oracle.com/cd/E41278_01/pdf/E41251.pdf

 

The SAS controller on it is just a LSI 2008 and it says it's using the IT firmware, this one might not require drivers at all? 

 

Edit:

Also it's cheap enough for me to just take the SSDs off it.

[Electronics Wizardy]

15 minutes ago, leadeater said:

These seem a little different to the Fusion io drives I've used,

The 4 drives was for the nytro drives, not the fusion io drives.

 

15 minutes ago, leadeater said:

The SAS controller on it is just a LSI 2008 and it says it's using the IT firmware, this one might not require drivers at all? 

 

yep, drivers are included in most oses.

 

16 minutes ago, leadeater said:

Also it's cheap enough for me to just take the SSDs off it.

There custom formfactor though, so unless you have an adapter, or a drive that needs its ssd replaced, it seems useless.

[leadeater]

12 minutes ago, Electronics Wizardy said:

There custom formfactor though, so unless you have an adapter, or a drive that needs its ssd replaced, it seems useless.

Form factor isn't a big issue for what I'd want them for, not that it matters since it'll work as is.

 

Got confused, thought you meant these were actually Fusion io under the hood, which yea I wouldn't buy those.

 

Edit:

I actually need them to show up as multiple drives btw, down side to PCIe/NVMe is if you need multiple due to software requirements the cost is too high.

 

In fact the more I think about it the more perfect these are, I can use the 3.5" bays for HDDs and have the SSDs internal and not waste 2 or 4 bays to SSDs.

[Windows7ge]

34 minutes ago, Electronics Wizardy said:

So this is just your home file server? Reason why all ssd?

I'm kind of tired of the limitation of mechanical storage. Especially when the rest of my hardware can go so much faster. A array of eight 7200RPM drives in raidz2 gave me a max performance of around 550MB/s writes (not a particularly optimized array, quite frankly because I don't know how). Now I could just add more disks but I feel like that's not the way to go when an SSD is insurmountably faster in fact a RAID0 of 6 drives would push me beyond 20Gbit (ignoring controller limitations), they're smaller, use less power, will probably survive a drop to the floor more often not that that's a frequent occurrence with me. The only immediately apparent issues are cost and making sure which ones you buy are compatible with the application. Then there's TRIM, the buffer thing I mentioned, etc, etc.

 

46 minutes ago, Electronics Wizardy said:

Vms are normally sync, so esxi is all sync. NFS can be either, CIFS/samba is async, so a slog won't help with async. Getting a 900p as a  slog may help with vms a good amount.

Oh, so you WERE talking about the same thing. I learned the hard way that file server = async (simply put) and that a L2ARC device is useless for that application (objective to increase write performance). Doing sync=always forces the files to be written to the L2ARC device but it will only do it as fast as the mechanical array can take it and since latency is being introduced it often results in even lower performance. Solution? Bigger HDD array or full SSD array...I did one. Now I want to try the other. However I can leverage the IOPS of the SSD array to run a series of VMs as well as handle the file share. Should the two be mixed in one array? Probably not recommended but if the array goes offline it isn't detrimental to the VMs I'll have it running. They won't be doing anything mission critical.

 

55 minutes ago, Electronics Wizardy said:

Whats the network speed? Write caching looks normally like a burst, then steady. If its changing lots, esp on 1gbe its probably different file sizes of loads from other things.

At the time of the test it was 10Gbit over a fiberoptic P2P connection. I have now doubled that to 20Gbit and introduced a 16 port SFP+ switch so now I can have more 10Gbit or 20Gbit clients attached at once.

 

The test itself was like I described. Full SSD performance for about 2 seconds then it dropped down to about 1MB or <1MB for 6~7 seconds then shot back up to full SSD speeds in under a second then repeat again, and again until the move completed.

 

1 hour ago, Electronics Wizardy said:

What are the clients using it doing? video editing? databases? VMs? Fileserver can be many things.

It's being used for general storage of what a typical techy would have on their server/NAS. Videos, Music, Photos, many many pieces of random software and projects. Also I should mention the number of users is 1...I built it and I'm the only one who uses it. It's basically a junk drawer for all my software bits. I do someday hope to find a real purpose for it with multiple users being able to utilize it but that likely won't exceed typical home use needs like music/videos streaming or more long term storage.

 

I admit I built something SOOO overpowered for what I use it for. I easily could have bought a external HDD for maybe $150 and have that be sufficient for my needs but nope. Gotta built something that I could probably use to service the entire town I live in.

 

1 hour ago, Electronics Wizardy said:

You should be able to do this with hdds easily. A 10 hdd drive array should do this. ZFS can be a bit slow here though with large arrays compared to therotical speeds.

My mechanical 8 drive array only goes as high as 550MB/s. Lets say I buy 8 more drives I might see 1,250MB/s but a 16 drive array sounds excessive with 2 drives for parody and I don't like how much power that would draw 24/7 (even though I will never recoup was a full SSD array will cost) then as I mentioned before I have 20Gbit which would imply using even more disks...SSDs, they're expensive but I'll hit 20Gbit so much faster and easier.

 

1 hour ago, Electronics Wizardy said:

Thats 6gbit per drive, 48gbit per controller, that is not a limit at all here.

My understanding is that you can only see a max of 6Gbit per controller. This being why enthusiasts would RAID0 only 2 SSDs in a desktop because the controller would already be near max with one but maxed out with 2. Unless server cards are a special exception but then why do 12Gbit SAS cards exist when 15,000RPM drives don't touch 6Gbit? So it's my understanding that per card (or per controller) it's 6Gbit not per port even though each port alone is capable of 6Gbit it doesn't scale beyond that inside the controller.

 

1 hour ago, Electronics Wizardy said:

Big thing is depends on the vm. I can run 100 vms on a single hdd, but its going to suck, so depends on the vm workload. Im currently running my vms on 3x 4 drive raidz with a slog and a l2arc and its io limited a good amount, I should really get a ssd for vms.

Makes sense. Build a smaller version of what I want to build. Maybe a 3/4 drive RAID5.

[Electronics Wizardy]

1 minute ago, Windows7ge said:

Oh, so you WERE talking about the same thing. I learned the hard way that file server = async (simply put) and that a L2ARC device is useless for that application (objective to increase write performance). Doing sync=always forces the files to be written to the L2ARC device but it will only do it as fast as the mechanical array can take it and since latency is being introduced it often results in even lower performance. Solution? Bigger HDD array or full SSD array...I did one. Now I want to try the other. However I can leverage the IOPS of the SSD array to run a series of VMs as well as handle the file share. Should the two be mixed in one array? Probably not recommended but if the array goes offline it isn't detrimental to the VMs I'll have it running. They won't be doing anything mission critical.

You got l2arc and slog confused. l2arc is read only, slog is sync write only.

 

If you don't care about the last 5 seconds of data, id just turn off sync and get the good amount boost in performance. Sync is still a slowdown with ssds as there much slower than ram.

 

3 minutes ago, Windows7ge said:

It's being used for general storage of what a typical techy would have on their server/NAS. Videos, Music, Photos, many many pieces of random software and projects. Also I should mention the number of users is 1...I built it and I'm the only one who uses it. It's basically a junk drawer for all my software bits. I do someday hope to find a real purpose for it with multiple users being able to utilize it but that likely won't exceed typical home use needs like music/videos streaming or more long term storage.

 

I admit I built something SOOO overpowered for what I use it for. I easily could have bought a external HDD for maybe $150 and have that be sufficient for my needs but nope. Gotta built something that I could probably use to service the entire town I live in.

Yea I see ssds a bit overkill here, but you know and ssds are great, just making sure.

 

4 minutes ago, Windows7ge said:

My mechanical 8 drive array only goes as high as 550MB/s. Lets say I buy 8 more drives I might see 1,250MB/s but a 16 drive array sounds excessive with 2 drives for parody and I don't like how much power that would draw 24/7 (even though I will never recoup was a full SSD array will cost) then as I mentioned before I have 20Gbit which would imply using even more disks...SSDs, they're expensive but I'll hit 20Gbit so much faster and easier.

Is that write or read? How full? You should see more than that, Probably some tweaking can be done here.

 

5 minutes ago, Windows7ge said:

My understanding is that you can only see a max of 6Gbit per controller. This being why enthusiasts would RAID0 only 2 SSDs in a desktop because the controller would already be near max with one but maxed out with 2. Unless server cards are a special exception but then why do 12Gbit SAS cards exist when 15,000RPM drives don't touch 6Gbit? So it's my understanding that per card (or per controller) it's 6Gbit not per port even though each port alone is capable of 6Gbit it doesn't scale beyond that inside the controller.

Nope, its 6gbit per port. Here is a ssd raid 0 benchmark on a consumer platform. https://www.tomshardware.com/reviews/ssd-raid-benchmark,3485-3.html Well over the 6gbit limit. 

 

One controller with a sas expander will have almost no performance difference here.

 

 

[Windows7ge]

1 minute ago, Electronics Wizardy said:

You got l2arc and slog confused. l2arc is read only, slog is sync write only.

 

If you don't care about the last 5 seconds of data, id just turn off sync and get the good amount boost in performance. Sync is still a slowdown with ssds as there much slower than ram.

Wait...there's, SLOG, L2ARC, and there's ZIL. One is meant to describe or categorize what the other two are...I think...I did that experiment A WHILE ago and it took me a good week or two to learn it wasn't possible to do or at least to use in the way I was hoping.

 

If I don't care about the last 5 seconds of data...oh, in the event of power loss? I have a UPS hooked up and the power supply I have no concern of it dying any time soon so that's fine.

 

10 minutes ago, Electronics Wizardy said:

Is that write or read? How full? You should see more than that, Probably some tweaking can be done here.

To be more specific that was a write. It's a 16TB array (8x2TB) 12TB usable and about 1.5TB used (negligible amount).

A quick Diskmark test with a 2GB file:

Reads 608.2MB/s peak

Writes 355.1MB/s peak

 

Geese the server hates Diskmark. I know I get 500~550MB/s writes when moving 2~5GB video files to the server.

 

19 minutes ago, Electronics Wizardy said:

Nope, its 6gbit per port. Here is a ssd raid 0 benchmark on a consumer platform. https://www.tomshardware.com/reviews/ssd-raid-benchmark,3485-3.html Well over the 6gbit limit. 

You just redefined my understanding of SATA controllers. That means I could have used two quad port SFF-8087 HBA cards instead of 3 LSI's and have no performance penalty.

 

Eh, the LSI's were cheaper in total and I had the extra slot to spare.

[Electronics Wizardy]

21 minutes ago, Windows7ge said:

Wait...there's, SLOG, L2ARC, and there's ZIL. One is meant to describe or categorize what the other two are...I think...I did that experiment A WHILE ago and it took me a good week or two to learn it wasn't possible to do or at least to use in the way I was hoping.

L2ARC is a read cache

ZIL is a database of actions that the array will do soon, this is always in ram

SLOG is a device that holds the zil, so if there is power outage, you can recount your steps and lose no data.

 

You always have a zil, l2arcs helps with large datasets and lots ram with reads. SLOGS helps with sync writes.

 

23 minutes ago, Windows7ge said:

f I don't care about the last 5 seconds of data...oh, in the event of power loss? I have a UPS hooked up and the power supply I have no concern of it dying any time soon so that's fine.

If you don't care about power loss and those 5 seconds, just turn off sync and no need for a slog then.

 

25 minutes ago, Windows7ge said:

To be more specific that was a write. It's a 16TB array (8x2TB) 12TB usable and about 1.5TB used (negligible amount).

A quick Diskmark test with a 2GB file:

Reads 608.2MB/s peak

Writes 355.1MB/s peak

 

Geese the server hates Diskmark. I know I get 500~550MB/s writes when moving 2~5GB video files to the server.

Still seems a bit low, but can be cpu limits or compression(you got that on?)or many ohter things. My test array with a 5x 2tb raidz1 is getting about 400mB/s write and 600mB/s read.

38 minutes ago, Windows7ge said:

You just redefined my understanding of SATA controllers. That means I could have used two quad port SFF-8087 HBA cards instead of 3 LSI's and have no performance penalty.

Look at expanders in the future aswell. Each one of those controllers can drive like 256 drives with an expander like this guyhttps://www.ebay.com/itm/Intel-SAS-RAID-Expander-Card-Serial-ATA-600-Serial-Attached-SCSI-PCI-Express-x4/352203474853?hash=item5200f687a5:g:UWoAAOSw44BYCkCa

 

You don't need pcie, just splits one sff8087 into 5, and with 24gbit on a sff8087 your probably not badwidth limited.

[Windows7ge]

1 hour ago, Electronics Wizardy said:

You always have a zil, l2arcs helps with large datasets and lots ram with reads. SLOGS helps with sync writes.

Thank you for clarifying what they all mean and do.

1 hour ago, Electronics Wizardy said:

Still seems a bit low, but can be cpu limits or compression(you got that on?)or many ohter things. My test array with a 5x 2tb raidz1 is getting about 400mB/s write and 600mB/s read.

I don't know. Reading a 9.5GB file cached in server RAM to my desktop showed me peaks of 1.55GB/s~1.61GB/s and sustained between 1.35GB/s~1.45GB/s. Unless their's severe overhead in writing it to the array (which there undoubtedly could be) then CPU speed I don't think is the cause. Besides AFAIK with multichannel & RSS (I know RSS is pretty standard on most NICs) it should be paralleling the process across at least 2 threads or more lowering the necessity of a faster core-clock so I don't think that's it.

 

Also as a side note I just tried using the dd tool...I think it was a waste of time because it gave me numbers nowhere near the speed of the array:

dd if=/dev/urandom of=testfileR bs=512 count=10000; sync;

 

I tested it with block sizes 512, 1024, 2048, 4096, 8192, 16384, 32,768, 65,536. The highest speed it showed I would compare to that of a single 5400RPM HDD, around 108MB/s. Although if you multiply that by 6 and factor in parity that's about the max speed I'm seeing with my array...interesting...

 

1 hour ago, Electronics Wizardy said:

Look at expanders in the future aswell. Each one of those controllers can drive like 256 drives with an expander like this guyhttps://www.ebay.com/itm/Intel-SAS-RAID-Expander-Card-Serial-ATA-600-Serial-Attached-SCSI-PCI-Express-x4/352203474853?hash=item5200f687a5:g:UWoAAOSw44BYCkCa

 

You don't need pcie, just splits one sff8087 into 5, and with 24gbit on a sff8087 your probably not badwidth limited.

I'll keep that in mind for the future.

[Electronics Wizardy]

46 minutes ago, Windows7ge said:

their's severe overhead in writing it to the array

do you have compression on? That can use a lot of cpu depending on settings.

 

46 minutes ago, Windows7ge said:

Also as a side note I just tried using the dd tool...I think it was a waste of time because it gave me numbers nowhere near the speed of the array:

dd if=/dev/urandom of=testfileR bs=512 count=10000; sync;

DDing form urandom is normally a benchmark as urandom can be slow. You can try of=/dev/null(it just deletes the result) to see how fast urandom is.

 

 

[Windows7ge]

1 hour ago, Electronics Wizardy said:

do you have compression on? That can use a lot of cpu depending on settings.

It's off. What about sync? There's no log device assigned to the array but I can find the setting and see if it's off or not. Kind of a crapshoot idea, I know.

 

Oh, I also just remembered I enabled encryption on the array. That's CPU intensive. Though it's only AES-128, I'd prefer 256 or 512 even since we're dealing with *NIX here. Regardless that would lower performance a tad.

1 hour ago, Electronics Wizardy said:

DDing form urandom is normally a benchmark as urandom can be slow. You can try of=/dev/null(it just deletes the result) to see how fast urandom is.

Benchmark. Would this be a decent method for testing block size? An SSD array might benefit from a block size larger than what's typical (64KB I think is avg). I'd have to learn how to change an arrays block size before piling data into it but it sounds like a way to get the most out of a very expensive setup. I could write a script that just tests each and like you've suggested null to just delete the file when it's done with each test. Without that I'd just use rm followed by every file I made as the last command.

[jde3]

8 hours ago, Windows7ge said:

Oh, I also just remembered I enabled encryption on the array. That's CPU intensive. Though it's only AES-128, I'd prefer 256 or 512 even since we're dealing with *NIX here. Regardless that would lower performance a tad.

Check if cryptodev and aesni are loaded and working. (kldstat & dmesg | grep aesni)

 

As for benchmarks there might be some benchmarks in ports that would be more reliable.

 

[Electronics Wizardy]

8 hours ago, Windows7ge said:

It's off.

Try it on, compression is free speed and filespace normally. Turn on lz4.

 

8 hours ago, Windows7ge said:

What about sync?

Your not doing sync writes at 600mB/s to hdds and samba/cifs doesn't use sync.

 

 

[Windows7ge]

2 hours ago, jde3 said:

Check if cryptodev and aesni are loaded and working. (kldstat & dmesg | grep aesni)

kldstat

Id Refs Address            Size     Name
 1   80 0xffffffff80200000 2067000  kernel
 2    1 0xffffffff825ef000 ffd1c    ispfw.ko
 3    1 0xffffffff826ef000 7151     freenas_sysctl.ko
 4    1 0xffffffff82811000 592f     fdescfs.ko
 5    1 0xffffffff82817000 3337ee   vmm.ko
 6    1 0xffffffff82b4b000 30c4     nmdm.ko
 7    1 0xffffffff82b4f000 fabd     geom_mirror.ko
 8    1 0xffffffff82b5f000 47a1     geom_stripe.ko
 9    1 0xffffffff82b64000 ffc4     geom_raid3.ko
10    1 0xffffffff82b74000 16e03    geom_raid5.ko
11    1 0xffffffff82b8b000 5915     geom_gate.ko
12    1 0xffffffff82b91000 4e75     geom_multipath.ko
13    1 0xffffffff82b96000 828      dtraceall.ko
14    9 0xffffffff82b97000 41822    dtrace.ko
15    1 0xffffffff82bd9000 4883     dtmalloc.ko
16    1 0xffffffff82bde000 5aae     dtnfscl.ko
17    1 0xffffffff82be4000 67e1     fbt.ko
18    1 0xffffffff82beb000 58b81    fasttrap.ko
19    1 0xffffffff82c44000 1769     sdt.ko
20    1 0xffffffff82c46000 cf1e     systrace.ko
21    1 0xffffffff82c53000 ce87     systrace_freebsd32.ko
22    1 0xffffffff82c60000 53b6     profile.ko
23    1 0xffffffff82c66000 8496     ipmi.ko
24    1 0xffffffff82c6f000 f09      smbus.ko
25    1 0xffffffff82c70000 1bdde    hwpmc.ko
26    1 0xffffffff82c8c000 cfe9     t3_tom.ko
27    2 0xffffffff82c99000 45be     toecore.ko
28    1 0xffffffff82c9e000 15c71    t4_tom.ko
29    1 0xffffffff82cb4000 296e     uhid.ko
30    1 0xffffffff82cb7000 3620     ums.ko

dmesg | grep aesni

aesni0: <AES-CBC,AES-XTS,AES-GCM,AES-ICM> on motherboard

 

2 hours ago, jde3 said:

As for benchmarks there might be some benchmarks in ports that would be more reliable.

Ports? If ATTO worked with network drives I'd try testing it with that. As for accuracy I don't think the results would be swayed so far off that it'd make a monstrous difference if I'm off by one or two. Also in theory wouldn't a larger block size be less stress on the CPU(s). Kind of eliminate the core clock as a bottleneck.

[Windows7ge]

2 hours ago, Electronics Wizardy said:

Try it on, compression is free speed and filespace normally. Turn on lz4.

I turned on lz4 and waited a little while. Compression Ratio has not changed from 1.00x (lz4 I believe usually does around 1.33x) and performance has not altered either. A peak of 550MB/s and sustained around 450~475MB/s.

[Electronics Wizardy]

Just now, Windows7ge said:

I turned on lz4 and waited a little while. Compression Ratio has not changed from 1.00x (lz4 I believe usually does around 1.33x) and performance has not altered either. A peak of 550MB/s and sustained around 450~475MB/s.

what type of files? Compression is only used in zfs if it can hit a raito of about 1.12 or higher. You probably are dealing with lots of files like video files that aren't compressible at all, and compression will be turned off for the file.

 

But with ZFS, there really isn't a reason to turn compression off for most uses.