RAID Storage Solution - Video Editing & Caching

[leadeater]

9 hours ago, Strikermed said:

Any thoughts why I'm seeing 100% utilization, and only seeing 200MB/s activity and getting stuttery playback on proven formats? 

 

Ideally I want a format that doesn't loose much if any quality when transcoding, and that will play without any issues on the timeline. 

 

These results are pre-Cachecade.

 

I am considering trying different stripe sizes and various other adjustments on a few different RAID0 configurations to test those results. 

 

This is probably a good time to start going a little more in-depth in to storage I/O and how to manage/monitor performance. Disk utilization isn't really a good indicator of anything other than the disks are doing something all of the time during the sample period/points, it doesn't say what kind of demand or performance stress is happening.

 

 The other factor is the size of the I/O request and the pattern but I'll cover that later.

 

Information below is for performance counters in Windows Perfmon

 

The most important performance metric that you need to look at is avg disk sec/transfer which directly translates to latency. Storage latency and I/O request size have a very important relationship. When you are doing your tests monitor this perf counter and see how high it is, what this counter shows is how long it is taking for each storage transaction request where lower is better.

 

The other important performance metric that you should monitor is Current Disk Queue Length and Average Disk Queue Length. A high disk queue alone is not an indicator of a performance issue and this can be as high as the number of disk spindles in an array without issue. What this perf counter shows is how many storage transaction requests are pending in the queue waiting for disk access, if this is high but the avg disk sec/transfer is low then most of the time you are fine. This just means there is a lot of storage transactions being processed and it is happening happily and quickly. When the Current Disk Queue Length is high and avg disk sec/transfer is also high this means there is a performance issue and storage transactions and piling up in the queue and applications are getting very slow response times to storage access.

 

https://blogs.technet.microsoft.com/askcore/2012/03/16/windows-performance-monitor-disk-counters-explained/

 

Information below is how to calculate storage throughput/goodput 

 

As mentioned avg disk sec/transfer and storage I/O Size have an important relationship. You can use these two pieces of information to calculate the actual throughput you have.

 

Example 1:

avg disk sec/transfer:  0.005 (5ms)

I/O Size: 64KB

Throughput: (1/0.005) * 64 = 12800 KB/s or 12.5MB/s

 

Example 2:

avg disk sec/transfer:  0.005 (5ms)

I/O Size: 1024KB

Throughput: (1/0.005) * 1024 =204800 KB/s or 200MB/s

 

Example 3:

avg disk sec/transfer:  0.02 (20ms)

I/O Size: 1024KB

Throughput: (1/0.02) * 1024 = 51200 KB/s or 50MB/s

 

As you can quickly see this is why SSDs are so fast, it's all down to latency. Many HDDs can also have very high throughput as long as the access method is very sequential, random access patterns cause the latency to go up as the read/write heads have to seek and lock where an SSD does not.

 

What is hard about this is it can often be hard to know what your I/O size is, and this can vary a lot depending on application and partition allocation unit size.

 

Information below is on RAID strip size and stripe width/size sizing

 

I'll sum this one up quickly since there is a ton of information online, in general you want your strip size larger than your I/O size. Some hardware vendors use stripe width but actually mean strip size. When you set your strip size to say 64KB I/O is broken up in to 64KB chunks (unless it is smaller). The number of disks (stripe width) * strip size = stripe size.

 

 

Where this is important is that it impacts storage latency, but also sequential throughput. If your strip size is smaller than your I/O size to read or write data multiple HDD read/write heads have to seek and lock and the more that has to do this the longer the I/O request takes, you only want to hit one disk. The down side to this is you get lower theoretical throughput, I say theoretical as latency directly impacts this.

 

Storage I/O will still balance across the disks giving increased performance/throughput but no single I/O request will span more than one disk. If you are doing highly sequential workloads you may want to span multiple disks but this really only works if the I/O size is large enough to utilize the multiple disks and the latency hit does not outweigh the gain.

 

--------------------------------------------------

 

So on to your issue about stuttering and dropped frames what I think is happening is either the different formats use different size I/O and the smaller I/O combined with high latency is not giving you enough throughput for smooth playback, and/or the other possibility is the different formats compress more, high CPU load, but contain the same effective bit rate/quality. I say this since the raw footage plays without issue so if it's not I/O size then it has to be compression.

 

The smaller I/O size and latency seems to make a bit of sense to me regarding you trying the half resolution but still getting stutters.

 

For this CacheCade may help a lot since this is what it is designed to help with and SSDs have very good latency, the only problem is caching works much better for writes than reads. There is a term called cache warming which really only applies to reads so any data not in the cache will take time to be moved up in to the cache.

[Strikermed]

@leadeater

Wow, you are just a wealth of information.  I think to better understand this I will need to do some real world tests for myself, but it seems that I may not have chosen wisely.  I may not have fully understood how a controller writes/reads data to the disks. 

 

My thought, when originally setting up my RAID 0, and my RAID 6 was that I'm working with files that are minimum (DNG) 30-60MB in size, and could max out at about 30GB if I really wanted to shoot something long.  My thought was to create a stripe size (1024)  that was large so that the files could be stretched across the disks, but less than if I chose a smaller stripe (default 64KB) to reduce seek times, or Latency, of the drives.  I suspect I may have been wrong in anticipating that.

 

I've read in the LSI recomendatiosn that using a 64KB stripe size is optimal for CacheCade, do you think that is still the case?  Also, the questions stands, without Cachecade, what is the best stripe to utilize for video work.  And I think this questions should be answered before that: What kind of read/write activity comes from video editing?  I assumed it was sequential...

 

[leadeater]

4 hours ago, Strikermed said:

@leadeater

Wow, you are just a wealth of information.  I think to better understand this I will need to do some real world tests for myself, but it seems that I may not have chosen wisely.  I may not have fully understood how a controller writes/reads data to the disks. 

 

My thought, when originally setting up my RAID 0, and my RAID 6 was that I'm working with files that are minimum (DNG) 30-60MB in size, and could max out at about 30GB if I really wanted to shoot something long.  My thought was to create a stripe size (1024)  that was large so that the files could be stretched across the disks, but less than if I chose a smaller stripe (default 64KB) to reduce seek times, or Latency, of the drives.  I suspect I may have been wrong in anticipating that.

 

I've read in the LSI recomendatiosn that using a 64KB stripe size is optimal for CacheCade, do you think that is still the case?  Also, the questions stands, without Cachecade, what is the best stripe to utilize for video work.  And I think this questions should be answered before that: What kind of read/write activity comes from video editing?  I assumed it was sequential...

 

When using CacheCade it's best to stick with their recommendations, they would have done a lot of testing to find the optimal setup. Also for SSDs a smaller strip size works well as there is no seek penalty increasing latency.

 

Video editing is a bit of a mixture and depends on what you are doing. The raw footage in the editor untouched would be sequential, except when you are skipping back and forward using the timeline when it's a mixture of random and sequential. Once you start adding layers and effects you are adding more data streams so becomes more and more random in workload, the application should be doing a fair amount of memory caching to help with this though which is why video editing needs so much ram.

 

Once your SSDs turn up I would run your tests on one of these to get a best case baseline, if it doesn't work for some reason on that then there is an issue that needs investigating. If a pure SSDs can't perform then CacheCade has no hope. Run the test with just one SSD using a motherboard SATA connector then do it again on the LSI controller using both passthourgh JBOD and again using a single SSD RAID 0.

[Strikermed]

50 minutes ago, leadeater said:

When using CacheCade it's best to stick with their recommendations, they would have done a lot of testing to find the optimal setup. Also for SSDs a smaller strip size works well as there is no seek penalty increasing latency.

 

Video editing is a bit of a mixture and depends on what you are doing. The raw footage in the editor untouched would be sequential, except when you are skipping back and forward using the timeline when it's a mixture of random and sequential. Once you start adding layers and effects you are adding more data streams so becomes more and more random in workload, the application should be doing a fair amount of memory caching to help with this though which is why video editing needs so much ram.

 

Once your SSDs turn up I would run your tests on one of these to get a best case baseline, if it doesn't work for some reason on that then there is an issue that needs investigating. If a pure SSDs can't perform then CacheCade has no hope. Run the test with just one SSD using a motherboard SATA connector then do it again on the LSI controller using both passthourgh JBOD and again using a single SSD RAID 0.

I will do that, thanks for the break down.  I'm also going to utilize PerfMon to monitor it. 

 

Thanks!

[Strikermed]

On 7/12/2016 at 10:07 PM, leadeater said:

 

This is probably a good time to start going a little more in-depth in to storage I/O and how to manage/monitor performance. Disk utilization isn't really a good indicator of anything other than the disks are doing something all of the time during the sample period/points, it doesn't say what kind of demand or performance stress is happening.

 

 The other factor is the size of the I/O request and the pattern but I'll cover that later.

 

Information below is for performance counters in Windows Perfmon

 

The most important performance metric that you need to look at is avg disk sec/transfer which directly translates to latency. Storage latency and I/O request size have a very important relationship. When you are doing your tests monitor this perf counter and see how high it is, what this counter shows is how long it is taking for each storage transaction request where lower is better.

 

The other important performance metric that you should monitor is Current Disk Queue Length and Average Disk Queue Length. A high disk queue alone is not an indicator of a performance issue and this can be as high as the number of disk spindles in an array without issue. What this perf counter shows is how many storage transaction requests are pending in the queue waiting for disk access, if this is high but the avg disk sec/transfer is low then most of the time you are fine. This just means there is a lot of storage transactions being processed and it is happening happily and quickly. When the Current Disk Queue Length is high and avg disk sec/transfer is also high this means there is a performance issue and storage transactions and piling up in the queue and applications are getting very slow response times to storage access.

 

https://blogs.technet.microsoft.com/askcore/2012/03/16/windows-performance-monitor-disk-counters-explained/

 

Information below is how to calculate storage throughput/goodput 

 

As mentioned avg disk sec/transfer and storage I/O Size have an important relationship. You can use these two pieces of information to calculate the actual throughput you have.

 

Example 1:

avg disk sec/transfer:  0.005 (5ms)

I/O Size: 64KB

Throughput: (1/0.005) * 64 = 12800 KB/s or 12.5MB/s

 

Example 2:

avg disk sec/transfer:  0.005 (5ms)

I/O Size: 1024KB

Throughput: (1/0.005) * 1024 =204800 KB/s or 200MB/s

 

Example 3:

avg disk sec/transfer:  0.02 (20ms)

I/O Size: 1024KB

Throughput: (1/0.02) * 1024 = 51200 KB/s or 50MB/s

 

As you can quickly see this is why SSDs are so fast, it's all down to latency. Many HDDs can also have very high throughput as long as the access method is very sequential, random access patterns cause the latency to go up as the read/write heads have to seek and lock where an SSD does not.

 

What is hard about this is it can often be hard to know what your I/O size is, and this can vary a lot depending on application and partition allocation unit size.

 

Information below is on RAID strip size and stripe width/size sizing

 

I'll sum this one up quickly since there is a ton of information online, in general you want your strip size larger than your I/O size. Some hardware vendors use stripe width but actually mean strip size. When you set your strip size to say 64KB I/O is broken up in to 64KB chunks (unless it is smaller). The number of disks (stripe width) * strip size = stripe size.

 

 

Where this is important is that it impacts storage latency, but also sequential throughput. If your strip size is smaller than your I/O size to read or write data multiple HDD read/write heads have to seek and lock and the more that has to do this the longer the I/O request takes, you only want to hit one disk. The down side to this is you get lower theoretical throughput, I say theoretical as latency directly impacts this.

 

Storage I/O will still balance across the disks giving increased performance/throughput but no single I/O request will span more than one disk. If you are doing highly sequential workloads you may want to span multiple disks but this really only works if the I/O size is large enough to utilize the multiple disks and the latency hit does not outweigh the gain.

 

--------------------------------------------------

 

So on to your issue about stuttering and dropped frames what I think is happening is either the different formats use different size I/O and the smaller I/O combined with high latency is not giving you enough throughput for smooth playback, and/or the other possibility is the different formats compress more, high CPU load, but contain the same effective bit rate/quality. I say this since the raw footage plays without issue so if it's not I/O size then it has to be compression.

 

The smaller I/O size and latency seems to make a bit of sense to me regarding you trying the half resolution but still getting stutters.

 

For this CacheCade may help a lot since this is what it is designed to help with and SSDs have very good latency, the only problem is caching works much better for writes than reads. There is a term called cache warming which really only applies to reads so any data not in the cache will take time to be moved up in to the cache.

Ok, so I'm running different codecs through Premiere, and looking at the results, but I'm having trouble decypering the results, and locating my issue.  When looking at current Queue Length, I see values from .038 to 3.000

 

When looking at avg Queue Length I see as low as .345 and as high as 1.809. 

 

For transfers I'm seeing as low as .000 and as high as .022

 

I'm not understanding what is high and what is low in these situations to wrap my head around whether it's a good throughput or not.  I did my tests with DNxHR codec and DNxHR RGB 444 which is a much higher bit rate, and essentially lossless (so much larger file).

 

Is this enough info to help with diagnostic?

[leadeater]

6 hours ago, Strikermed said:

Ok, so I'm running different codecs through Premiere, and looking at the results, but I'm having trouble decypering the results, and locating my issue.  When looking at current Queue Length, I see values from .038 to 3.000

 

When looking at avg Queue Length I see as low as .345 and as high as 1.809. 

 

For transfers I'm seeing as low as .000 and as high as .022

 

I'm not understanding what is high and what is low in these situations to wrap my head around whether it's a good throughput or not.  I did my tests with DNxHR codec and DNxHR RGB 444 which is a much higher bit rate, and essentially lossless (so much larger file).

 

Is this enough info to help with diagnostic?

What were you testing it on, the single SSD?

 

The sec/transfer seems rather good, suggesting it is an SSD. The latency is sub ms to 22ms which is getting a little high but not that bad, to put it in context Microsoft recommend database file storage latency should not be higher than 20ms and transaction log less than 10ms.

 

Using perfmon to look at the SSD tests likely isn't going to be too useful, just write down what you see. It will be more useful when using the HDD's and different RAID strip sizes etc.

 

You can also use the below tools to test the raw performance of different storage configurations.

 

Here's an HD Tune latency test of my 256GB 840 Pro

 

Here's another tool you can use to do raw performance tests, what is shown is a 4 SSD Storage Space RAID 0 equiv, 

[Strikermed]

@leadeater

Thanks!  Those tools help really visualize latency and IO.  I did notice recently that Crystal Mark is not reliable, as it is showing odd results with RAID 0, and RAID 6.  Writes were larger than Reads, and RAID 0 was performing under RAID 6 by about half. 

 

Current test show that my SSD latency is similar to the image you posted last.  I found that my RAID 0 and RAID 6 had very similar latency results with most request staying in the 20ms range, but then when it got to the 1MB test it went into the 40ms range.  I suspect, as you did that this is my problem with jumping from clip to clip.

 

After running these tests I also ran various test with different codecs.  I rendered the same clip from my Inspire 1 that was 4K 8bit .mov files.  It was 45 seconds long, and I played those files on the time line, allowing the drives to settle before playing each clip.  I tried several codecs including:

 

AVI - Uncompressed at 4K

Cineform - Max Quality 4k

Cineform - regular 4k

DNxHR  HQ  4K

DNxHR HQ UHD 4K

DNxHR HQX 4K

DNxHR LB 4K

DNxHR RGB 444 4K

DNxHR RGB 444 UHD

JPEG 2000 MXF OP1a - RGB 444 12 bit BCS@L5 4K

JPEG 2000 MXF OP1a - YUV 422 10 bit BCS@L5 4K

JPEG sequence 4K

PNG Sequence 4K

Quicktime - ANimation

Quicktime - Cinepak

Quicktime - None Uncompressed RGB 8bit

Quicktime - Uncompressed YUV 8bit 422

Quicktime - Uncompressed YUV 10bit 422

TIFF Sequence

 

I'll report on what worked, if I omit a category, it's because I had dropped frames.  I'll talk about the omitted in my conclusion, which raised more questions.

 

For my RAID 0 with 4 drives and single SSD:

 

Quicktime - Uncompressed YUV 8 bit 422.  According to task manager this ran at 540MB/s and utilized 100% of the hard drive.  This responded better on the SSD due to the latency being better.  Once I got the RAID 0 array to play the clip, queueing it up, it would play without any issue.

 

JPEG 200 MXF op1a YUV 422 10bit BCS@L5 - This codec played at 80MB/s on average at 12% utilization of the RAID 0 and SSD. 

 

DNxHR HQ 4K - This one ran at 145-200 MB/s at 30% utilization on both drives. (Note: this codec has choppy scrubbing ability)

 

DNxHR HQ UHD at 4K - This ran at 130MB/s at 23% utilization (Note: this codec has choppy scrubbing ability)

 

DNxHR HQX at 4K - This ran at 150MB/s at 28% utilization.  (Note: this codec has choppy scrubbing ability)

 

DNxHR LB at 4k - This one ran from 30 to 70 MB/s at 15% utilization.  (Note: this codec has decent scrubbing ability)

 

Cineform Max Quality at 4K - played at 150MB/s, but it would chug along at 95% utilization.  (This is part of the mystery in the conclusion)

 

Cineform at 4K - ran at 30-40 MB/s at 7% utilization

 

AVI Uncompressed - Ran at 170 MB/s at 30% utilization.  (Need to retest with this as it scaled my image to a lower resolution, with black filling the 4K extra area.)

 

Conclusion - I have found a few codecs that scrub well and play well with either drive.  Those being Cineform, DNxHD/HR, and a few uncompressed formats.  The strange thing that I found is when I got into better color spaces, like codecs with 444 color space.  Those didn't need nearly as high transfer rates, but they would chug along at times in Premiere CC 2015.3.  They also drew 95 to 100% CPU from an i7 3930K (six core, 12 hyperthreaded). 

 

To sum this up, I think Cachecade will help with timeline performance, and projects that I open, while RAID 0 will continue to provide fast IO before the cache can be created.  Here's the million dollar, well more like a $1500 question.  Should I even use RAID0, or should I commit to growing my RAID 6 array and use the SSD with cachecade (that is, if testing works as expected). 

 

As of right now my workflow includes copying files to RAID 0 to work on, and then archiving them on RAID 6, along with backing up my NAS to the RAID6.  If I'll get the same results with RAID 6 in video editing as I do with RAID0, then it would be pointless to use RAID 0, especially if the IO isn't extremely high.  I could always restructure the RAID0 and create a RAID 5 for exports or some other purpose.  Any thoughts on that mentality?

 

Next tests will come with 2 SSD in RAID 0, and then utilizing cachecade.

[leadeater]

1 hour ago, Strikermed said:

To sum this up, I think Cachecade will help with timeline performance, and projects that I open, while RAID 0 will continue to provide fast IO before the cache can be created.  Here's the million dollar, well more like a $1500 question.  Should I even use RAID0, or should I commit to growing my RAID 6 array and use the SSD with cachecade (that is, if testing works as expected). 

 

As of right now my workflow includes copying files to RAID 0 to work on, and then archiving them on RAID 6, along with backing up my NAS to the RAID6.  If I'll get the same results with RAID 6 in video editing as I do with RAID0, then it would be pointless to use RAID 0, especially if the IO isn't extremely high.  I could always restructure the RAID0 and create a RAID 5 for exports or some other purpose.  Any thoughts on that mentality?

 

Next tests will come with 2 SSD in RAID 0, and then utilizing cachecade.

Personally I would use RAID 6 with write-back cache on with a BBU or flash cache (supercap). For reads RAID 6 is very good and for writes they will hit the write-back cache which is even faster than an SSD but very small. Combine this with CacheCade and there will be very few scenarios where this wouldn't have the required performance, extremely demanding read workloads.

[Strikermed]

@leadeater

I'm thinking this as well, with 2 drive redundancy, it would be super effective in a product environment.  I say this with the experience I'm having now expanding my RAID 6 with a 5th drive.  It is taking on average 12 hours for 20%, which leads me to believe it will take 2 1/2 days to replace a bad drive.  While the drive is rebuilding, the entire RAID card is ineffective, as other RAID sets are extremely slowed by the rebuild.  The RAID 0 SSD would only give me 8.5MB/s while the rebuild was happening, and wouldn't play media files without waiting 30 secs for it to play.  Leaving me with more redundancy, would allow me to back up the drives, and finish a project before I replace a drive.

 

Now, I have 4 2TB WD black, and I need to figure out what I want to do with them :).  I'll most likely play around with stripe sizes, and use them as my test subjects before I fill up the RAID 6 beyond the NAS storage I have on hand that I use to locally back up files.  Thanks for the help, and I'll report back with my findings about stripe size in terms of 4k and HD video editing, with RAW and non-RAW workflows. 

 

Thanks!  Time to take a month and test out Codec's breaking points and CacheCade Pro 2.0!

[scottyseng]

7 hours ago, Strikermed said:

-snip-

Haha, happy to hear you got things sorted out. Did you ever get the expander to work? Yeah, I was going to reply, but leadeater pretty much posted everything you needed.

 

Also, you might check your background task rates for your LSI MegaRAID controller using the MegaRAID Storage Manager software. By default the background tasks (rebuilding, initializing, and consistency check) are at 10% if I remember right. I remember sitting around for 7 days, waiting for my four drive RAID10 array to get built....I actually found this setting and changed it to 100% (Though keep in mind, you're telling the controller to dedicate 100% of the speed towards the background tasks...your working speeds will drop) and that same four drive RAID10 array took only 10 hours to build. So by some chance you have a dead drive, you might try adjusting this background task rate to speed up the rebuild / expansion of the array. I leave mine at 100% because I honestly just use my NAS for coding, music, and anime...none of which are taxing to it at all.

 

The also nice bonus of RAID6 is that you can expand down the road without having to make a new array (RAID10 doesn't do this...).

 

I have to admit though, you did some really thorough testing (With the codecs, stripe sizing, and disk latency/speed), I'm sure if you made your results into a video, you could give Linus there a run for his money. haha.

[Strikermed]

Hi @scottyseng, I was able to get the expander with absolutely no trouble.  All my RAID sets identified themselves just like it would with an unmanaged network switch would do with a small network of computers.  Mounting it was a little tricky, but I was able to find 2 screw holes on my Rosewill 12 Bay 4U case that linedup with 2 of the holes of the expander so I could mount it semi-properly.  I supplied power to it via 4pin, and I have both the SFF8087 from my LSI 9266 8i going to the first 2 ports on the expander (http://www.newegg.com/Product/Product.aspx?Item=9SIA2F84293102) . 

 

I took the bracket off, and mounted it flat, as I didn't have a 4x pci slot available, and I had 3 of my 16x/8x pci slots full with graphics card, 10GbE network card, and RAID card.  I broke off those to my RAID 0 with 4 drives, and my RAID 6 with 5 drives (soon after the reconstruction is done), and 2 SSD's currently in RAID 0.

 

Thanks for the tip on the background tasks, I found that setting and changed it to 90%, but I didn't see any kind of change in the data rate of the rebuilt, nor did I see a faster build over a period of time.  It's a 6TB drive that's rebuilding, so I suspect it will take time...  The only trouble I have is the incredible performance hit I took on my other drive groups.  Both my SSD and HDD RAID 0 groups are chugging along at like 10MB/s.  It seems like the card is already heavily building this 5th drive, and doesn't have any resources for the others...  I'm curious if there is a way to see whether it's processor and various other resources are consumed by the rebuild.  This all happened prior to adjusting the reconstruction rate from 30 to 90%.  If anyone knows if this is normal, please let me know. Thanks.

 

P.S. I wouldn't dare try and compete with Linus, his team really works hard to test some formats, and I'm sure his tests had a lot more merit than just throwing codecs at a wall and see which ones stuck.  It did open my eyes, but I have more configuring to do before I nail that down as well.

[leadeater]

59 minutes ago, Strikermed said:

Thanks for the tip on the background tasks, I found that setting and changed it to 90%, but I didn't see any kind of change in the data rate of the rebuilt, nor did I see a faster build over a period of time.  It's a 6TB drive that's rebuilding, so I suspect it will take time...  The only trouble I have is the incredible performance hit I took on my other drive groups.  Both my SSD and HDD RAID 0 groups are chugging along at like 10MB/s.  It seems like the card is already heavily building this 5th drive, and doesn't have any resources for the others...  I'm curious if there is a way to see whether it's processor and various other resources are consumed by the rebuild.  This all happened prior to adjusting the reconstruction rate from 30 to 90%.  If anyone knows if this is normal, please let me know. Thanks.

RAID 6 on the LSI 9266 is very demanding for that card, it has an LSI SAS2208 controller which is a dual core 800MHz chip so during a rebuild it's CPU is likely pretty maxed out. The 12Gb LSI 9361/SAS3108 is 50% faster CPU performance

 

Normally there is also a setting you can turn on with a name like "RAID 6 advanced performance for degraded array", I don't know cant remember it and each vendor gives it a different name. Turning this one during a single disk failure keeps the performance of the array at almost normal levels, not sure what effect this has on other arrays on the card.

 

Stripe size effects the CPU load also btw, the smaller the stripe size the more work the RAID CPU has to work. Try increasing the stripe size for the RAID 0 SSD array and see if it goes faster during the rebuild, not much you can do about the existing RAID 6 while it's rebuilding. Also not sure if that card supports online RAID level migration for RAID 6 to RAID 6 of a different stripe size.

[wpirobotbuilder]

On 7/11/2016 at 0:46 PM, Strikermed said:

My RAID controller has 1GB of onboard Cache.  What does LSI call these cache devices as mine doesn't have swappable RAM (I wish it did as I would put the biggest chip I can in there).

I haven't used a 9266 card before, if the cache is onboard then you can't swap it out (I believe there are some cards which you can add caching to later on). 1GB of cache is probably the biggest you'll find on a reasonably priced card, since it's basically just DDR2/3 ram chips on there.

 

On 7/11/2016 at 0:46 PM, Strikermed said:

Turning off Hyperthreading is interesting, as I haven't tried that, and I've heard of some people finding success in certain applications.  I can try that as well for playback performance.  I'll give that a go prior to attempting Cachecade to see what kind of differences I see.

I'm not sure if you'll see a benefit to performance, I would turn off hyperthreading, then run your workflow, and see if your CPU usage is 90% or higher (whereas before, with hyperthreading on, you would see 60%). If this is true, then you are maxing out the physical cores -- no amount of threading can get you more processing. If this is not true, then your CPU may not be the bottleneck.

 

Did you try this, and what did you see?

[scottyseng]

12 hours ago, Strikermed said:

-snip-

Ah, happy the expander works.

 

So in total, you have:

RAID0 with four drives

RAID6 with five drives

RAID0 with two SSDs (Cachecade)

 

Yeah, I'm really sure like what leadeater said, you're hitting the limits of the RAID controller. I myself noticed that my second RAID6 array took significantly longer than the first to build (First RAID6 of eight drives took 10 hours, second RAID6 of six drives took 7 days). I would also double check the temperatures of the RAID card, since it does throttle under high temps (I got mine to throttle when I had it sandwiched between two FirePros GPUs in crossfire...poor thing got so hot I couldn't touch it...I had to put a fan on it.) Even now in the server chassis, I still have a fan blowing straight onto the RAID card to keep it cool.

 

What's interesting is that for me, I didn't get a performance hit while building my second RAID6 array, even with it set to 100% task rates. However, you have three arrays, which I'm sure is even harder on your RAID card.

[Strikermed]

18 hours ago, leadeater said:

RAID 6 on the LSI 9266 is very demanding for that card, it has an LSI SAS2208 controller which is a dual core 800MHz chip so during a rebuild it's CPU is likely pretty maxed out. The 12Gb LSI 9361/SAS3108 is 50% faster CPU performance

 

Stripe size effects the CPU load also btw, the smaller the stripe size the more work the RAID CPU has to work.

At this point I haven't touched stripe size, as I would have to new builds and eliminate my RAID sets all together, the 9266 card doesn't allow for stripe size migration.  However, it does allow for online migrations, which I did that when I first started with a RAID 5 and upgraded to RAID 6.

 

It's unfortunate that I'm seeing high utilization, but this makes perfect sense since my Synology Disk station does the same thing if rebuilding, scanning for viruses, or anything else heavy on the processor, and it's a dual core also.  The GUI get's very slow just like the Mega RAID Software is.

 

Quote

@wpirobotbuilderI

'm not sure if you'll see a benefit to performance, I would turn off hyperthreading, then run your workflow, and see if your CPU usage is 90% or higher (whereas before, with hyperthreading on, you would see 60%). If this is true, then you are maxing out the physical cores -- no amount of threading can get you more processing. If this is not true, then your CPU may not be the bottleneck.

 

Did you try this, and what did you see?

I did not try this as I haven't seen a logical reason to do it yet... I know that some codecs utilize CPU more than others, and I know that some utilize disk speed more than others.  I would probably attempt this once I get my RAID configuration locked down.  I need to figure out stripe size, and I need to figure out my read and write and IO policies to get the best performance, but most of all it's the stripe size as that can't be altered once the RAID group is set.  That will most likely come when I'm fine tuning my codec choice.

 

Quote

@scottyseng

What's interesting is that for me, I didn't get a performance hit while building my second RAID6 array, even with it set to 100% task rates. However, you have three arrays, which I'm sure is even harder on your RAID card.

I have RAID 0 with 4 drives, RAID 6 with 5 drives reconstruting/expanding, and RAID 0 with 2 SSD without cachecade right now.

 

Can you show me/link me to a fan that you used for your RAID card?  Mine isn't directly next to anything because it has a 2x slot between my video card and the RAID card,  Above the RAID card is my network card, again separated by a 2x slot.

 

I thought it was weird I saw poor performance, even with that many RAID sets, because prior to the build everything ran fine, and was working.  This leads me to believe that how many RAID sets you have doesn't matter, or at least as long as you aren't accessing them all at once. The throughput of my card is 12gbs, so I should be able to get that kind of performance for what ever RAID I have.  And now that I write that out, it may be faulse, haha.  If someone could answer that scientifically, that would be awesome.  

 

I think the reconstruction/calculations are what's taking up all the RAID card's processing power.  I'm hoping if I decide to destroy this RAID for a smaller stripe size (right now 1MB stripe, and I'm thinking 64kb if tests of the RAID 0 show good results).

[scottyseng]

44 minutes ago, Strikermed said:

I have RAID 0 with 4 drives, RAID 6 with 5 drives reconstruting/expanding, and RAID 0 with 2 SSD without cachecade right now.

 

Can you show me/link me to a fan that you used for your RAID card?  Mine isn't directly next to anything because it has a 2x slot between my video card and the RAID card,  Above the RAID card is my network card, again separated by a 2x slot.

 

I thought it was weird I saw poor performance, even with that many RAID sets, because prior to the build everything ran fine, and was working.  This leads me to believe that how many RAID sets you have doesn't matter, or at least as long as you aren't accessing them all at once. The throughput of my card is 12gbs, so I should be able to get that kind of performance for what ever RAID I have.  And now that I write that out, it may be faulse, haha.  If someone could answer that scientifically, that would be awesome.  

 

I think the reconstruction/calculations are what's taking up all the RAID card's processing power.  I'm hoping if I decide to destroy this RAID for a smaller stripe size (right now 1MB stripe, and I'm thinking 64kb if tests of the RAID 0 show good results).

Ah, I kind of did my fan the ghetto way...the server is horizontal so I just have a 120mm Be Quiet Pure Wings fan just free standing by itself...ghetto, but it works. haha. When my RAID card was in a normal PC though, I wasn't able to figure out a way to cool it (It was sandwiched between the GPUs). You might be able to have a fan just stay on top of the slots, but it depends on the slot configuration. Though if your chassis has enough airflow, you should be good (I had to add the above fan before I removed the SuperMicro stock 6000RPM 80mm fans and replaced them with a bunch of 120 /140mm Be Quiet Pure Wing fans...I have this in my room so the 60dB the stock fans were making was hard to sleep with...

 

Well, what's strange is that for my RAID card, I was building a RAID6 (the six drive array of WD Reds) and I ran crystal disk mark and did not notice any performance drop when transferring large files (Got the full 1GB/s when transferring files or 550MB/s when transferring from the OS SSD drive from the primary RAID6 array of eight WD Re drives). Though I was building a RAID6 array, I wasn't expanding an existing one (I would assume both are taxing on the RAID card...I only have two arrays as well and no SSDs, while you have two arrays and a SSD array...it's a lot of load).

 

Yeah, I'm curious to see what RAID stripe size will affect your performance. I left mine at the default 64kb (I have a mix of large and small files on my array). I hope after the build is done, your speeds return to normal.

[leadeater]

7 minutes ago, scottyseng said:

Well, what's strange is that for my RAID card, I was building a RAID6 (the six drive array of WD Reds) and I ran crystal disk mark and did not notice any performance drop when transferring large files (Got the full 1GB/s when transferring files or 550MB/s when transferring from the OS SSD drive from the primary RAID6 array of eight WD Re drives). Though I was building a RAID6 array, I wasn't expanding an existing one (I would assume both are taxing on the RAID card...I only have two arrays as well and no SSDs, while you have two arrays and a SSD array...it's a lot of load).

No idea if this is true or not but I would think an expand is more CPU demanding than a rebuild. With an expand you have to recalculate and re-balance all data and both parity bits, a rebuild is just reconstructing from 1 or 2 parity stripes and writing to a single disk or 2 disks.

[scottyseng]

7 minutes ago, leadeater said:

No idea if this is true or not but I would think an expand is more CPU demanding than a rebuild. With an expand you have to recalculate and re-balance all data and both parity bits, a rebuild is just reconstructing from 1 or 2 parity stripes and writing to a single disk or 2 disks.

Yeah, I can imagine expansion taking more CPU. I wish I had one or two more drives to expand my RAID6 of eight drives to see how much of a performance hit I get while expanding, but no funds for that yet.

[Strikermed]

11 hours ago, scottyseng said:

Well, what's strange is that for my RAID card, I was building a RAID6 (the six drive array of WD Reds) and I ran crystal disk mark and did not notice any performance drop when transferring large files (Got the full 1GB/s when transferring files or 550MB/s when transferring from the OS SSD drive from the primary RAID6 array of eight WD Re drives). Though I was building a RAID6 array, I wasn't expanding an existing one (I would assume both are taxing on the RAID card...I only have two arrays as well and no SSDs, while you have two arrays and a SSD array...it's a lot of load).

 

Yeah, I'm curious to see what RAID stripe size will affect your performance. I left mine at the default 64kb (I have a mix of large and small files on my array). I hope after the build is done, your speeds return to normal.

My speeds did return, now I'm just waiting for the background initialization to complete, although all my speed is there, which is reassuring.  @leadeater had it right when he said:

Quote

With an expand you have to recalculate and re-balance all data and both parity bits, a rebuild is just reconstructing from 1 or 2 parity stripes and writing to a single disk or 2 disks.

I thought about this, and I can imagine that it would take a lot of computing power.  Just creating a new RAID group doesn't take much, but if you expand, it has to redistribute the data, while still keeping it intact, and again re-balacing like he said.

 

I think that's a lesson I'll remember if I'm trying to play with different RAID levels, I'll just ensure I have a back up, and destroy the RAID and create a new one instead of migrating, and expanding in the future.  I currently have everything backed up on my Synology NAS, and via the web, so I could have just cut my reconstruction time to a single day and created a new RAID 6 group.  I'll do that with my testing the Stripe size with the RAID 0 sometime this week. 

 

@scottyseng What RAID Card are you using?

 

[scottyseng]

4 hours ago, Strikermed said:

-snip-

A 9260-8i CV (Mine's the basic mid range 92xx series card). I wish I had another one or two drives to expand my primary RAID6 array because I'm curious on how much damage the speeds would take. I only have two arrays of hard drives as well, while you have three of them (One of them being an SSD array). Your setup is quite a bit harder on the RAID card.

 

Yeah, I'm curious to see how your times of destroying and building an array would be since you have two other active arrays while you're doing the build.

[Strikermed]

I'll let you know as I find out.  Right now I need to get a few edits done before I can break a RAID, plus I need to make sure that everything is backed up via the web, which until Google Fiber gets to my street, is going to be at a slow 2.5MB/s.