Plans to build a 12 drive SSD array

[Windows7ge]

4 minutes ago, Electronics Wizardy said:

what type of files?

The test file I just used to get those scores was a 9.5GB .mp4 file. The array as a whole has many many different files so I take it compression would work on some but not others. I can try it again with an .ISO or an archive (.rar or .zip) if those are compressible.

[Electronics Wizardy]

Just now, Windows7ge said:

The test file I just used to get those scores was a 9.5GB .mp4 file. The array as a whole has many many different files so I take it compression would work on some but not others. I can try it again with an .ISO or an archive (.rar or .zip) if those are compressible.

mp4 files aren't compressable, same with archives. Normally compression work well for things like vm images, data bases, and text. 

[Windows7ge]

1 minute ago, Electronics Wizardy said:

mp4 files aren't compressable, same with archives. Normally compression work well for things like vm images, data bases, and text. 

I do have a series of VM backups from Oracle VM VirtualBox projects in college. They're .ova files. 2~3GB in size. I have no databases but I do have numerous .txt files for all sorts of things.

[Electronics Wizardy]

1 minute ago, Windows7ge said:

I do have a series of VM backups from Oracle VM VirtualBox projects in college. They're .ova files. 2~3GB in size. I have no databases but I do have numerous .txt files for all sorts of things.

Id just leav it on and see what the compression value changes to. For my main storage my average compression ratie is about 1.08 and I have lots of videos. Measuring the performance is hard with compression, but it also saves disk space that is nice.

[Windows7ge]

12 minutes ago, Electronics Wizardy said:

Id just leav it on and see what the compression value changes to. For my main storage my average compression ratie is about 1.08 and I have lots of videos. Measuring the performance is hard with compression, but it also saves disk space that is nice.

Saving disk space is always nice. It will be very nice with SSD storage considering how expensive it is. I'll leave it to run but I have to say so far it is showing no indicator at all that it's working. Looking at the server itself the drive activity LED indicators aren't blinking any more than they usually do so I can't say it's compressing any of the data at all.

[Electronics Wizardy]

2 minutes ago, Windows7ge said:

Saving disk space is always nice. It will be very nice with SSD storage considering how expensive it is. I'll leave it to run but I have to say so far it is showing no indicator at all that it's working. Looking at the server itself the drive activity LED indicators aren't blinking any more than they usually do so I can't say it's compressing any of the data at all.

the lazy test is just to run dd if=/dev/zero of=testfile.zero bs=1m. This should be very fast.

[Windows7ge]

15 minutes ago, Electronics Wizardy said:

the lazy test is just to run dd if=/dev/zero of=testfile.zero bs=1m. This should be very fast.

Mentioning that this would create a file infinite in size would have been appreciated. I had to do Ctrl+C after I noticed the file exceeded 1TB. However it does state that on disk it is only 1MB.

 

As a side note. 1MB blocks yielded 3.2GB/s...seems a tad bit inaccurate for a mechanical array of 8 drives.

[Electronics Wizardy]

34 minutes ago, Windows7ge said:

Mentioning that this would create a file infinite in size would have been appreciated. I had to do Ctrl+C after I noticed the file exceeded 1TB. However it does state that on disk it is only 1MB.

 

As a side note. 1MB blocks yielded 3.2GB/s...seems a tad bit inaccurate for a mechanical array of 8 drives.

Yea... It will do that.

 

Thats compression working for you and the speed at which the cpu can compress files. Way faster than the disks. here.

[Windows7ge]

2 hours ago, Electronics Wizardy said:

Thats compression working for you and the speed at which the cpu can compress files. Way faster than the disks. here.

Hm, it still doesn't appear to be doing anything for the data on the array nor data I add except for dd test files. I'm left to assume a majority of my data is non-compressible? That or somethings wrong with the server. For the heck of it I'll leave it on if it's overall beneficial. I wonder if it'd be necessary for me to dump and rebuild the array in order to get lz4 running correctly.

 

For no more reason than to see what would happen I wrote this script and ran it inside the array.

dd if=/dev/urandom of=test512bytes bs=512 count=8388608 ; sync ;
dd if=/dev/urandom of=test1kb bs=1024 count=4194304 ; sync ;
dd if=/dev/urandom of=test2kb bs=2048 count=2097152 ; sync ;
dd if=/dev/urandom of=test4kb bs=4096 count=1048576 ; sync ;
dd if=/dev/urandom of=test8kb bs=8192 count=524288 ; sync ;
dd if=/dev/urandom of=test16kb bs=16384 count=262144 ; sync ;
dd if=/dev/urandom of=test32kb bs=32768 count=131072 ; sync ;
dd if=/dev/urandom of=test64kb bs=65536 count=65536 ; sync ;
dd if=/dev/urandom of=test128kb bs=131072 count=32768 ; sync ;
dd if=/dev/urandom of=test256kb bs=262144 count=16384 ; sync ;
dd if=/dev/urandom of=test512kb bs=524288 count=8192 ; sync ;
dd if=/dev/urandom of=test1mb bs=1048576 count=4096 ; sync ;
dd if=/dev/urandom of=test2mb bs=2097152 count=2048 ; sync ;
dd if=/dev/urandom of=test4mb bs=4194304 count=1024 ; sync ;
dd if=/dev/urandom of=test8mb bs=8388608 count=512 ; sync ;
dd if=/dev/urandom of=test16mb bs=16777216 count=256 ; sync ;
dd if=/dev/urandom of=test32mb bs=33554432 count=128 ; sync ;
dd if=/dev/urandom of=test64mb bs=67108864 count=64 ; sync ;
rm test512bytes test1kb test2kb test4kb test8kb test16kb test32kb test64kb test128kb test256kb test512kb test1mb test2mb test4mb test8mb test16mb test32mb test64mb

The resulting output was an interesting benchmark

8388608+0 records in
8388608+0 records out
4294967296 bytes transferred in 93.420209 secs (45974713 bytes/sec)

4194304+0 records in
4194304+0 records out
4294967296 bytes transferred in 64.683139 secs (66400107 bytes/sec)

2097152+0 records in
2097152+0 records out
4294967296 bytes transferred in 50.486010 secs (85072425 bytes/sec)

1048576+0 records in
1048576+0 records out
4294967296 bytes transferred in 43.274379 secs (99249658 bytes/sec)

524288+0 records in
524288+0 records out
4294967296 bytes transferred in 40.603359 secs (105778620 bytes/sec)

262144+0 records in
262144+0 records out
4294967296 bytes transferred in 38.675974 secs (111050011 bytes/sec)

131072+0 records in
131072+0 records out
4294967296 bytes transferred in 37.857852 secs (113449842 bytes/sec)

65536+0 records in
65536+0 records out
4294967296 bytes transferred in 37.262448 secs (115262618 bytes/sec)

32768+0 records in
32768+0 records out
4294967296 bytes transferred in 36.733378 secs (116922743 bytes/sec)

16384+0 records in
16384+0 records out
4294967296 bytes transferred in 36.815393 secs (116662270 bytes/sec)

8192+0 records in
8192+0 records out
4294967296 bytes transferred in 36.647644 secs (117196274 bytes/sec)

4096+0 records in
4096+0 records out
4294967296 bytes transferred in 36.691257 secs (117056968 bytes/sec)

2048+0 records in
2048+0 records out
4294967296 bytes transferred in 36.558845 secs (117480935 bytes/sec)

1024+0 records in
1024+0 records out
4294967296 bytes transferred in 36.640542 secs (117218988 bytes/sec)

512+0 records in
512+0 records out
4294967296 bytes transferred in 36.700429 secs (117027714 bytes/sec)

256+0 records in
256+0 records out
4294967296 bytes transferred in 37.509314 secs (114504021 bytes/sec)

128+0 records in
128+0 records out
4294967296 bytes transferred in 37.889963 secs (113353695 bytes/sec)

64+0 records in
64+0 records out
4294967296 bytes transferred in 38.256744 secs (112266931 bytes/sec)

Between the block sizes 512KB & 8MB seems to be as good as it gets. If FreeNAS sets up the block size like any other system then mine is probably set to 64KB which checking this shows...huh. weird:

512             # sectorsize                                            
2000398934016   # mediasize in bytes (1.8T)                             
3907029168      # mediasize in sectors                                  
0               # stripesize                                            
0               # stripeoffset                                          
243201          # Cylinders according to firmware.                      
255             # Heads according to firmware.                          
63              # Sectors according to firmware.                        
WD-WMC6N0H11HCD # Disk ident.                                           
Not_Zoned       # Zone Mode

So the disk has a size of 512 but the array has a block size of...0? That's definitely incorrect.

[Electronics Wizardy]

10 minutes ago, Windows7ge said:

Hm, it still doesn't appear to be doing anything for the data on the array nor data I add except for dd test files. I'm left to assume a majority of my data is non-compressible?

It compresses all new data written, the existing data will not be compressed. You can do a manual compress doing as script that copies a files, then delete the old copy, but i wouldn't bother.

 

 

[Windows7ge]

4 minutes ago, Electronics Wizardy said:

It compresses all new data written, the existing data will not be compressed. You can do a manual compress doing as script that copies a files, then delete the old copy, but i wouldn't bother.

 

 

Yeah, not worth the hassle. I'll make sure to leave lz4 enabled on the new array when I build it. Copying all the data over will compress it all then the mechanical array can just be back-up and/or misc.

[jde3]

On 4/30/2018 at 8:12 AM, Windows7ge said:

kldstat

Id Refs Address            Size     Name
 1   80 0xffffffff80200000 2067000  kernel
 2    1 0xffffffff825ef000 ffd1c    ispfw.ko
 3    1 0xffffffff826ef000 7151     freenas_sysctl.ko
 4    1 0xffffffff82811000 592f     fdescfs.ko
 5    1 0xffffffff82817000 3337ee   vmm.ko
 6    1 0xffffffff82b4b000 30c4     nmdm.ko
 7    1 0xffffffff82b4f000 fabd     geom_mirror.ko
 8    1 0xffffffff82b5f000 47a1     geom_stripe.ko
 9    1 0xffffffff82b64000 ffc4     geom_raid3.ko
10    1 0xffffffff82b74000 16e03    geom_raid5.ko
11    1 0xffffffff82b8b000 5915     geom_gate.ko
12    1 0xffffffff82b91000 4e75     geom_multipath.ko
13    1 0xffffffff82b96000 828      dtraceall.ko
14    9 0xffffffff82b97000 41822    dtrace.ko
15    1 0xffffffff82bd9000 4883     dtmalloc.ko
16    1 0xffffffff82bde000 5aae     dtnfscl.ko
17    1 0xffffffff82be4000 67e1     fbt.ko
18    1 0xffffffff82beb000 58b81    fasttrap.ko
19    1 0xffffffff82c44000 1769     sdt.ko
20    1 0xffffffff82c46000 cf1e     systrace.ko
21    1 0xffffffff82c53000 ce87     systrace_freebsd32.ko
22    1 0xffffffff82c60000 53b6     profile.ko
23    1 0xffffffff82c66000 8496     ipmi.ko
24    1 0xffffffff82c6f000 f09      smbus.ko
25    1 0xffffffff82c70000 1bdde    hwpmc.ko
26    1 0xffffffff82c8c000 cfe9     t3_tom.ko
27    2 0xffffffff82c99000 45be     toecore.ko
28    1 0xffffffff82c9e000 15c71    t4_tom.ko
29    1 0xffffffff82cb4000 296e     uhid.ko
30    1 0xffffffff82cb7000 3620     ums.ko

dmesg | grep aesni

aesni0: <AES-CBC,AES-XTS,AES-GCM,AES-ICM> on motherboard

 

Ports? If ATTO worked with network drives I'd try testing it with that. As for accuracy I don't think the results would be swayed so far off that it'd make a monstrous difference if I'm off by one or two. Also in theory wouldn't a larger block size be less stress on the CPU(s). Kind of eliminate the core clock as a bottleneck.

Sorry, just getting back to this now.

 

It looks like cryptodev isn't loaded but that's ok because aesni0 shows hardware crypto on the cpu.

Ports is the package build system for FreeBSD. (similar to gentoo's portage that was inspired from it).. I don't know what would be a good drive benchmark but there is a section in ports for them (also available probably from pkg)

 

In one of your other posts you said you turned on LZ4 but it didn't show anything compressed.. LZ4 only compressed stuff when it's written to the disk. moving the data within the pool dosen't re-write it, it just changes the inode. LZ4 won't try to compress data that is already compressed also such as media files.. when it starts to write it if it isn't compressing then it just skips it and turns compression off (this is what you want)