Beginner music and content creation Server w/Compression algorithm advice

[Specter FXXK]

Hey guys, been a while since I've been on here!, anyways im looking to build a unique maybe server to store huge amounts of semi-medium files (mp3, WAV, ACID-WAV, AIFF, MP4 and PSD, AI, INDD, CAD, C4D, High res JPEGs/PNGs

also looking for best compression algorithm for storing/compressing PSD, AI, INDD, CAD, C4D Etc.

then also storing/compressing WAV.

 

I have over 3 TB of big wav files ranging from 30-500MB different library's of genre music making files, other files for other projects, because I don't know what to do in life right now lmao.

but yeah, I got a LSI MegaRAID on the way, heard it's pretty good for RAID 6 and another LSI card 9211-8i on the way too for my other NAS I wanna build or server. or just a backup card incase the first one dies for what ever reason XD

so what you guys recommend I got 2x1TB Seagate drives (home ones) and 3 random 500GB-1TB drives  different makers laying around and could use them as spares along the RAID 6 Array

also what compression algorithm? best?

mainly looking for best compression at a good or average speed (compressing) and decompression idc about atm 

but say I want both of both worlds what's good?

 

im currently using LZMA/LZMA2 and ZSTD compression algorithm

Edited March 17, 2021 by Specter FXXK
forgot to add the compression algorithms im using atm

[Windows7ge]

With those hardware controllers if I'm not mistaken you're going to need at minimum 5 drives of identical capacity and ideally rotational speed. Keep that in mind.

 

Why not Software RAID? I don't see a compelling reason for the expenditure on a hardware controller here.

 

Did you have an OS in mind? I think LZ4 on ZFS compresses pretty well. I'm not an audiophile so I don't know of the formats you listed which already incorporate compression and which don't. If they're already heavily compressed like with .MP3 adding another compression layer on-top of them isn't going to do much regardless of algorithm.

[Specter FXXK]

1 minute ago, Windows7ge said:

With those hardware controllers if I'm not mistaken you're going to need at minimum 5 drives of identical capacity and ideally rotational speed. Keep that in mind.

 

Why not Software RAID? I don't see a compelling reason for the expenditure on a hardware controller here.

 

Did you have an OS in mind? I think LZ4 on ZFS compresses pretty well. I'm not an audiophile so I don't know of the formats you listed which already incorporate compression and which don't. If they're already heavily compressed like with .MP3 adding another compression layer on-top of them isn't going to do much regardless of algorithm.

I have a 1 core CPU (AMD A4-7300 APU with Radeon HD Graphics) XD, I got more drives if they are required for RAID 6, Im planning on using TrueNas

[Specter FXXK]

6 minutes ago, Windows7ge said:

With those hardware controllers if I'm not mistaken you're going to need at minimum 5 drives of identical capacity and ideally rotational speed. Keep that in mind.

 

Why not Software RAID? I don't see a compelling reason for the expenditure on a hardware controller here.

 

Did you have an OS in mind? I think LZ4 on ZFS compresses pretty well. I'm not an audiophile so I don't know of the formats you listed which already incorporate compression and which don't. If they're already heavily compressed like with .MP3 adding another compression layer on-top of them isn't going to do much regardless of algorithm.

MP3 isn't a main problem it's, WAV, ACID-WAV, AIFF Etc that I use in Ableton 11 and FL Studio. I wanna maximize storage and have redundancy incase my drives decides it time to die.

 

other formats such as PSD, etc yeah that gets bigger lmao, that's why im looking at different compression algorithms 

[Windows7ge]

10 minutes ago, Specter FXXK said:

I have a 1 core CPU (AMD A4-7300 APU with Radeon HD Graphics) XD, I got more drives if they are required for RAID 6, Im planning on using TrueNas

If you're worried about the security of your data do not mix TrueNAS (ZFS) with a hardware controller. Use a HBA. ZFS can't protect your drives or report S.M.A.R.T. data about potential drive failures if they're behind a RAID controller.

 

8 minutes ago, Specter FXXK said:

MP3 isn't a main problem it's, WAV, ACID-WAV, AIFF Etc that I use in Ableton 11 and FL Studio. I wanna maximize storage and have redundancy incase my drives decides it time to die.

 

other formats such as PSD, etc yeah that gets bigger lmao, that's why im looking at different compression algorithms 

Well, FreeNAS/TrueNAS CORE supports LZ4 by default and you have the option of varying levels of ZSTD, GZIP, ZLE, & LZJB.

[Specter FXXK]

42 minutes ago, Windows7ge said:

If you're worried about the security of your data do not mix TrueNAS (ZFS) with a hardware controller. Use a HBA. ZFS can't protect your drives or report S.M.A.R.T. data about potential drive failures if they're behind a RAID controller.

 

Well, FreeNAS/TrueNAS CORE supports LZ4 by default and you have the option of varying levels of ZSTD, GZIP, ZLE, & LZJB.

alright, so software raid is recommended?

[Windows7ge]

With TrueNAS. Yes. Some RAID Controllers support JBOD mode. Some can be flashed to IT mode but an HBA that has worked well for me with ZFS is the LSI 9207-8i.

[Electronics Wizardy]

2 hours ago, Specter FXXK said:

I got a LSI MegaRAID on the way, heard it's pretty good for RAID 6

Id probably go software raid here, but if you use that card, your gonna want the battery, otherwise performance is gonna be pretty bad for writes in parity raid.

 

2 hours ago, Specter FXXK said:

im currently using LZMA/LZMA2 and ZSTD compression algori

ZSTD is about as good as you can get for high speed + realtime. 

 

But most of those files like MP3, AIFF, FLAC and others are already compressed, so ZSTD ain't helping. You can re encode the audio file, but your losing quality.

 

2 hours ago, Specter FXXK said:

got 2x1TB Seagate drives (home ones) and 3 random 500GB-1TB drives  different makers laying around and could use them as spares along the RAID 6 Array

Raid 6's don't like mixed drive sizes, your wasting the extra capacity. Id get identical drives, or use something made for mixed drive sizes.

 

2 hours ago, Specter FXXK said:

also looking for best compression algorithm for storing/compressing PSD, AI, INDD, CAD, C4D Etc.

Really with the amount of space you seem to have, just buy more drives, drives are cheap, and then there is no quality change.

[Specter FXXK]

14 minutes ago, Electronics Wizardy said:

Id probably go software raid here, but if you use that card, your gonna want the battery, otherwise performance is gonna be pretty bad for writes in parity raid.

 

ZSTD is about as good as you can get for high speed + realtime. 

 

But most of those files like MP3, AIFF, FLAC and others are already compressed, so ZSTD ain't helping. You can re encode the audio file, but your losing quality.

 

Raid 6's don't like mixed drive sizes, your wasting the extra capacity. Id get identical drives, or use something made for mixed drive sizes.

 

Really with the amount of space you seem to have, just buy more drives, drives are cheap, and then there is no quality change.

alright, ill take the advice, greatly appreciated, these drives are laying around thought I'll repurpose them XD, yeah the battery I got too