Need help & suggestion on building "Portable" ARM-based NAS.

[TimedPing]

Recently my friend asked me to built his company a "portable" ARM-based NAS, he explained that his team need NAS rather than portable drive since they need to do work collab on their Adobe suite and other multimedia program with temporary project storage, he's need it to be portable because his team usually work on shared office space and coffee shop (He doesn't want to use finished off the shelf product). His basic requirement listed down below:

• Support up to 4 client access wired and support wireless mode (for quick preview on their phone and ipads), 2.5GBe networking for wired and atleast AC WiFi for wireless (need to have dedicated router for security purpose).
• Has UPS battery backup atleast for 1 hour with all services still running.
• The system need to be low powered (65W or lower) and support PD power input (He's planning to use 100W PD charger for this).
• Atleast 10TB of storage with good IO Speed (he told me atleast need to be 1.5-2GB/s  ish R/W raw disk pool speed).
• Optional disk encryption and ofcourse, since it's portable, need to somehow safe for frequent transport.
• He willingly spend up to $1.5k for the project (he understand that this is custom system solution).
• His team already have several 5/10GBe USB/Thunderbolt NIC and used it regularly on their laptops/macbooks for accesing their primary NAS at the office (which also I built), so using 2.5GBe solution is a must for them.

My current plan for the system is:

• Using RK3588 based device such as Radxa Rock 5B with 8GB of RAM (Thx to fully exposed full PCIe 3.0 x4 NVMe slot and secondary M.2 E key slot running at PCIe 2.0 x1, and also have 2.5 NIC built in).
• Running the OS on 32GB eMMC.
• Dual NVMe drive connected to ASMedia ASM2812 PCIe bifurcation card (plugged to M.2 NVMe via riser).
• Using USB HDD or Dedicated low power HBA (HBA connected to M.2 E key at PCIe 2.0 x1, I think that enough for handling 4 HDD bandwidth)
• Using 2x 64GB Intel Optane M10 as SSD cache. 
• Using 4x 4TB HDD on RAID-Z1 config.
• Dedicated Router with NanoPi R5C as base, 1st 2.5GBe NIC connected to NAS, 2nd 2.5GBe NIC connected to network switch, Mediatek RTL8812AU USB WiFi + Realtek Gbit USB NIC + Fibocom L860 4G modem for WAN connection
• Dedicated AC/AX WiFi Access Point (ill think the option for this later on).

Now, I have several question to ask:

1. I planned to use OMV (OpenMediaVault) on Debian since neither Unraid or TrueNAS have ARM compability, does OMV provide stable provide for basic NAS purpose? how good are their ZFS plugin? Or any other OS recommendation?
2. Does Linux BCache (Block Cache) play well on ARM devies, especially when used in conjunction with OMV? Any other suggestion to use SSD as L2 Cache if BCache is bad?
3. Does any 2.5 inch external USB HDD support SMART monitoring on Linux + OMV? Like WD My passport / Seagate expansion drive? or any specific SATA3 to USB 3.0 IC/Chip that support both UASP and SMART monitoring?
4. if USB not an option, any recommendation for low power SATA HBA that easy to compile/have native support on ARM kernel? I planned to use ASMedia ASM1064 based card.
5. Is built in realtek NIC on both NAS and router are OK for this type of system configuration?

Any suggestion is welcomed (Except told him to buy off the shelf soluion  ). Sorry for my messy english, i'm not native english speaker.

 

[AbydosOne]

19 minutes ago, TimedPing said:

The system need to be low powered (65W or lower) and support PD power input (He's planning to use 100W PD charger for this).

This is the biggest problem I see with this plan: assume each drive is 5-10W each and you've eaten up most of the power budget.

 

19 minutes ago, TimedPing said:

Any suggestion is welcomed

I know you said not to, but is buying a 4-bay NAS, 10G switch, and a standard UPS really that hard? USB-PD isn't going to happen, and by removing that requirement, it drastically simplifies this project.

 

EDIT: This is my rough idea...

 

Case ($260): https://www.amazon.com/Gator-Rotationally-Rolling-Recessed-G-PROR-4U-19/dp/B003ELQ2YK/

NAS ($620): https://www.amazon.com/QNAP-TS-431XeU-8G-US-Short-Depth-Rackmount-Version/dp/B0768PLS29/

UPS ($500): https://www.amazon.com/APC-Smart-UPS-SMT1500RM2U-1440VA-Rackmount/dp/B004F09CVO

Switch ($150): https://www.amazon.com/12-Port-Gigabit-UnManaged-Multi-Gig-XGS1010-12-ZZ0101F/dp/B084MLC83G/

[Kilrah]

56 minutes ago, TimedPing said:

• Atleast 10TB of storage with good IO Speed (he told me atleast need to be 1.5-2GB/s  ish R/W raw disk pool speed).

That pretty much means all flash storage, and isn't even gonna fit in the budget.

  

56 minutes ago, TimedPing said:

Does any 2.5 inch external USB HDD support SMART monitoring on Linux + OMV? Like WD My passport / Seagate expansion drive? or any specific SATA3 to USB 3.0 IC/Chip that support both UASP and SMART monitoring?

Both the 4TB WD and 5TB Seagate support SMART. Seagate's got a bit better performance (I have several of both).

They're SMR and slow/not great for an array, but that's all you'll get in 2.5" form factor.

 

 

[TimedPing]

1 hour ago, Kilrah said:

That pretty much means all flash storage, and isn't even gonna fit in the budget.

My bad, maybe you misunderstood what I'm mean (again sorry for my bad English), raw here means all the bulk drive is buffered both to RAM and L2/BCache SSD. I say "Raw" because the real world performance will be bottleneck by 2.5GBe NIC anyways.

1 hour ago, Kilrah said:

They're SMR and slow/not great for an array, but that's all you'll get in 2.5" form factor.

Ah I forgot about that, so maybe hooking up 3.5 WD RED to SATA to USB is solution? Any recommendations for good chip that support both SMART and UASP? I can make custom PCB backplane integrating the chip into it, most manufacturers website like ASMedia and VIA didn't mention any of this (well they mention UASP but not SMART), and I don't want the hassle to buy bunch of SATA to USB converter to test that

[TimedPing]

2 hours ago, AbydosOne said:

I know you said not to, but is buying a 4-bay NAS, 10G switch, and a standard UPS really that hard? USB-PD isn't going to happen, and by removing that requirement, it drastically simplifies this project.

Can't imagine people at Starbucks seeing my friend plugging all of that and look's like hacker guy on Hollywood movies. The point is to be portable, that's why I choose ARM and minimize the power requirements, the "UPS" AKA battery power here to handle sudden spike/overcurrent from less than 100W PD power. His company already have faster NAS setup at the main office (like I mention on the first post)

Anyway, thank you for the input, maybe I can put external 12v 20A PSU, and PD just for backup incase my friend is too lazy to bring the power brick. Good point. Or I can wait the Chinese made 240W PD trigger board instead :⁠,⁠-⁠)

[Blue4130]

3 hours ago, TimedPing said:

Can't imagine people at Starbucks seeing my friend plugging all of that and look's like hacker guy on Hollywood movies. The point is to be portable, that's why I choose ARM and minimize the power requirements, the "UPS" AKA battery power here to handle sudden spike/overcurrent from less than 100W PD power. His company already have faster NAS setup at the main office (like I mention on the first post)

Anyway, thank you for the input, maybe I can put external 12v 20A PSU, and PD just for backup incase my friend is too lazy to bring the power brick. Good point. Or I can wait the Chinese made 240W PD trigger board instead :⁠,⁠-⁠)

I'd honestly get something like this...

https://www.amazon.com/qnap-tbs-453dx-8g-us-quad-core-nasbook-10gbe/dp/b07ndd5cfx

 

It's going to be way less janky than whatever you Cobble together and look less "Hollywood hacker".

[10leej]

You can get laptops that can fit multiple SSD's, then you can get switches that are powered via USB. For a boot drive all you really need is a bog standard usb drive and you can get ones that can fit 30+GB's in size and are physically super small. 

 

Use the USB key fore the boot drive and setup the SSD's to run in at least a Raid 1 configuration.

 

Honestly though, why not use a VPN and a NAS that has a high speed internet connection. Services like Tailscale basically make the NAS appear like it's on a local network.

 

or better yet..

Get an office.

[TimedPing]

45 minutes ago, 10leej said:

You can get laptops that can fit multiple SSD's, then you can get switches that are powered via USB. For a boot drive all you really need is a bog standard usb drive and you can get ones that can fit 30+GB's in size and are physically super small. 

 

Use the USB key fore the boot drive and setup the SSD's to run in at least a Raid 1 configuration.

I first think of this kind of solution too, but I need to convince him telling that X86 platform is not that bad (failed my first attempt on that).

48 minutes ago, 10leej said:

Honestly though, why not use a VPN and a NAS that has a high speed internet connection. Services like Tailscale basically make the NAS appear like it's on a local network.

 

or better yet..

Get an office.

You didn't really read my first post, He already have an office and proper NAS with 340TB of usable storage, he need another NAS for remote work, and not every country and region here have 100TB/s WiFi 10 Standards connection on starbucks/shared office space like the US. even if we had that kind of connection, using live project file on remote NAS will be horrible because of the latency, and we already test that before, and he hated it, that's why he willing to spend 1.5k just to get a Portable NAS.

[TimedPing]

5 hours ago, Blue4130 said:

I'd honestly get something like this...

https://www.amazon.com/qnap-tbs-453dx-8g-us-quad-core-nasbook-10gbe/dp/b07ndd5cfx

 

It's going to be way less janky than whatever you Cobble together and look less "Hollywood hacker".

This one is awesome, atleast on paper, but it's cost around $1K over here so yeah....

[Blue4130]

34 minutes ago, TimedPing said:

This one is awesome, atleast on paper, but it's cost around $1K over here so yeah....

Yea, but it will work flawlessly and not be a cobbled together mess. Convinience and polished products cost.

[TimedPing]

9 hours ago, Blue4130 said:

Yea, but it will work flawlessly and not be a cobbled together mess. Convinience and polished products cost.

I'll already told him to buy this instead, still waiting for his reply, anyway that $1k is not including the drive tho (+ 1 month shipping)

[bloodthirster]

I have a Honeycomb LX2 and have had issues with:

- warm boots don't work (later board revisions fix this but Solid Run won't update old boards like mine)

- SATA has signaling issues so you're maxed out at 3Gbps, at 6Gbps there's so many errors the drive drops out

- building firmware is flaky at best.  I've built firmware before but could never get anything I built to boot, even following their instruction in docker and native

- included heatsink needs upgrading if you're doing anything heavy

 

It's a shame since if they fixed these issues I could actually recommend it here.   I am still using mine as a NAS with a SuperMicro SATA/SAs controller running in JBOD with ZFS, but I wouldn't want anyone to go through the pain I went through.   It was originally going to be an ARM test and dev system but because the warm boot issue it's useless for that.

[TimedPing]

9 hours ago, bloodthirster said:

I have a Honeycomb LX2 and have had issues with:

- warm boots don't work (later board revisions fix this but Solid Run won't update old boards like mine)

- SATA has signaling issues so you're maxed out at 3Gbps, at 6Gbps there's so many errors the drive drops out

- building firmware is flaky at best.  I've built firmware before but could never get anything I built to boot, even following their instruction in docker and native

- included heatsink needs upgrading if you're doing anything heavy

 

It's a shame since if they fixed these issues I could actually recommend it here.   I am still using mine as a NAS with a SuperMicro SATA/SAs controller running in JBOD with ZFS, but I wouldn't want anyone to go through the pain I went through.   It was originally going to be an ARM test and dev system but because the warm boot issue it's useless for that.

Thanks for the suggestion! Sadly after seeing the price, I rather take the @Blue4130recommendation instead, doesnt need the 16 core CPU and definetly the power consumption rating of that thing, anyhow, thank you, didn't really know theres "middle in between" ARM server boards like Honeycomb before tho, even through i've quite an embedded system dev myself, great info!

[TimedPing]

Alright folks, after few back and forth with my friend, reading and considering every suggestion, He agree to ditch ARM altogether and willing to compromise several other factor, so I came out with this draft, most of the hardware I already have lying around my office (more profit for me LOL):

1. Case & PSU: with ditching ARM alltogether, I need to manage to still have lower than usual power consumption with have small unit footprint, I need to find a case that fits, furtunately I find this case recommended by my fellow PC building friend: https://enlight-indonesia.com/product/enlight-en-120/ which is low profile case but support Micro-ATX motherboard, with just enough expansion option I need, the case also include 300W 80+ bronze SFX PSU (which my friend claim is actually effecient and durable, sadly won't needing those). 
2. Platform: The rest of the hardware specs is listed below:
   • CPU: Intel Core i5-6500T 4C no HT (I have this lying around) + Intel stock cooler with copper core.
   • Motherboard: Biostar HiFi B150S5 (I also have several of these board), it's offers solid expansion with built in M.2 NVMe, been using this board for several NAS deployment over 3 years without any failure. It's also support VT-D .
   • RAM: Just using 2x 8GB DDR4 2400MT/s Samsung module lying around in the office.
   • SSD for L2 Cache: 2x Intel Optane M10 64GB, one on M.2 slot, one on PCIe x16 slot with adapter. Will configured in RAID 0.
   • HDDs: 3x WD Purple 8TB CMR drive (WD84PURZ) which sold rebranded as Alhua CCTV HDD without custom firmware installed (brand new, got bulk discount for this).  Will configured as RAID-Z1.
   • OS Storage: 2x Sandisk 24GB mSATA SSD with SATA adapter, mirrored. Running TrueNAS Core (?).
   • Networking: 2.5GBe Intel i225v PCIe x1 adapter (with LP bracket).
3. How to power those (I guess): Will built custom UPS system with 25Wh LiPo Battery, with 12V 240W DC input from external adapter. Connecting those with 200W PicoPSU to power the system.
4. About the Router situation: still need help for this one, should I running VM inside the OS to handle routing or I just put dedicated networking hardware like my first plan?

   On 2/10/2023 at 12:26 AM, TimedPing said:

   Dedicated Router with NanoPi R5C as base, 1st 2.5GBe NIC connected to NAS, 2nd 2.5GBe NIC connected to network switch, Mediatek RTL8812AU USB WiFi + Realtek Gbit USB NIC + Fibocom L860 4G modem for WAN connection

Thank you for all the input, I still need many more suggestion for this project, any more input welcomed!

[AbydosOne]

6 minutes ago, TimedPing said:

SSD for L2 Cache: 2x Intel Optane M10 64GB, one on M.2 slot, one on PCIe x16 slot with adapter. Will configured in RAID 0.

6 minutes ago, TimedPing said:

Running TrueNAS Core (?).

I wouldn't bother with the L2ARC (in ZFS parlance) in a situation where the NAS won't be online for long enough periods to fill the ARC or for workflows that don't access the same data constantly (think database server). Keep in mind that the L2ARC is invalidated on restart (it isn't persistent), so you'd be better off putting more RAM in instead (ZFS loves RAM).

 

10 minutes ago, TimedPing said:

About the Router situation: still need help for this one, should I running VM inside the OS to handle routing or I just put dedicated networking hardware like my first plan?

Honestly, one of those mini travel routers would be good in this case, using a multi-gig switch (depending on how many users, either a TRENDnet 2.5-gig switch, or a Ubiquiti Flex XG if you want to bump up to 10-gig with fewer ports) to do the heavy lifting. The router only needs to assign the IPs, then the switch can take over (i.e. even if the router is 1-gig, you can still pass traffic from device to device through the switch at multi-gig).

(though after looking at the NanoPi, it isn't a bad option, just the travel router may have a nicer interface/firmware)

 

13 minutes ago, TimedPing said:

Networking: 2.5GBe Intel i225v PCIe x1 adapter (with LP bracket).

Since you won't need the L2ARC and have a free x16 slot, I would grab an Intel X540-T2 or a Mellanox ConnectX-3 (with SFP+ media converter) for networking instead to get 10G. Not sure if TrueNAS is on the bleeding edge when it comes to Intel NIC drivers, so I can't guarantee it has support for the I225v yet (probably, but not sure).

[TimedPing]

26 minutes ago, AbydosOne said:

I wouldn't bother with the L2ARC (in ZFS parlance) in a situation where the NAS won't be online for long enough periods to fill the ARC or for workflows that don't access the same data constantly (think database server). Keep in mind that the L2ARC is invalidated on restart (it isn't persistent), so you'd be better off putting more RAM in instead (ZFS loves RAM).

 

Honestly, one of those mini travel routers would be good in this case, using a multi-gig switch (depending on how many users, either a TRENDnet 2.5-gig switch, or a Ubiquiti Flex XG if you want to bump up to 10-gig with fewer ports) to do the heavy lifting. The router only needs to assign the IPs, then the switch can take over (i.e. even if the router is 1-gig, you can still pass traffic from device to device through the switch at multi-gig).

(though after looking at the NanoPi, it isn't a bad option, just the travel router may have a nicer interface/firmware)

 

Since you won't need the L2ARC and have a free x16 slot, I would grab an Intel X540-T2 or a Mellanox ConnectX-3 (with SFP+ media converter) for networking instead to get 10G. Not sure if TrueNAS is on the bleeding edge when it comes to Intel NIC drivers, so I can't guarantee it has support for the I225v yet (probably, but not sure).

Awesome info, will think about this and discuss it with my friend, I'll let you know the update soon.

[Blue4130]

1 hour ago, TimedPing said:

Alright folks, after few back and forth with my friend, reading and considering every suggestion, He agree to ditch ARM altogether and willing to compromise several other factor, so I came out with this draft, most of the hardware I already have lying around my office (more profit for me LOL):

1. Case & PSU: with ditching ARM alltogether, I need to manage to still have lower than usual power consumption with have small unit footprint, I need to find a case that fits, furtunately I find this case recommended by my fellow PC building friend: https://enlight-indonesia.com/product/enlight-en-120/ which is low profile case but support Micro-ATX motherboard, with just enough expansion option I need, the case also include 300W 80+ bronze SFX PSU (which my friend claim is actually effecient and durable, sadly won't needing those). 
2. Platform: The rest of the hardware specs is listed below:
   • CPU: Intel Core i5-6500T 4C no HT (I have this lying around) + Intel stock cooler with copper core.
   • Motherboard: Biostar HiFi B150S5 (I also have several of these board), it's offers solid expansion with built in M.2 NVMe, been using this board for several NAS deployment over 3 years without any failure. It's also support VT-D .
   • RAM: Just using 2x 8GB DDR4 2400MT/s Samsung module lying around in the office.
   • SSD for L2 Cache: 2x Intel Optane M10 64GB, one on M.2 slot, one on PCIe x16 slot with adapter. Will configured in RAID 0.
   • HDDs: 3x WD Purple 8TB CMR drive (WD84PURZ) which sold rebranded as Alhua CCTV HDD without custom firmware installed (brand new, got bulk discount for this).  Will configured as RAID-Z1.
   • OS Storage: 2x Sandisk 24GB mSATA SSD with SATA adapter, mirrored. Running TrueNAS Core (?).
   • Networking: 2.5GBe Intel i225v PCIe x1 adapter (with LP bracket).
3. How to power those (I guess): Will built custom UPS system with 25Wh LiPo Battery, with 12V 240W DC input from external adapter. Connecting those with 200W PicoPSU to power the system.
4. About the Router situation: still need help for this one, should I running VM inside the OS to handle routing or I just put dedicated networking hardware like my first plan?

Thank you for all the input, I still need many more suggestion for this project, any more input welcomed!

HAha This is coffee shop friendly? Your coffee shops must have BIG tables...

[TimedPing]

5 minutes ago, Blue4130 said:

HAha This is coffee shop friendly? Your coffee shops must have BIG tables...

it never meant to be place on top of table on the first place, I gonna put it on a suitcase style frame with locking wheel, around 22 inch diagonally, adding couple ventilation holes equipped with several 80/120mm fan and IO ports for cabling to client and power socket, the PC case itself is just gonna be internal chasis inside the outside the outer case. If I gonna go with ARM option, it's will be smaller, around 15 inch across. The main purpose of this thing is to minimize hardware setup in every deployment environment, just plug in the power plug, plug all the ethernet jack, and turn it on, after finish just undo the ethernet and power cable and it's ready to transport. and yes, you can put this thing BELOW THE COFFE SHOP TABLE no problem (I'm trully sorry not mentioning how the system gonna placed before). Even putting UPS sized NAS like 4 bay Synology one feels weird inside the coffee shop though . Just imagine it like Portable power station put below your camping table but it's look like a travel suitcase instead (vertical orientation).

[Needfuldoer]

A 'workstation' desktop-replacement laptop ticks all your boxes except for network connectivity.

[Blue4130]

7 hours ago, TimedPing said:

it never meant to be place on top of table on the first place, I gonna put it on a suitcase style frame with locking wheel, around 22 inch diagonally, adding couple ventilation holes equipped with several 80/120mm fan and IO ports for cabling to client and power socket, the PC case itself is just gonna be internal chasis inside the outside the outer case. If I gonna go with ARM option, it's will be smaller, around 15 inch across. The main purpose of this thing is to minimize hardware setup in every deployment environment, just plug in the power plug, plug all the ethernet jack, and turn it on, after finish just undo the ethernet and power cable and it's ready to transport. and yes, you can put this thing BELOW THE COFFE SHOP TABLE no problem (I'm trully sorry not mentioning how the system gonna placed before). Even putting UPS sized NAS like 4 bay Synology one feels weird inside the coffee shop though . Just imagine it like Portable power station put below your camping table but it's look like a travel suitcase instead (vertical orientation).

You have some odd ideas... A 4 Bay nas and switch looked too "hacker like" but random wires plugged into a suitcase doesn't?

[Echothedolpin]

This is going to be a massive theft target.

 

Also, you might want to take into consideration that you’re putting components into a rolling frame without suspension that were never engineered with those types of stresses in mind.

[ToboRobot]

11 hours ago, TimedPing said:

it never meant to be place on top of table on the first place, I gonna put it on a suitcase style frame with locking wheel, around 22 inch diagonally, adding couple ventilation holes equipped with several 80/120mm fan and IO ports for cabling to client and power socket, the PC case itself is just gonna be internal chasis inside the outside the outer case. If I gonna go with ARM option, it's will be smaller, around 15 inch across. The main purpose of this thing is to minimize hardware setup in every deployment environment, just plug in the power plug, plug all the ethernet jack, and turn it on, after finish just undo the ethernet and power cable and it's ready to transport. and yes, you can put this thing BELOW THE COFFE SHOP TABLE no problem (I'm trully sorry not mentioning how the system gonna placed before). Even putting UPS sized NAS like 4 bay Synology one feels weird inside the coffee shop though . Just imagine it like Portable power station put below your camping table but it's look like a travel suitcase instead (vertical orientation).

If you are planning on fitting it all in a case like a Pelican 1510, I'd start there as your max dimensions for the project and go from there.

If you plan on pulling it around on the wheels and care about data or dead drives, I would opt for solid state rather than spinning drives which hate impacts.

So then try to pick the storage drives you want to use.

I think you are going to need to re-evaluate the budget, and should avoid custom solutions as much as possible.

[TimedPing]

15 hours ago, Blue4130 said:

You have some odd ideas... A 4 Bay nas and switch looked too "hacker like" but random wires plugged into a suitcase doesn't?

if i gonna use 4 bay NAS, my friend will need to connect that each time he make the setup to switch, then router, then plug all to a power terminal, which one more practical? just few ethernet cable + single power cable or off-the shelf solution? 

[TimedPing]

11 hours ago, ToboRobot said:

If you are planning on fitting it all in a case like a Pelican 1510, I'd start there as your max dimensions for the project and go from there.

If you plan on pulling it around on the wheels and care about data or dead drives, I would opt for solid state rather than spinning drives which hate impacts.

So then try to pick the storage drives you want to use.

I think you are going to need to re-evaluate the budget, and should avoid custom solutions as much as possible.

I agree with this, sadly it's not my personal project, atleast I have someone else budget to experiment with, and he agree to that terms . about mechanical drive situation, I totally agree, I already told him to go all flash with lower usable capacity, but he insist, so I still doing research around shock dampening the HDD, it's not gonna be easy, but it's gonna be fun (and stressfull) one.

[TimedPing]

11 hours ago, Echothedolpin said:

This is going to be a massive theft target.

 

Also, you might want to take into consideration that you’re putting components into a rolling frame without suspension that were never engineered with those types of stresses in mind.

Ikr, but thats none of my business, just trying to make my friend happy, yeah, about the frame, still having an headache for a week and broke few 500GB enterprise drive to try the drive dampening solution, wish me luck!