Help with NVMe RAID setup for Linux install

[bANONYMOUS]

I just switched fully from Windows to Linux, I was running a single 512GB NVMe as my boot when I switched, and everything was "fine", I was having some issues writing to my NTFS SSD's and HDD's, and one of my new HDD not coming up at all, after figuring some stuff out, I learned that when using NVMe, it disables the last two SATA ports on the motherboard, so I removed my SSD's entirely, replaced with NVMe's and replaced my boot drive with a new NVMe as well, so now there are three 1tb NVMe's, and moved the three 8TB HDD's down the SATA ports so now everything recognizes and we're all good.

I tried making a RAID5 in BIOS just like I normally would and it turns out that linux will not recognize NVMe if the SATA mode is set to RAID, it MUST be in AHCI, so software RAID is my only option, found out mdadm is the go to option everyone seems to recommend for it's stability which seems to be exactly what I want from what I found so far.

That's it, this is where I'm at, been looking at tons of guides and tutorials for mdadm and I just can't figure it out. I found a bunch of really poor youtube videos in different languages or really slow progression where it takes them multiple seconds in silence to type in the next command, so basically really bad sources for information and I can't find just one solid guide, start to finish, "this is how to configure NVMe's in RAID for a linux install" boom, done, but nothing like that seems to exist.

So far the most helpful information I was able to find was from Wendell at Level1Techs where he wrote a "guide" but skipped the entire first part as I'm assuming this was written for the more advanced users, and considering my last install just worked out of the box because it was to a single NVMe, it just worked, and I didn't have to make manual partitions or anything, it was all GUI, just a few clicks and it was ready to go, so naturally, I'm completely lost right now.

 

Mostly what I need help with right now is from Wendell's post.

Quote

First, partition the devices. In my case I’m using 4x nvme. The first partition should be about 200 Meg and marked efi.

You will have to format with mkfs.fat

How do I make the first partition? Is it all done in terminal, can I use "Disks" to make this partition, which NVMe is this done to, all of them? Just one? Just the first one? How do I mark the partition efi? Does he mean the partition name needs to be efi? Or is this a property option for formatting?

 

Quote

The second partition I made 1gb and formatted ext2. This will be /boot

Also how? exact same over thinking process occurred for me, I don't know what to do first or where to even start this process.

 

Once I get passed those two steps, I should be able to follow his guide further and try to figure this out, and if I get it working, probably write a more simplistic guide for the n00b into RAID.

I really thought there would have been way more information about this as this is pretty much step one into switching to Linux if you use a RAID array.

Like let's hypothetically say that if someone has been using a Hardware RAID with a Windows install as their boot drive for years, never has issues, super simple, wants to switch to Linux, decides to try it out, and then suddenly this process hits them in the face, I can see why it might steer people away from Linux when this is so brutally simple on Windows. I would honestly be switching back to Windows right now if I didn't already run benchmarks and see how much faster Ext4 was compared to NTFS, and how idle system resources are 0% on Linux, it's just so much better for my computer/hydro bill, and there's just so much more unlocked performance with Linux right out of the box, I just don't want to go back to Windows now that I know what I'm missing out on, so I'm willing to pull my hair out all night and try to learn how this works.

So, if anyone has some suggestions or commands for achieving the things I'm having issues with currently. That would be awesome.

[jdfthetech]

paritioning

( assuming your device is /nvme0 and probably partitioned with 1, 2, 3 etc so /nvme0n1 would be the first partition)

 

USING GDISK:

 

gdisk /dev/nvme0n1
o //clears
n // new partition
1st partition
512M //512 for efi
EF00 // efi

 

=-=-=-=-=-

 

 

=-==-- alternative cgdisk =-=-=-==-

 

first partition must be GPT for UEFI
cgdisk /dev/nvme0n1
use option ef02

 

=-=-==--

 

format:

mkfs.vfat -F32 /dev/<partition you want as fat>

 

=-=--=-=-

 

I'd recommend you make your boot partition a couple gigs.

You can run out of space there, so read up on how to clean it out after updating the kernel a few times

 

There is other partitioning software like fdisk out there as well, just depends on the distro you have.

 

[bANONYMOUS]

1 hour ago, jdfthetech said:

USING GDISK:

 

gdisk /dev/nvme0n1
o //clears
n // new partition
1st partition
512M //512 for efi
EF00 // efi

 

When you specified the size as 512M, how does this work?

When I type in lsblk, it shows my three non-partitioned NVME's as nvme0n1, nvme1n1, and nvme2n1.

So I type in sudo gdisk /dev/nvme0n1

There's no partitions so I just hit n

 

From there it asks the partition number, so I hit 1 to make the first partition

And then it asks for the first sector and I'm lost again, I'm trying to figure this out on the Arch wiki but my brain is literally mush right now, I've been trying to figure this out since 6pm here, it's almost 3am now and I still haven't figure out how to make the partitions.

For the first sector, it says default = 2048, but it doesn't say 2048 what, like, Bytes?, Kilobytes? Megabytes? Raccoons? I have no idea what to go off of there.
And then it says or +/- KMGTP which I'm assuming means Kilo/Mega/Giga/Tera/Petabytes, but what is the +/-?

From what I'm understanding from the Arch wiki, lets say to make this 512mb efi partition, when it asks for the first sector, I'm assuming I just hit enter since this is the first partition, so I'll just let it pick it's default starting sector point of 2048..raccoon.. or whatever element it's telling me that 2048 means, and then for the last sector, I would pick my partition size of 512M?

So if that's correct, and I do that, when I'm making the second partition, will it deduct the first partition amount, like is math involved? or does it figure it out on it's own?

 

Let's say when I make the next partition and it asks the same thing, do I have to input the first sector from where I left off and say to start at 512mb, or is it smart enough to figure that out on it's own that there is already 512mb allocated to partition 1, so if I just pick the default, it will just start the second partition from where the first ends at 512mb?
Then as for the last sector for the second partition, if I wanted a 2gb boot partition, would I enter 2gb? Or do I have to add the 512mb from the first partition to the last sector on the second partition? (making the last sector of the second partition 2.5gb instead to account for the 512mb on the first partition)

Like lets say I have a basket of that holds 10 apples, my starting sector is apple 0, and I want my efi to be 2 apples big, so my last sector is 2 apples, when I do the next partition, if I tell it I want the next partition to be 5 apples, it is going to make the next sector end at 5 apples capacity (leaving 5 apple sectors available for the total 10 apple sectors) or is it going to know that 2 apples are already pallocated, and it's going to add 5 more, allocating 7 apples total of the 10 available sectors?

There's a lot of information about this on the Arch wiki, but I'm literally diving into this straight vanilla off the n00b train, it's like trying to do something you don't know how to do in a different language, not only do I have no idea what to do, but I have no idea how to do it, and even if I knew what it was I need to do, I can't speak the language to ask in the correct terms to know what is it to even ask help for, It's defiantly making for an interesting night, that's for sure.

[jdfthetech]

Sector is not a partition, it's how much data is written in each subdivision of a track on the disk.

 

2048 is bytes.  This number comes from when dvd / cd roms were common and that was the standard, then ISOs started using it due to it's efficiency and so on.
To be clear:  the old standard was 512

 

Some folks specify different amounts for certain systems like servers and clusters so that is why the option is even there.

 

the commands I gave you for gdisk would work fine.

 

Those notes are what I keep when I do arch installs because I got sick of re-rereading the man pages.

 

so you can see how an lsblk should work, here is my current lsblk with an nvme partition.

Keep in mind, my device in this configuratoin is nvme1n1 simply because I used the 0n1 for a different purpose.

 

nvme1n1     259:0    0 465.8G  0 disk 
├─nvme1n1p1 259:3    0   500M  0 part /boot/efi
├─nvme1n1p2 259:4    0    48G  0 part [SWAP]
└─nvme1n1p3 259:5    0 417.3G  0 part /

Now, all that said, don't take this next comment as my being rude:

 

If this is getting a bit out of hand for you, perhaps it would be easier to install Endeavour OS.

Endeavour is arch with a calamari installer that takes care of the basics of system setup.

It will do most options normally, adds yay, and a couple small utility apps but nothing bad and nothing that can't easily be removed.

 

https://endeavouros.com/

 

If you are wanting to install the arch way and learn, then I applaud you, but if you just want an arch system up and running, this will do it much easier.

 

[bANONYMOUS]

The Arch wiki was just the best source for information, I used to be a distributor for Pop_OS! when I was a computer department manager awhile back so I just figured I would start there as it's basically as simple as it gets, it's pretty much Ubuntu without the corruption, and it has built in Nvidia drivers.

I did end up figuring it out, well, figured something out. I found this video on Youtube of someone showing almost exactly what I wanted to do with this setup. He does a RAID1 setup, so I just changed the commands when it says RAID1 and entered RAID5, and where he allocates two drives, I just repeated the step for three drives and everything is working perfect. It uses btrfs, which I wasn't prepared for, but it works, it's stupid fast, has snapshots to fall back to if I destroy something and I allocated a backup to each of the drives so if one fails, I literally just need to swap it out, it'll rebuild the failed partitions and I can keep on going, mission accomplished. I have no idea what btrfs is, or how all of this setup works, but I now have a fully functional NVMe RAID5 that boots Pop_OS! 20.04 perfectly and it is fast as all hell.  I would have preferred to use the traditional mdadm and Ext4 just as it seems to be the staple in how to do this, so learning the base on getting this working would have been idea for growth, but this setup with btrfs is working fantastic.

 

I'm not done with this yet though, I'm going to attempt to put together an old computer with parts I have laying around and keep messing around with madam until I learn how it works.

[bANONYMOUS]

I couldn't sleep knowing I didn't figure it out, so I wiped my drives again and continued going, I finally got it with mdadm in EXT4 and it's running great, I ran a benchmark and I got 3.2GBps on the RAID, where on BTRFS I got 2.8, so it's a pretty big improvement using EXT4 over BTRFS, so defiantly worth my time to continue trying to figure out mdadm.

Thanks for the help, I had to come back to your post like 42 times to reread what you suggested but it's up and running flawless now

[benitiv]

Was lurking on this thread. Interesting topic.

 

(bANONYMOUS - wendel took some of his precious time to reply to your post on the L1tech site. mb you want to inform him about your success as well? (sorry for beeing like i am ))

[bANONYMOUS]

1 hour ago, benitiv said:

wendel took some of his precious time to reply to your post on the L1tech site.

Yeah I saw that, that was how I finally figured it all out and got it working.
I just finished writing a full n00b guide on how to do this, I wnated to make sure I could repeat the process so I formatted everything again and did it over again a second time to make sure my process works for the guide and also made a video of the entire process that I need to edit now.
I'm going to make a second thread for the guide so it's not cluttered with the posts about trying to figure stuff out.