Howto: create your own linux home server using Debian

[Jaybird]

MG2R might want to check on owncloud tutorial. I returned with 404 errors on links. owncloud has been updated to 6.0.3 at the time of writing this. 

[MG2R]

MG2R might want to check on owncloud tutorial. I returned with 404 errors on links. owncloud has been updated to 6.0.3 at the time of writing this. 

Yeah, that's what you get with thingies like this, they get out-of-date rather quickly. If I find the time, I'll update the post. Thanks for the heads-up!

[lilingdatou]

An excellent guide for normal users, I will setup my home server following this, many thanks, MG2R.

[Shaqalac]

I'm about to create my RAID volume and I have made one big extended partition on each of my 4TB disks.

Should I do more before I can create my RAID with mdadm?

[MG2R]

I'm about to create my RAID volume and I have made one big extended partition on each of my 4TB disks.

Should I do more before I can create my RAID with mdadm?

If you want your drives to just be used completely, you don't even need the partitions. Just create a RAID device with the entire block devices (so, for example, /dev/sdb instead of /dev/sdb1).

[Shaqalac]

If you want your drives to just be used completely, you don't even need the partitions. Just create a RAID device with the entire block devices (so, for example, /dev/sdb instead of /dev/sdb1).

I've just deleted all the partition on the four drives and have followed this wiki and have executed this command:

mdadm --create --verbose /dev/md0 --level=6 --raid-devices=4 /dev/sdb /dev/sdc /dev/sdd /dev/sde 

I haven't continues as I want to be completely sure before I start building the array.

 

 

Everything is as default, chunk size at 512K ect., it tells me that all data on the drives will be lost.

In the end it says size is set to 3906887168K.

Does everything looks good?

 

Sorry for these questions that might be obvious but with all the problems I've had I can not bear another one.

[MG2R]

Everything is as default, chunk size at 512K ect., it tells me that all data on the drives will be lost.

In the end it says size is set to 3906887168K.

Does everything looks good?

Seems to be about right, if you're using 2 TB drives (4x2 TB - 2x2TB for parity = 2 x 2TB for data = 4 TB)

 

 

Sorry for these questions that might be obvious but with all the problems I've had I can not bear another one.

As always: happy to help

 

 

 

protip: remember that you'll need to use the virtual device /dev/md0 after doing this (just to be sure that you're not overwriting one of your drives when creating partitions, for example).

 

Also, I noticed that the wikipedia page doesn't contain the commands anymore. I'll try to fix the tutorial this evening

[Shaqalac]

Seems to be about right, if you're using 2 TB drives (4x2 TB - 2x2TB for parity = 2 x 2TB for data = 4 TB)

Hmm...

I have 4x4TB drives, which in a RAID6 should give me 8TB of usable storage.

protip: remember that you'll need to use the virtual device /dev/md0 after doing this (just to be sure that you're not overwriting one of your drives when creating partitions, for example).

md0 is the name of the RAID array I'm about to create, if I'm correct.

 

Also, I noticed that the wikipedia page doesn't contain the commands anymore. I'll try to fix the tutorial this evening

Well, that explains some of my confusion.

I guess that's a disadvantage of using an external source, they could change something or shut the domain down.

This is a noob speaking, the wiki I used looks pretty good, I was able to understand it so it can't be that bad.

Whether something in that wiki is wrong or completely irrelevant I can't speak of.

[MG2R]

Hmm...

I have 4x4TB drives, which in a RAID6 should give me 8TB of usable storage.

Could you give me the exact, word-for-word, line in which it gives you the size? mdadm has quite the interesting naming scheme... For instance "used dev size" means the actual size used per disk (so say you create a raid 1 with a 1TB and a 2TB drive, the used dev size would be 1 TB).

[Shaqalac]

Could you give me the exact, word-for-word, line in which it gives you the size? mdadm has quite the interesting naming scheme... For instance "used dev size" means the actual size used per disk (so say you create a raid 1 with a 1TB and a 2TB drive, the used dev size would be 1 TB).

Here's a screenshot.

Maybe I should have cropped it a bit before uploading it, just ignore the top part, just me messing about.

This is the point where I'm at right now.

[MG2R]

Here's a screenshot.

Maybe I should have cropped it a bit before uploading it, just ignore the top part, just me messing about.

This is the point where I'm at right now.

 

I'd just go for it, should be fine. After it's created, check with `sudo mdadm --detail /dev/md0` if the array indeed has the correct size

 If it isn't correct, you can always just destroy the array again

[Shaqalac]

I'd just go for it, should be fine. After it's created, check with `sudo mdadm --detail /dev/md0` if the array indeed has the correct size

 If it isn't correct, you can always just destroy the array again

Great success!

 

So as you said the size displayed before creating the array is probably the size of the smallest disk in the array.

I now have 8TB at my disposal, well I need to setup share first though.

[MG2R]

well I need to setup share first though.

Actually, you first need to create a filesystem on that array

[IdeaStormer]

Actually, you first need to create a filesystem on that array

 

Going to be a long wait for the prompt after hitting return

[MG2R]

Going to be a long wait for the prompt after hitting return

just append & to the command

[IdeaStormer]

just append & to the command

 

Yea, but then he's going to try to use it before its done, best not to use & on that one. This is where the old beep command came in handy, I need to revisit that command again or make one for those long important commands you want to know finished or are waiting on.

[MG2R]

Yea, but then he's going to try to use it before its done, best not to use & on that one. This is where the old beep command came in handy, I need to revisit that command again or make one for those long important commands you want to know finished or are waiting on.

You can just run screen to get detachable virtual terminals

[alpenwasser]

Yea, but then he's going to try to use it before its done, best not to use & on that one. This is where the old beep command came in handy, I need to revisit that command again or make one for those long important commands you want to know finished or are waiting on.

  

You can just run screen to get detachable virtual terminals

*cough* tmux *cough*

[MG2R]

  

*cough* tmux *cough*

^ that functionality is built into screen as well

[alpenwasser]

^ that functionality is built into screen as well

I know. My point was not to say screen isn't a viable solution,

just pointing out a possible alternative.

[Shaqalac]

Actually, you first need to create a filesystem on that array

My plan is to grow the array as I go along and need more space. Without any modding to my Lian-Li Q25 it can accommodate 7 3.5" drives, which can give me 20TB of storage.

I've done some research and it seems like XFS is the one to go for especially with a big RAID array.

Besides from Google searches I've looked through the 10TB show-off topic and most of them with a Linux based system is using XFS as their file system.

I stumbled upon a comment from this reply, that XFS stripe sizes any larger than 256K. Correct me if I'm wrong but stripe=chunk? In that case my RAID array is set at 512K.

So if I want to use XFS I should simply delete my RAID array, however that's done, and then recreate it with chunk size = 256K?

[MG2R]

I stumbled upon a comment from this reply, that XFS stripe sizes any larger than 256K. Correct me if I'm wrong but stripe=chunk? In that case my RAID array is set at 512K.

So if I want to use XFS I should simply delete my RAID array, however that's done, and then recreate it with chunk size = 256K?

Yeah, performance will be best if the stripe size of XFS matches the chunk size of your array.

[Demonicreature]

Who else read it as "Howto: create your own linus home server using Debian"

[Shaqalac]

Yeah, performance will be best if the stripe size of XFS matches the chunk size of your array.

I've now deleted and recreated my RAID array which was fairly easy.

My RAID now looks like this. Chunk size is 256K and should be ready for XFS.

 

I'm following this guide and as you can see it says that after you've created your RAID array you should the configuration to the mdadm conf with this command:

sudo mdadm --detail --scan >> /etc/mdadm/mdadm.conf

If I execute that it give me this

-bash: /etc/mdadm/mdadm.conf: Permission denied

After a quick google search I found this guide/tutorial from '11 on how to setup a RAID with mdadm.

It as well shows that you should edit the mdadm configuration file. (Take a look in the link, he uses a few more commands than the first guide.

He does however shows how it should look like after you've edited the configuration file:

/dev/md0:        Version : 1.2  Creation Time : Tue Nov 15 13:01:40 2011     Raid Level : raid5     Array Size : 2092032 (2043.34 MiB 2142.24 MB)  Used Dev Size : 1046016 (1021.67 MiB 1071.12 MB)   Raid Devices : 3  Total Devices : 3    Persistence : Superblock is persistent    Update Time : Tue Nov 15 13:57:25 2011          State : clean Active Devices : 3Working Devices : 3 Failed Devices : 0  Spare Devices : 0         Layout : left-symmetric     Chunk Size : 512K           Name : test:0           UUID : 2e46af23:bca95854:eb8f8d7c:3fb727ef         Events : 34    Number   Major   Minor   RaidDevice State       0       8       17        0      active sync   /dev/sdb1       1       8       33        1      active sync   /dev/sdc1       3       8       49        2      active sync   /dev/sdd1

As you can see, it looks similar to mine. So am I good to go.

[MG2R]

As you can see, it looks similar to mine. So am I good to go.

Basically, no, you're not. The reason you need to append something to /etc/mdadm/mdadm.conf is to auto-detect the array when your system boots.

 

If you execute

sudo mdadm --detail --scan

You'll get something similar to

simon@brol3:~$ sudo mdadm --detail --scan[sudo] password for simon:ARRAY /dev/md/0 metadata=1.2 name=brol3:0 UUID=36e7b80e:c0a45afe:1fa0c847:908f2d09

The line starting with ARRAY should be appended to the mdadm.conf file in order for mdadm to discover and start the array at boot time. It's weird that the system doesn't allow you to append the line to mdadm.conf by using sudo. You could try logging in as  root by executing `sudo su` and then try to execute

mdadm --detail --scan >> /etc/mdadm/mdadm.conf

again. If it still gives you 'permission denied', there's something really funky going on with your system.