Howto: create your own linux home server using Debian

[IdeaStormer]

Yeah, I know. I only recently started using it myself. I'm going to make a new tutorial about all of that jazz somewhere in the next two months (summer vacation)

 

PS: new tutorial is gonna be on LVM + bridge-utils + xen, which I'm learning/setting up now.

 

Test it out or compare it to Software RAID, you get a little better performance with LVM, not earth shattering but better is better.

[Vitalius]

Test it out or compare it to Software RAID, you get a little better performance with LVM, not earth shattering but better is better.

Is Linux Volume Manager effectively Storage Spaces for Linux?

For those who don't know what Storage Spaces are. Like that, but you have a much finer control over it. It's like RAID, except managed differently by the OS. Such as where a single volume can exist across multiple drives (almost like a JBOD, but intelligent). 

[MG2R]

Test it out or compare it to Software RAID, you get a little better performance with LVM, not earth shattering but better is better.

Why would RAID + LVM be faster than RAID + partitions? Seriously, I see no reason how it would be faster, unless you're creating a volume group that stripes over multiple RAID arrays...

 

 

Is Linux Volume Manager effectively Storage Spaces for Linux?

Storage spaces is like LVM + RAID in one unified package, yes.

[ionbasa]

Yeah, I know. I only recently started using it myself. I'm going to make a new tutorial about all of that jazz somewhere in the next two months (summer vacation)

 

PS: new tutorial is gonna be on LVM + bridge-utils + xen, which I'm learning/setting up now.

Use KVM instead of XEN: http://www.phoronix.com/scan.php?page=article&item=intel_haswell_virtualization&num=1

Also, make sure to use Virt-Manager: http://virt-manager.org/ It has a nice set of CLI tools.

FInally when making a VM or "dom" you can use LVM to assign it its own Logical Volume from an LVM2 Volume Group. 

 

The reason as to why let the VM use LVM instead of an image is due to I/O thrashing, an image file requires alot or read/writes that degrades performance.

Just a tip, if you use LVM, don't write a filesystem to the LV for the VM. You are essentially partition off a VIrtual disk for the VM to use.

 

If you want later today I can post or PM you instructions on how to do so.

[MG2R]

--KVM > Xen--

Have been thinking about this a lot and still haven't found a conclusive answer. By the time I finally fully figured out how to get Xen running correctly, I kinda got the idea KVM might have been a better choice. Then again, Xen is more mature, which is something to look for as well... Not at all sure about which one I'll end up deploying on my server. Will play with KVM as well before making my decision, though

 

Thanks for the benchmark!

 

--use LVM and create an LV for your VM--

Yep, that's concurrent with what I've been reading on the interwebs

 

Just a tip, if you use LVM, don't write a filesystem to the LV for the VM. You are essentially partition off a VIrtual disk for the VM to use.

I'm afraid I'm not following you here, could you try to re-word those two sentences?

[ionbasa]

I'm afraid I'm not following you here, could you try to re-word those two sentences?

After creating the Logical volume in LVM, do not create a filesystem on it.

So don't do: 

$ mkfs.ext4 /dev/VG/LV

Also I have a nice little writeup for setup if you want to give it a go:

how to install: KVM bridgeutils and lib

Asumptions:

You have LVM already setup, you have at least 1 Volume Group and some free space!

Requirements:

First check if your CPU supports hardware virtualization:

egrep '(vmx|svm)' --color=always /proc/cpuinfo

If you get a list of features back, you are all good to go!

Install:

apt-get install virt-manager qemu-kvm libvirt-bin bridge-utils

Configuring the bridge:

vim /etc/network/interfaces

Make your config look similar:

# This file describes the network interfaces available on your system

# and how to activate them. For more information, see interfaces(5).

# The loopback network interface

auto lo

iface lo inet loopback

# The primary network interface

auto eth0

iface eth0 inet manual

auto br0

iface br0 inet static

        address 192.168.1.15

        network 192.168.1.0

        netmask 255.255.255.0

        broadcast 192.168.1.255

        gateway 192.168.1.1

        dns-nameservers 192.168.1.1

        bridge_ports eth0

        bridge_fd 9

        bridge_hello 2

        bridge_maxage 12

        bridge_stp off

Save the file and restart networking by:

/etc/init.d/networking restart

reboot the system

In order to be able to manage virtual machines as regular user you should put this user into the libvirt group:

# adduser <youruser> libvirt

Steps:

1) Open Webmin, and select Hardware--> Logical Volume Management

2) Make sure you have free space available on your VG (Volume Group)

3) Click the "Logical Volumes" tab and click "Create a logical volume in xxx-vg." (substitute you VG name here)

4) Give the volume a name, in this cas VM2, and a volume size, 100Gb. Click create

run:

virt-install -n vmname -r 2048 --os-type=linux --os-variant=debianwheezy --disk /dev/vg_name/lv_name,device=disk,bus=virtio --graphics vnc,listen=0.0.0.0 -w bridge=br0,model=virtio --noautoconsole --location http://mirrors.usc.edu/pub/linux/distributions/debian/dists/wheezy/main/installer-amd64/%C2'> [this is the mirror which hosts the installer files of the distribution of your choice, in this case Debian Wheezy

This came straight out of Notepad++ so there is no formatting. I'll let you do that 

[MG2R]

After creating the Logical volume in LVM, do not create a filesystem on it.

So don't do: 

$ mkfs.ext4 /dev/VG/LV

There will be a filesystem created in there at some point, as there need to be files stored in there at some point.

[ionbasa]

There will be a filesystem created in there at some point, as there need to be files stored in there at some point.

Err, the guest creates the file system.

 

From the guest's perspective the LV is just a hard drive that is blank. The guest then creates its own partitions in the LV such as swap,/, and boot.

 

Creating an filesystem on the LV for the KVM will not work. Everything has to be done relative to the guest.

[ionbasa]

I should clarify my previous post: http://libvirt.org/storage.html#StorageBackendDisk

 

When creating the Logical Volume, the Hypervisor mounts it as a DISK. I recomend leaving a filesystem off of the Logical Volume, because the guest can then partion the filesystems to suit.

[IdeaStormer]

Why would RAID + LVM be faster than RAID + partitions? Seriously, I see no reason how it would be faster, unless you're creating a volume group that stripes over multiple RAID arrays...

 

Why exactly I do not know or have looked up in too much detail but its is better, I didn't believe it either and proved it to myself by setting up Software RAID running some tests then wiping the setup and re-doing it with LVM ran the same tests and the numbers were better. Again, not an order of magnitude but still better than Software RAID.

[ionbasa]

So, The only thing that is realistically missing is an FTP server!

 

May I recomend ProFTPd: mainly because there is a webmin module for it and allows for virtualhosts  .

Setting up ftps is also relatively easy to do, it requires making a certificate and key using openssl (or you can use Debian's snake oil key )

It supports both implicet and explicit FTPS.

 

Why would we need to add an FTP server?

Many web content managment systems like: Wordpress, Bulletin boards, Joomla, and  Drupal usualy require an FTP server to apply updates to their packages .

 

Ok, so if you are going to do this, please don't neglect to add passive ports, and jail/chroot users.  :mellow:

 

Some resouces: 

http://www.proftpd.org/

https://www.digitalocean.com/community/tutorials/how-to-set-up-proftpd-on-ubuntu-12-04 This is an ubuntu tut, but ubuntu is basically debian...

[MG2R]

Err, the guest creates the file system.

Yep, that's what I meant In the end, there will be a file system on there... Good, we're talking about the same thing then.

 

Why exactly I do not know or have looked up in too much detail but its is better, I didn't believe it either and proved it to myself by setting up Software RAID running some tests then wiping the setup and re-doing it with LVM ran the same tests and the numbers were better. Again, not an order of magnitude but still better than Software RAID.

Are you talking about software RAID vs LVM, or software RAID + LVM vs software RAID + partitions?

 

So, The only thing that is realistically missing is an FTP server!

I know, and I won't be including it any time soon.

[Shaqalac]

Actually, Transmission has its own built-in web server, no Apache needed

That should make it a bit easier, especially with your new Transmission guide.

Now that I'm at it I'll still set up Apache

 

That warning basically means that Apache didn't find any ServerName directive defined and it can't find any reverse DNS records for your IP (which is logical, as your IP is local). Neither Ubuntu nor Debian define a ServerName in the default Apache2 package, because they don't know what your server's name is. You can ignore this warning if you want, but if you want it resolved, simply create a file under /etc/apache2/conf.d and type "ServerName yourservername" without quotation marks in that file.

I prefer to fix it now instead of running into a problem in the future. Due to your answer, that was no problem whatsoever.

 

Apparently, in newer versions of Apache2, they've renamed the site to 'default'. Thanks for the heads-up. Consider it fixed.

That did the trick, I'm now able to copy.

Did you forget to change anything to the next part in the guide?

 

 

Then, change /etc/apache2/sites-enabled/000-default to the following:

ServerAdmin webmaster@[member='localhost']# Turn on the rewrite engineRewriteEngine On# Check if the request is HTTPRewriteCond %{HTTPS} !=on# Rewrite the client requestRewriteRule ^/?(.*) https://%{SERVER_NAME}/$1 [R,L]

If I execute one of the two following:

sudo vim /etc/apache2/sites-available/defaultsudo vim /etc/apache2/sites-enabled/000-default

it will bring to to this.

Has this changed from when you originally made this guide or should I just add those parameters from above to the end up /etc/apache2/sites-enabled/000-default ?

 

I've been saying forever that I'd be doing a Transmission guide. Let me write that up for you real quick. It's quite easy after all...

The guide looks so simple. Thank you so much!

[MG2R]

 

Did you forget to change anything to the next part in the guide?

You need to overwrite the contents of that file with the blurb of text I've given there

[IdeaStormer]

Are you talking about software RAID vs LVM, or software RAID + LVM vs software RAID + partitions?

 

 

Software RAID vs LVM, straight up no combo's.

[Shaqalac]

You need to overwrite the contents of that file with the blurb of text I've given there

I've now edited /etc/apache2/sites-enabled/000-default to this and when I write http://lianlinas , http://192.168.1.49 , 192.168.1.49 in Chrome nothing happens and in IE is says the page can't be displayed. However if I type, in Chrome, https://192.168.1.49 and then after that type 192.168.1.49 it will redirect me to https://192.168.1.49

Any suggestions of what could be wrong?

Is the text I've added to /etc/apache2/sites-enabled/000-default wrong ?

Or should I add the same text to /etc/apache2/sites-available/default ?

[MG2R]

Software RAID vs LVM, straight up no combo's.

But they do very different stuff. AFAIK, LVM can't do RAID5-ish stuff...

 

 

<all the config stuff>

/etc/apache2/sites-enabled/000-default and /etc/apache2/sites-available/default should point to the same file...

Are you sure you restarted/reloaded apache?

 

Chrome redirecting to https has more to do with cache than anything else, AFAIK...

[ionbasa]

But they do very different stuff. AFAIK, LVM can't do RAID5-ish stuff...

Striping with parity is supported:

Eg: lvcreate --type raid5 -l 20%VG -i 2 -n lvm_raid5 vg0

 

By Now LVM can do:

RAID0,1,4,5,6, and10

EDIT: see http://wiki.gentoo.org/wiki/LVM#Different_storage_allocation_methods

Honestly the Gentoo wiki has one of the best writups on LVM that I can find on the Internet.

[MG2R]

Striping with parity is supported:

Ah. All the docs I had read so far only spoke of striping as in raid 0. This is interesting! Thanks for that wiki!

[Shaqalac]

/etc/apache2/sites-enabled/000-default and /etc/apache2/sites-available/default should point to the same file...

Are you sure you restarted/reloaded apache?

 

Chrome redirecting to https has more to do with cache than anything else, AFAIK...

I did a complete shut-down/boot of of the system, after I've restarted/reloaded apache.

If I type in http://192.168.1.49 or http://lianlinas it gives me nothing, not even a "this page does not exist", if I just type lianlinas or 192.168.1.49 however, it will redirect me to https.

Sweet!

 

By the way, I just noticed this. I think you forgot to include the message.

Installing a torrent client (Transmission)

After doing that, you should be able to visit http://192.168.1.150:9091 and be greeted with this message:

Edited July 2, 2014 by Shaqalac

[ionbasa]

So as to why LVM may get better performance than mdadm.

The two work in verry different ways:

With mdadm's raid 0 implementation you get simple striping that looks like: 

But with LVM stipes work differently represented as: 

Where PV0 and PV1 are two different disks. I believe the reason on why LVM is faster in RAID0 configurations compared to mdadm is becasue files are aranged to be contiguous on 1 disk at a time. This results in lower read and write latency.

 

Also, LVM has only been faster than mdadm recently due to better CPUs. LVM places file based on its hash/size, where as mdadm places paced based on simple striping rules. 

[IdeaStormer]

So as to why LVM may get better performance than mdadm.

The two work in verry different ways:

With mdadm's raid 0 implementation you get simple striping that looks like: 

But with LVM stipes work differently represented as: 

Where PV0 and PV1 are two different disks. I believe the reason on why LVM is faster in RAID0 configurations compared to mdadm is becasue files are aranged to be contiguous on 1 disk at a time. This results in lower read and write latency.

 

Also, LVM has only been faster than mdadm recently due to better CPUs. LVM places file based on its hash/size, where as mdadm places paced based on simple striping rules. 

 

At first @MG2R got my interest peaked on finding out the difference between LVM and SR, so did some digging and found this nice article on the subject.

 

http://www.olearycomputers.com/ll/linux_mirrors.html

 

Some interesting information for sure, read up!

[Shaqalac]

I've installed and configured Transmission but there's something missing however.

To be able to add a torrent directly from a website I've installed the Chrome extension .torrent to Transmission.

Can anybody tell me if my settings are about right?

 

Right now Transmission will place all my downloads at one location, in my case /Storage/Downloads. I'm not particularly happy with that as it means I will have to move the file after the download has finished. That's not efficient at all, I should one move that file once and that's when I add it to Transmission. On my Windows machine I use uTorrent. When I add a new torrent to uTorrent I'm able to tell uTorrent where to place the new file. Is this feature possible in Transmission, maybe through a plugin?

 

Are there any other plugins I should consider to install?

[Guest govinda Hinduja]

Thanks 

[MG2R]

By the way, I just noticed this. I think you forgot to include the message.

Damnit

 

 

<all the specific Transmission questions>

Sorry man, I've never been a Transmission power user. I hope someone else will be able to answer your questions.