Howto: create your own linux home server using Debian

[nik96]

-SNIP-

 

You might be interested in this since ur using Transmission.

 

https://code.google.com/p/transmisson-remote-gui/

 

It's a program which is remote GUI for transmission, ive used it before and it was solid.

[Vitalius]

-snip-

They look right. If you haven't changed/configured your RPC settings, then they should be identical as they are there. 

That's what people generally use other plugins for. This is for FreeNAS, but the same concept can be applied anywhere, so replace "FreeNAS" with "Linux":

 

So, the idea is that Sickbeard & CouchPotato download the torrents as they become available auto-magically, then Sickbeard auto-moves the downloaded files to the Plex folder where Plex sorts them and adds metadata like summaries, posters, casts, etc. 

I'm pretty sure Sickbeard can auto-move non-Sickbeard downloaded files, so you could put downloads you manually did in the "Downloads" folder in the above example and Sickbeard would auto-sort them as such. You'd have to read up on it though to be sure.

It basically makes it all automated. I had issues getting it to work due to file permissions, where all those plugins could access the same folders and such from inside different Jails, but that's a FreeNAS specific issue. I imagine it'd be a ton easier to get done in Linux.

The guide for doing it on FreeNAS is available here, so you could possibly use it as a reference if you want to make it all automated and just translate FreeNAS stuff to Linux stuff (the commands are similar, if not the same as BSD & Linux look slightly similar to me however I'm a noob at both).

[Jade]

@MG2R - I promised you I'd write these a while back. Delivery obligatory. 

 

How to install the Java (Runtime Environment)

 

Login as a user with root privileges. Execute 'su' to get said privileges.

 

Here, we're going to install 2 repos, so apt-get can find and install the right Java.

echo "deb http://ppa.launchpad.net/webupd8team/java/ubuntu precise main" | tee /etc/apt/sources.list.d/webupd8team-java.listecho "deb-src http://ppa.launchpad.net/webupd8team/java/ubuntu precise main" | tee -a /etc/apt/sources.list.d/webupd8team-java.list

 

Now, we're going to add the keys to trusted so packages can be authenticated as valid.

apt-key adv --keyserver keyserver.ubuntu.com --recv-keys EEA14886

 

Update the package lists...

apt-get update

 

Finally time to install Java. You'll have to accept their license. If you wish to do this in an automated method, you can use the contents of the spoiler.

"echo oracle-java7-installer shared/accepted-oracle-license-v1-1 select true | sudo /usr/bin/debconf-set-selections"

apt-get install oracle-java7-installer

 

Just to double check that java installed properly, execute the following command.

java -version

 

This should return something along the lines of

java version "1.7.0_65"

 

This means java is functioning and usable, and you can be on your way.

[Jade]

How to set up a Minecraft Server on Debian. (Bonus: How to install Screen.)

 

If you haven't already installed Java, follow my other post's instructions. You can find that here.

 

Going to set this up in two sections -- vanilla Minecraft, and Bukkit. Most any online server is running Bukkit or some variation of it.

 

First, you'll need to run on over to the Minecraft downloads page. Once there, locate the .jar (typically on the last line of the "Multiplayer Server" section.) Take note of the URL, and for this tutorial, we'll be using the latest version as of posting.

Make a directory so things aren't all running amuck in your home directory, then enter that directory.

Now, using the URL you noted earlier, you're going to download it into this folder.

*OPTIONAL: Rename the Minecraft server file so it's easier to manage and locate.

Now that we have the Minecraft server, and Java functioning, we can actually work on getting the server itself online. This next line will actually launch the server. The first boot will have a few errors about missing files, and it will generate those on its own. To break down the command we're running further, see the spoiler.

"java" calls the java executable installed in the other tutorial. -Xmx512M tells Java that it's only allowed to use 512MB of RAM. This can be changed to your own setup. -Xms128M tells Java that it would like for it to use a minimum of 128MB at a time. -jar tells Java that it's going to be running the "jarfile" minecraft.jar, and nogui stops minecraft.jar from opening the GUI that is built in (can't see it in ssh/bash/etc, anyway.)

As per Mojang's new EULA, you need to accept their EULA. My favorite SSH text editor is nano, however you can use vi or similar if you'd rather.

Now, change the "eula=false" to "eula=true", by navigating with your arrow keys. Once done, press Control+X (same time), Y, Enter. Now you're back at the familiar bash prompt. Run the Java command above, once more. The server will take a few moments to generate the world and appropriate files to run. For an example of a working first-run, you can look at the example here.

When you're finished with the generate, type /stop in that console and give it a moment while it shuts down. Now you can modify the server.properties file, and similar, again, using nano or other text editors. You'll now be able to connect to the server (assuming it's running) on port 25565, using the internal IP, eg. 192.168.0.200.

 

 

*Note: Bukkit isn't officially supported by Mojang, and therefore, has some delay in releasing updated builds. Dev builds should not be run unless you're comfortable troubleshooting and handling potentially corrupted or damaged data/files of the server/playerfiles/etc.

 

I highly recommend keeping Bukkit in its own folder to keep things tidy. You can do this by executing the following.

You're now ready to follow the remainder of the tutorial.

 

Go to the Bukkit download page and download a build of CraftBukkit. - dl.bukkit.org I recommend running the beta builds over the dev builds if you're not too sure what you're doing.

Note the URL for the latest version you wish to download - in my case, http://dl.bukkit.org/latest-beta/craftbukkit.jar .

 

Back to SSH, we run the download command.

Once done, you can run the Bukkit Jar.

It will spew a few errors on the first launch about missing files and that's perfectly a-okay. It'll make them itself. Once it says "done", you can type stop in the console.

 

I'd highly recommend modifying server.properties and maybe bukkit.yml if you feel the need. For the sake of time and simplicity, I'll only go over server.properties.

To modify the server.properties, we'll be using nano, but you can use vi, or any other familiar text editor.

You'll see something along the lines of this example. Modify this as you wish.

Press control+X, Y, Enter.

Then you can relaunch the server, and play on it using your internal network. 

 

If you wish to add plugins from Bukkit's library of thousands, you can go to http://plugins.bukkit.org, find a few that you may like (Essentials, WorldEdit, VoxelSniper, and Factions are some common ones.) Download the jar from the site/note the link, and put them in the "plugins" folder. They'll then generate their appropriate configuration and you can modify those as you please.

 

 

If you want to leave the server running when not connected to SSH, you can use screen. A simple

should do the trick, and you can now launch the server with your previously used Java command, just prefixed with screen. For example,

And to get out of a screen, you can press Ctrl+A+D. To reattach to the screen and see the console, you can do screen -list, determine the appropriate one, and then do screen -r [4 digit pid].

mkdir minecraftcd minecraft

wget https://s3.amazonaws.com/Minecraft.Download/versions/1.7.10/minecraft_server.1.7.10.jar

mv minecraft_server.1.7.10.jar minecraft.jar

java -Xmx512M -Xms128M -jar minecraft.jar nogui

nano eula.txt

mkdir bukkitcd bukkit

wget http://dl.bukkit.org/latest-beta/craftbukkit.jar

java -Xmx512M -Xms128M -jar craftbukkit.jar

nano server.properties

apt-get install screen

screen java -Xmx256M -Xms128M -jar minecraft.jar nogui

[Aadi]

IS there a guide to taking file snapshots in Linux? That would be cool if it could also expose the snapshots to the shadowcopy tab in Windows 8.1 like FreeNAS...But from what I've heard, LVM in Linux is kind of a pain to setup snapshots...Any feedback is appreciated.

[IdeaStormer]

..java install...

 

As usual there's more than one way to skin a cat, this one also allows users with non-root access to have fun with Java, from the official source of Java in case some of you think its a hack/or bogus:

 

http://docs.oracle.com/javase/7/docs/webnotes/install/linux/linux-jdk.html

 

Don't forget your browser plugins!

 

http://docs.oracle.com/javase/7/docs/webnotes/install/linux/linux-plugin-install.html

 

Great for testing new releases or older depending on you needs without having to convert your whole system over to a new/old version.

[Aadi]

Oh and BTW I have a great suggestion for you MG2R! I was looking at your guide and noticed that you had trouble with SSH-Key based logins and disabling password-based login because you would not be able to manage the server from new computers. The solution is to go to www.authy.com and setup two-factor auth on your ssh server. You have to open your own developer account (The basic tier is free...and includes enough auth calls for most home/small business) and install the Authy program onto your linux machine from this website (https://www.authy.com/products/ssh#features). Then, install the program and it will take your existing sshd_config settings and add a couple more settings and it will ask for your Authy API key...If you created a developer account, you should have created a new application from the developer account, which would have given an Authy API key...just submit that into the server and you now have two factor auth on your SSh server. (assuming you created the account and installed the authy app) Now, in order to access the SSH server, you enter the password and a two-factor authentication token from your phone's authy app

 

In order to enable two factor authentication the instructions are here https://www.authy.com/products/ssh#installation

 

Well, I gotta run, hopefully I can flesh out this guide a bit more in the near future

[Jade]

@Aadi I'll write a guide for 2FA when I get home, if  @MG2R wants it.

[Aadi]

@Aadi I'll write a guide for 2FA when I get home, if  @MG2R wants it.

Thanks, I'm a bit busy atm. I could probably write/edit it during August after I finish travelling and things like that, but feel free to start on it

[Kabini]

It would be nice if you included a how-to in regards to the installation of smartmontools to the server  .

 

I find it very useful to have smartd consistently checking the SMART status of my drives and alerting me of any issues that arise. It's already saved my ass once when an older drive I used to own started acting up. 

 

If anyone is interested, this website has a good guide: http://blog.shadypixel.com/monitoring-hard-drive-health-on-linux-with-smartmontools/

[Master Valafar]

Very nice guide, I just used it to setup my home server. There are things that should be specified and that I would have liked to seen when I had troubles :

 

Installing virtualbox was a nightmare. Basically it said that the linux-headers package wasn't found. Then I switched to various forums' advice to get from the virtualbox website using backports. They all failed. I was needing the linux-headers-3.2.04-amd64 but apt said it wasn't going to be installed. Even after switching to multiple mirrors, same error and very little progress. Said gcc-4.6 was not installable. Various lib-* packages that virtualbox depended upon were not installable.

 

Now I decided to simulate a fresh install on a VM on my Windows computer. I said to myself : "Why not try the US mirrors when you install Debian?". After having installed exactly the same way (apart from the mirror), I directly ran

$ sudo apt-get install virtualbox

and BAM ! Virtualbox installed on the Debian VM.

 

I then -after installing emacs- checked the mirrors in the VM :

$ sudo emacs /etc/apt/sources.list

deb http://ftp.us.debian.org/debian wheezy maindeb-src http://ftp.us.debian.org/debian wheezy maindeb http://security.debian.org/ wheezy/updates maindeb-src http://security.debian.org/ wheezy/updates main# wheezy-updates, previously known as 'volatile'deb http://ftp.us.debian.org/debian/ wheezy-updates maindeb-src http://ftp.us.debian.org/debian/ wheezy-updates main

So I went to my actual server that was before refusing to install most packages related to virtualbox and edited the /etc/apt/sources.list to match the configuration above.

I actually commented (putting # as first character on the line) all other mirrors. I then ran

sudo apt-get install virtualbox

and BOOM! It worked ! The linux-header that wasn't going to be installed was actually "generated".

 

tl;dr : If the linux-headers is "not going to be installed" or you're having a hard time installing Virtualbox, change your apt-get mirrors (/etc/apt/sources.list) to match the configuration above.

 

As you've probably noticed, I'm putting some not important information in my post. This is because I hope google search can actually pick-up this search and help anyone having the same difficulties. I would've loved to see someone saying that the United States was the one to go with.

[MG2R]

--snip--

Thanks, I've linked to your post in the Virtualbox howto.

[pals0007]

Need help

 

I just recently created a Ubuntu server machine I have the os on 120 gb SSD and I have (4) 2TB WD RED drives I want to raid using mdadm. I am new to this so I am very very lost. All computers (clients) in my network are windows based, so what file system should I use for the RAID 5, does it matter, and should I use samba or NFS for sharing, I have read NFS is faster please help

[ionbasa]

Need help

 

I just recently created a Ubuntu server machine I have the os on 120 gb SSD and I have (4) 2TB WD RED drives I want to raid using mdadm. I am new to this so I am very very lost. All computers (clients) in my network are windows based, so what file system should I use for the RAID 5, does it matter, and should I use samba or NFS for sharing, I have read NFS is faster please help

If you want native Windows mounting of a network drive, you have to use SAMBA. Regarding the file system, you can use anything you want, although EXT3/4 is definitely preferred.

If you opt to use NFS, you'll need a third part application to mount the network share.

[alpenwasser]

Need help

 

I just recently created a Ubuntu server machine I have the os on 120 gb SSD and I have (4) 2TB WD RED drives I want to raid using mdadm. I am new to this so I am very very lost. All computers (clients) in my network are windows based, so what file system should I use for the RAID 5, does it matter, and should I use samba or NFS for sharing, I have read NFS is faster please help

 

 

If you want native Windows mounting of a network drive, you have to use SAMBA. Regarding the file system, you can use anything you want, although EXT3/4 is definitely preferred.

If you opt to use NFS, you'll need a third part application to mount the network share.

Pretty much ^this. For providing shares from a Linux server to Windows clients,

I'd recommend SMB over NFS in almost all scenarios. If you ever have Linux clients,

you can still share the same shares via NFS if you want to, I actually do that with

our setup without issues. However, SMB works very well for Linux -> Linux as well,

you don't really need NFS.

As a reference, when I set up my server, I did various benchmarks comparing NFS and

SMB, and at least in my setup I can't legitimately say NFS is faster, quite the opposite

in fact:

(click for full res so that it's actually readable; CIFS == Samba)

Yes, the results of my experiment may or may not be applicable to your setup, but

those are the numbers I got during repeated runs (so the results were replicable

over multiple runs, not just one-offs), just to give you an idea. And yes, I did

significant research into NFS tuning and optimizing its performance, and those

are the highest numbers I was able to get for NFS.

As for the filesystem, ext4 is a good recommendation to use on top of mdadm, but

what you use has no influence on your Samba or NFS shares. Basically, the server

will deal with the filesystem itself and just provide the data to the Windows clients,

they won't notice what file system is actually running, that's part of the purpose

of NFS/Samba (a bit oversimplified, but you get the idea); heck you can even provide

different shares using different filesystems and the clients won't notice a thing.

Aside from ext4, I'm also using JFS and XFS, in practice I haven't really noticed

a difference in my usage scenarios to be honest (and then there's of course ZFS,

but that's an entirely different beast).

[pals0007]

 

 

As for the filesystem, ext4 is a good recommendation to use on top of mdadm, but

what you use has no influence on your Samba or NFS shares. Basically, the server

will deal with the filesystem itself and just provide the data to the Windows clients,

they won't notice what file system is actually running, that's part of the purpose

of NFS/Samba (a bit oversimplified, but you get the idea); heck you can even provide

different shares using different filesystems and the clients won't notice a thing.

Aside from ext4, I'm also using JFS and XFS, in practice I haven't really noticed

a difference in my usage scenarios to be honest (and then there's of course ZFS,

but that's an entirely different beast).

I am having trouble setting up mdadm

 

these are the drives I want to use:

sdb1

sdc1

sdd1

sde1

 

The command I used was:

sudo mdadm --create /dev/md0 --chunk=64 --level=5 --raid-devices=4 /dev/sdb1 /dev/sdc1 /dev/sdd1 /dev/sde1

 

I am stuck on what commands to use next?

 

Also should I format the drives first?

 

And where do I put the command to use ext4?

[IdeaStormer]

I am having trouble setting up mdadm

 

these are the drives I want to use:

sdb1

sdc1

sdd1

sde1

 

The command I used was:

sudo mdadm --create /dev/md0 --chunk=64 --level=5 --raid-devices=4 /dev/sdb1 /dev/sdc1 /dev/sdd1 /dev/sde1

 

I am stuck on what commands to use next?

 

Also should I format the drives first?

 

And where do I put the command to use ext4?

 

This should help you out, the best software RAID tutorial I have found: http://www.iprobot.net/linux-tutorial/how-to-setup-software-raid-in-debian-ubuntu/

 

You're just missing the mkfs.ext4 command, see the link for details. I would first break your RAID array to experience the bad before putting actual data on it. You can't live in a perfect world without knowing what it looks like broken a few times.

[Shaqalac]

Hi guys

 

It's been a while since I've posted here but the time has come to expand/grow my Raid 6 array with a HDD similar to the four existing HDDs in the Raid array.

 

I'm a bit unsure of how it's done as I've read a few different ways of doing it.

First place I looked was at the Wiki page for Growing, but it doesn't seem very in-depth and lack some information.

I've also looked at two guides.

First guide.

Second guide.

The first guide seems rather simple, but the second one seems a bit more confusing.

So long story short, I was hoping some of you could help to clarify of which one to follow or how it should be done. I really don't want to risk 7TB of data because I screw something up.

 

Details about my system:

OS: Debian 7.4

HDDs: 4 x Seagate NAS ST4000VN000 4TB

HDD to be added: Seagate NAS ST4000VN000 4TB

RAID config: mdadm with 4 x 4TB in RAID6

RAID hot spare: N/A

File system: XFS for the RAID, and ext4 for the OS

 

@alpenwasser , @MG2R

Thanks in advance.

 

Cheers Shaqalac

[MG2R]

Such RAID. Much upgrade. Wow. So very.

 

What seems wrong/lacking to you in the wiki? The steps in there seem pretty straightforward to me...

 

I actually did the with my array already following the wiki and that went fine. Are you using LVM on top of your array?

[Shaqalac]

What seems wrong/lacking to you in the wiki? The steps in there seem pretty straightforward to me...

 

I actually did the with my array already following the wiki and that went fine. Are you using LVM on top of your array?

It's been a while since I made the post but I think the word "lacking" is misplaced. I got confused that the wiki showed two lines

mdadm --add /dev/md1 /dev/sdb3mdadm --grow --raid-devices=4 /dev/md1

and the two remaining guide show multiple lines.

 

My drives aren't partitioned as it wasn't needed to create the Raid array. So would I just need to plug the drive in, check that the drive is registered (would probably be named "sdf" as I already have an SSD with OS and four HDDs) and then running the following commands:

mdadm --add /dev/md1 /dev/sdfmdadm --grow --raid-devices=5 /dev/md1

I'm 99% sure I don't use a LVM on top of my array. Every information about my array would be in my previous post, that's be the filesystem for my array (XFS).

[MG2R]

mdadm --add /dev/md1 /dev/sdfmdadm --grow --raid-devices=5 /dev/md1

Seems good, assuming md1 is your array and sdf is the new drive

 

 

Every information about my array would be in my previous post, that's be the filesystem for my array (XFS).

Just to be sure: how do you mount the array? (what is in your fstab, if it's mounted automagically)

 

EDIT: might be easier to just show the output of `lsblk`

[Shaqalac]

Seems good, assuming md1 is your array and sdf is the new drive

I'll definitely have to double-check this but as far as I can remember that's what my array and the new drive is called. The sda, sdb and so on is an automatic naming scheme right?

 

Just to be sure: how do you mount the array? (what is in your fstab, if it's mounted automagically)

 

EDIT: might be easier to just show the output of `lsblk`

The array mount automatically on startup, that's what I think at least. When I boot the NAS my array is up and running.

I'm not home at the moment, so I'll have to check this when I get home.

So I'll just run the command "lsblk" and it should show the info you need to tell how my array is mounted? 

[MG2R]

The sda, sdb and so on is an automatic naming scheme right?

Yes, it depends on the SATA port you plug it in at (on older kernels). If you're on Debian 8, this is very random and shouldn't be trusted at all.

 

 

So I'll just run the command "lsblk" and it should show the info you need to tell how my array is mounted? 

Yes

[Shaqalac]

Yes, it depends on the SATA port you plug it in at (on older kernels). If you're on Debian 8, this is very random and shouldn't be trusted at all.

I'm still on Debian 7.4. I haven't really considered upgrading the OS for two reasons. First I don't have any problems now, so don't fix it if it ain't broken. Second, and I haven't done any research on this, I'm afraid it will be troublesome to upgrade concerning my Raid array.

 

I'll post a screenshot of lsblk as soon as I get home 

[MG2R]

I'm still on Debian 7.4. I haven't really considered upgrading the OS for two reasons. First I don't have any problems now, so don't fix it if it ain't broken. Second, and I haven't done any research on this, I'm afraid it will be troublesome to upgrade concerning my Raid array.

 

I'll post a screenshot of lsblk as soon as I get home 

The RAID array is actually no problem at all. I recently reinstalled my server with a fresh Debian 8 image and the raid got detected just fine once I installed mdadm. There are some subtle differences between 7 and 8 due to the new systemd service manager in 8, though. One of those subtle differences is in block device names.