A Beginners Guide to Debian CLI Based File Servers

[Windows7ge]

People are often intimidated by the idea of setting up a server using nothing but a CLI (Command-line Interface) but it's not as difficult as you might think. For the most basic setup there's not very much to it.

 

What I'm going to be using for this example:

• Debian Linux - Ubuntu Server 19.04
• File System - ZFS (software RAID)
• SAMBA - CIFS/SMB (for Windows clients)
• SSH/SFTP (for remote clients)

Index

Spoiler

1. Preparing the Installer
    1.1 - Preparing from Windows
    1.2 - Preparing from Linux (Debian)
    1.3 - Preparing from MacOS
2. Installing the OS
3. Installing Programs & Drivers
4. Configuring ZFS (zfsutils-linux)

    4.1 - RAID Levels
        4.1.1 - Mirror (RAID1)
        4.1.2 - Striped Mirrors (RAID10)
        4.1.3 - RAIDZ1 (RAID5)
        4.1.4 - RAIDZ2 (RAID6)
        4.1.5 - RAIDZ3 (3-bit Parity)

    4.2 - Datasets

    4.3 - Setting Permissions

        4.3.1 - Assigning a Folder/Dataset Owner

        4.3.2 - How to Create & Assign a Group

        4.3.3 - Assigning a Group to a Dataset

        4.3.4 - Setting Group Permissions
5. Configuring SMB/CIFS (samba)
    5.1 - How to Add UNIX Users
    5.2 - How to Add SAMBA Users
    5.3 - How to Configure SMB Share
6. Configuring SSH/SFTP (openssh-server)
    6.1 - Logging in over SSH/SFTP
        6.1.1 - On Windows
        6.1.2 - On Linux

7. Conclusion

 

1. Preparing the Installer

Spoiler

To start you need to download the OS. This example is going to be based on Ubuntu Server 19.04 (if you have reasons to use 18.04 or another Debian distro it should be fine but your mileage may vary). This .ISO can be prepared for installation from a Window, Linux, or MacOS computer. To do so download the .ISO & plug a thumb drive that is 8GB in size or more into the computer.

 

1.1 - Preparing from Windows - Tested with ubuntu-19.04-live-server-amd64.iso by Windows7ge

Spoiler

Download and install a program called Rufus. Rufus is a very convenient small application for creating bootable USB media. If no other thumb drives are connected it should detect and select the drive automatically. If not make sure you select the one you want to use as it will be formatted and all existing data on the drive will be deleted.

 

As of Rufus V3.6 click on "SELECT". This should bring up a search box. From here navigate to where your .ISO file was downloaded and click on it, then Open. You'll be brought back to the main Rufus page and the settings below will have changed. For most situations these settings will be fine but depending on your specific setup it may be desired to change some variables based on your hardware.

 

1.2 - Preparing from Linux (Debian) - Tested with ubuntu-19.04-live-server-amd64.iso by Windows7ge

Spoiler

Open a terminal and run the command:

lsblk

This will list your connected drives, capacities, & partitions. From this find the name of your thumb drive which should follow the structure "sd*".

Once you have that information make sure the thumb drive isn't mounted by using the following command (replace "c" with your drive letter):

umount /dev/sdc

You may receive an error saying your drive wasn't mounted. That's good. We just wanted to make sure that it wasn't.

Next up partitioning the drive. If you have a reason to use a different file system (such as FAT32) that's OK but for this example we're going to use ext4 (replace "c" with your drive letter):

sudo mkfs.ext4 /dev/sdc

Once your drive is formatted you can write your .ISO file to the thumb drive with:

sudo dd if=ubuntu-19.04-live-server-amd64.iso of=/dev/sdc status=progress

NOTE: Where "ubuntu-19.04-live-server-amd64".iso is the name of your .ISO file and sd"c" is the name of your thumb drive.

 

1.3 - Preparing from MacOS - Tested with ubuntu-19.04-live-server-amd64.iso by Den-Fi

Spoiler

Start by formatting the thumb drive to the FAT32 file system, next what we're going to do is use a program called Etcher.

 

NOTE: At this time Etcher does not work on MacOS Catalina as it is in beta.

 

Once you've downloaded and installed this:

1. Click "Select Image" (locate and choose our debian_server.iso)
2.  Click "Select Drive" (locate and choose the sacrificial thumb drive)
3. Click "Flash"

The bootable thumb drive should now be ready to use.

 

2. Installing the OS

Spoiler

Once your bootable USB is ready plug it into the server and turn it on. You may have to manually boot to it if USB devices aren't on the boot order list. Once the thumb drive boots you'll be brought to a GRUB menu with various options. All you want to do right now is install your OS so select "Install Ubuntu Server".

 

From here we will:

1. Select a language
   1. Update the installer (a message you may not see)
2. Keyboard Configuration
3. Network Connections
   1. You'll want to configure a Static IP so you can find the server on the network easily (be sure the IP you use won't be handed out by your Router's DHCP server)
   2. Go to your network adapter (eth) -> Edit IPv4 -> Change IPv4 Method to Manual. Now you will need to setup the Network
      1. Subnet - Example: 192.168.0.0/24
      2. Address - IP the server NIC will use. Example: 192.168.0.254
      3. Gateway - The IP address of the Default Gateway. Example: 192.168.0.1
      4. Name Servers - Your DNS servers. Example 8.8.8.8,8.8.4.4 | 1.1.1.1,1.0.0.1
      5. Search Domains - If applicable (not required)
   3. Configure proxy (if applicable)
4. Configure Ubuntu archive mirror (can be left default unless you have a reason to change it)
5. File System Setup
   1. For a basic setup you can "Use An Entire Disk"
      1. Select the disk you want to install the OS onto [Enter]
      2. If you don't have anything to modify you can just hit Done (Do NOT create a RAID here)
6. Profile setup
   1. Your name
   2. Your server's name - How other computers will see it on the network.
   3. Pick a username
   4. Choose a password
   5. Confirm your password
7. SSH Setup
   1. Install OpenSSH server - Tick this box.
   2. Import SSH identity - No (unless you have a reason to)
8. Featured Server Snaps
   1. There will be many snaps listed here. If you don't need any of them all can be left un-selected
9. Installation Complete!
   1. It may start downloading Security Updates. Allow this to finish then hit Reboot

Once all of the setup here is complete and you reboot you'll be brought to a login screen. Once you login you'll be brought to a prompt. You want to make sure all of your packages are up to date so run the following commands:

First you need to become root:

sudo -i

This will prompt you for your user password. Type this in and hit Enter. Next to run the updater:

apt update && apt upgrade

Once this finishes the system may need to be restarted. After this you're up-to-date.

 

3. Installing Programs & Drivers

Spoiler

With your OS installed and your packages up to date you can install the components you'll need. During the OS setup you may or may not have remembered to install the SSH(openssh-server) package. If you did you can omit it from the command below. If you didn't we will install it now. As root run the command:

apt install zfsutils-linux samba openssh-server

1. ZFS (Zettabyte File System) is our drive pool file system. This will also be responsible for handing our RAID pool.
2. SAMBA is our SMB/CIFS client that will allow our Windows clients to connect to the server over the local network.
3. OpenSSH is our remote access client that will allow us to SSH/SFTP into the server from remote locations.

You can check if packages have or have not been installed with:

dpkg -s package-name | grep Status
Ex. dpkg -s openssh-server | grep Status

NOTE: Linux is case sensitive. The "S" in Status must be capitalized.

 

If you receive the following output:

dpkg-query: package 'package-name' is not installed and no information is available
Use dpkg --info (= dpkg-deb --info) to examine archive files.

Then the package is not installed. If it is installed you will see this:

Status: install ok installed

Once you've verified your packages are installed you can move on to Step 4.

 

4. Configuring ZFS (zfsutils-linux)

Spoiler

ZFS is a very powerful, very resilient File System that prioritizes data integrity over speed. This does not mean that you cannot achieve high performance on a 10GBASE-T or higher network it simply means there are faster file systems out there. However, for this tutorial we are going to prioritize protecting the data.

 

NOTE: RAID is not a backup. You should always keep an independent copy of your data as any system regardless of file system is capable of losing everything.

 

Start by identifying the drives you plan to use with the command:

lsblk

Ideally the drives will all be of the same capacity so they should be easy to tell from your boot drive or other miscellaneous drives. The identifier you're looking for is "sd*" where * is the unique letter given to the drive(s) you want to use for the ZFS pool (your RAID).

 

4.1 - RAID Levels

Spoiler

There are a variety of different RAID levels to choose from and which one is best will be determined by your intended application.

 

4.1.1 - Mirror (RAID1)

Spoiler

Related Information:

1. Requires 2 disks or more.
2. Allows at least 1 disk to fail while keeping the pool online & data intact.
3. Fine for small office/home office applications.
4. Of the total storage only 50% is usable.
5. Does not improve Read/Write performance.

Example command:

zpool create pool-name mirror /dev/sdb /dev/sdc

 

4.1.2 - Striped Mirrors (RAID10)

Spoiler

Related Information:

1. Requires 4 disks or more (even numbers only).
2. Allows at least 2 disks to fail if they contain different chunks of data.
3. Ideal for VM server applications (increased IOPS in comparison to parity RAID).
4. You lose 50% of your usable storage with a 2 way mirror.
5. Read/Write performance improves.

Example Command:

zpool create pool-name mirror /dev/sdb /dev/sdc mirror /dev/sdd /dev/sde

 

4.1.3 - RAIDZ1 (RAID5)

Spoiler

Related Information:

1. Requires 3 disks or more.
2. Allows 1 disk to fail while keeping the pool online & data intact.
3. You lose 1 disk worth of usable storage.
4. Improved Read/Write performance

 Example Command:

zpool create pool-name raidz /dev/sdb /dev/sdc /dev/sdd

 

4.1.4 - RAIDZ2 (RAID6)

Spoiler

Related Information:

1. Requires 4 disks or more (recommend 5 or more).
2. Allows 2 disks to fail while keeping the pool online & data intact.
3. You lose 2 disks worth of usable storage.
4. Improved Read/Write performance. Less performance over raidz1 (for equal number of disks)

Example Command:

zpool create pool-name raidz2 /dev/sdb /dev/sdc /dev/sdd /dev/sde /dev/sdf

 

4.1.5 - RAIDZ3 (3-bit Parity)

Spoiler

Related Information:

1.  Requires 6 disks or more (recommend 7 or more)
2. Allows 3 disks to fail while keeping the pool online and data intact.
3. You lose 3 disks worth of usable storage
4. Improved Read/Write performance. Less performance over raidz1/raidz2 (for equal number of disks)

 Example Command:

zpool create pool-name raidz3 /dev/sdb /dev/sdc /dev/sdd /dev/sde /dev/sdf /dev/sdg /dev/sdh

 

Once you've formed your pool if you have not received any errors run the following command to verify that the pool is healthy and online:

zpool status

In the event that you made a mistake while forming the pool you can start over by destroying it:

zpool destory pool-name

Right now your pool should be showing up in your "/pool-name" directory. You can navigate there with:

cd /pool-name

 If you would like to view the file system to see what your usable storage in the pool is run the command:

df -h /pool-name

 

4.2 - Datasets

Spoiler

Creating datasets are a way to increase functionality and control within a ZFS pool. It adds features such as permissions, compression, deduplication, snapshots, and quotas to individual sections of the pool. These things won't be discussing as they go beyond the scope of this guide but you can research them if you're interested.

 

To crease a dataset use the following command as an example:

sudo zfs create pool-name/kate

Within the pool (pool-name) this will create a dataset named kate. Datasets act like folders with more functionality. If you ever want to check the used space of the datasets you've created you can run the command:

zfs list

This is a way to track how much space each dataset takes up. You can create as many datasets as you like but it's generally recommended to create 1 dataset per data type. Media, photos, text documents, spredsheets, programs, etc. You can also create datasets for each user if your use case can benefit from the features that offers.

 

4.3 - Setting Permissions

Spoiler

Before you can use your network share for anything you'll need the permission to read/write/execute. Once you've created your datasets it's recommended to give dataset permissions to the user(s) who will need access to them.

NOTE: Do not change the owner of the parent dataset (your pool) leave the owner root. Datasets within your pool can be given to users who need access.

 

4.3.1 - Assigning a Folder/Dataset Owner

Spoiler

To assign a user a dataset:

sudo chown kate:kate /pool-name/kate

What this has done is assigned kate who belongs to the group kate to her own dataset we had created in step 4.2. When we configure SAMBA in step 5 and kate logs in she will have full control over the contents of this dataset.

 

This can be repeated for each user.

 

4.3.2 - How to Create & Assign a Group

Spoiler

If you want multiple users to have access to the same dataset create a group:

sudo groupadd employees

You can replace "employees" with anything so long as it's one word. Recommend all lowercase.

Now add each of your users to the group:

sudo usermod -a -G employees kate
sudo usermod -a -G employees dale
sudo usermod -a -G employees matt

You can verify if they are currently in the group by running:

id kate
id dale
id matt

If the employees (or other) group doesn't show up log the user out and back in. It should be there now.

 

4.3.3 - Assigning a Group to a Dataset

Spoiler

To assign the group to the dataset you want people to share:

sudo chown kate:employees /pool-name/kate

Now kate is the owner but everyone in the employees group can have permission to access the folder contents. Setting specific permissions will be discussed next.

 

4.3.4 - Setting Group Permissions

Spoiler

The following diagram will help explain how assigning folder/dataset permissions are done:

 Owner Group Other
4 r     r     r
2  w     w     w
1   x     x     x

What you do is add the numbers together for the permissions you want the file/folder/dataset to have. When you run the command "ls -l" permissions are listed on the left of the file name.

 

Using the diagram above to set the permissions for our employees group we want:

chmod 775 kate

What we have now is Kate and everyone else who belongs to the employee's group can now access the dataset.

 

If we wanted to lock out everyone not in the group we could use the command:

chmod 770 kate

Now only Kate and everyone else in the group can access the folder. All others will be denied entry.

 

For more explanation on how to setup permission controls you can do more research on your own.

 

With that the pool is ready to be used.

 

5. Configuring SMB/CIFS (samba)

Spoiler

SAMBA is what will allow your Windows clients to talk to the server over the network. You have already installed the service (if not please refer to step 3) now you need to configure it.

 

5.1 - How to Add UNIX Users

Spoiler

Adding UNIX users is only a couple of step process. Use the command below as a example:

sudo adduser kate

It will then prompt you for:

1. New password
   1. Retype new password
2. Full Name []: - Can be left blank
3. Room Number []: - Can be left blank
4. Work Phone []: - Can be left blank
5. Home Phone []: - Can be left blank
6. Other []: - Can be left blank

If the data is correct hit Y

 The new user kate has been added. Repeat this for however many users you intend to have.

 

5.2 - How to Add SAMBA Users

Spoiler

In addition to UNIX users which are for the system itself SAMBA keeps a list of it's own users & passwords. This needs to be populated in order for people on the network to access the share. Use the command below as an example for how to add users to SAMBA:

sudo smbpasswd -a kate

This will create a SAMBA account for your UNIX user kate. You will be prompted to create a password for each account.

NOTE: SAMBA account passwords can be different or the same as their UNIX account passwords. Which is better depends on your application.

 

Repeat this for each user you want to give access to the share.

 

5.3 - How to Configure the SMB Share

Spoiler

For CLI based text editors two are installed by default, vim & nano. Of the two nano is simpler to use but vim has greater functionality. Being a beginners guide we'll use the simpler of the two. To edit the SMB config run the following command:

sudo nano /etc/samba/smb.conf

You will be brought into the nano text editor and the smb.conf file will be loaded. Using the down arrow key go to the very end of the file. There you will append the File Share:

[share-name]

path = /pool-name
available = yes
valid users = user1 user2 user3
read only = no
browseable = yes
public = yes
writable = yes

After adding the above lines you only need to edit 3 variables:

1. [share-name] - In-between the brackets is what the Windows users will see as the name for the share folder on the network. This name must not contain any spaces.
2. path - This points to the folder you want network users to use for file storage. You want to point the path to your ZFS pool discussed in Step 4.
3. valid users - This is your permission list for users with a SAMBA account/password to access the share (UNIX usernames should match SAMBA usernames)

After you finish your edits Ctrl+O then Enter to save then Ctrl+X to Exit.

 

Now restart the SMB service:

sudo /etc/init.d/smbd restart

To make sure everything was done correctly test the smb.conf file with the command:

testparm

You want to see an output similar to:

Load smb config files from /etc/samba/smb/conf
Loaded services file OK.
Server role: ROLE_STANDALONE

Press enter to see a dump of your service definitions

If you receive an error instead re-check your smb.conf file for typos.

Beyond this the Network Share is ready to use.

 

6. Configuring SSH/SFTP (openssh-server)

Spoiler

If all you want is password level protection then there isn't much for you to do here. By default with OpenSSH installed and running you can remote in using your UNIX username & password.

 

6.1 - Logging in over SSH/SFTP

Spoiler

6.1.1 - On Windows

Spoiler

Windows has a number of programs to let you remote into clients/servers a few of the most popular include PuTTY & WinSCP.

1. PuTTY - is a SSH program that allows you to remote into the CLI of your servers.
2. WinSCP - is a GUI SFTP program that allows you to remote into the folders of your server.

The programs are fairly self-explanatory so I won't go over them here but if you need help there are lots of online resources for both.

 

If you want something that adds functionality to Windows while acting as an SSH client you can look into Git. Git adds some Linux like functionality to Windows while allowing you to remote into other Linux systems. How to login using this method will be discussed in Step 6.1.2.

 

You can also explore WSL (Windows Sub-system for Linux) which enables you to run Linux on Windows. This will allow you to use the Linux SSH client discussed in Step 6.1.2 on Windows.

 

6.1.2 - On Linux

Spoiler

There are multiple methods available on Linux to remote into a server. Two include SSH (built-in to the Terminal) & FileZilla

 

Remoting into a Linux server from a Terminal is easy (the following is an example):

ssh kate@192.168.0.254

Make sure the user you want to login as exists on the server before trying to login as said user. You may receive a message about trusting the source. If you know the address is correct you can say Y.

 

FileZilla is a nice GUI application for making SFTP connections. Guides online can teach you how to use it.

 

For both Windows & Linux these methods can be applied to port forwarding to allow you to connect to the server from outside the network. I will not be going into the details of port forwarding but something to note is by default OpenSSH will use port 22. This is what you'll want to configure in your router for the local IP address of the server. It's ill-advised to use port 22 on the public side of your router as it is common for bots on the Internet to use that port to attempt to access your network. If you can try to use a port >20,000 and even better to pick a port not being officially used by any protocols/applications.

 

 

7. Conclusion

Spoiler

For as much information as there appears to be it's not an overly complicated process to set up something like this on your own. Steps can be omitted to suit your use case simplifying the process further.

 

If anyone sees any errors or something that needs revising let me know. If anybody thinks of something that should be added let me know and I'll look into it.

[Master Disaster]

Excellent job, very useful for newcomers.

 

You could expand this by including Apache (or indeed LAMP) or NGINX, Transmission, Maria/mySQL and miniDNLA.

 

That's most of what most people would want running on a server.

 

I realise LAMP is a pretty involved process though, there's lots of config files to edit in order to get everything running properly.

[Windows7ge]

8 hours ago, Master Disaster said:

Excellent job, very useful for newcomers.

 

You could expand this by including Apache (or indeed LAMP) or NGINX, Transmission, Maria/mySQL and miniDNLA.

 

That's most of what most people would want running on a server.

 

I realise LAMP is a pretty involved process though, there's lots of config files to edit in order to get everything running properly.

Thank you. I put a significant amount of time & research into compiling tutorials.

 

The thought entered my mind to write tutorials based on CentOS, OpenSUSE, or RHEL. Of the suggestions you've listed I only have a basic understanding of CentOS, MySQL, & Apache as I was forced to take a class on this in college. I'm not familiar with any of the other acronyms.

 

A lot of what I write about pertain to personal projects or other relative information. I don't have a use for CentOS, MySQL, or things like phpMyAdmin, etc. Putting the time into researching them to write a tutorial I don't feel would yield a good tutorial. For me motivation is a necessity and that motivation comes from me having a use for what I'm writing about.

 

Don't take that as discouragement though, I'm open to guide suggestions especially in the networking/server domain. I'm also looking to help people with common/general issues too though. Things like re-occurring problems submitted to the forum. "How do I install Windows 10?", "I'm new to FreeNAS...", "How do I dual-boot Windows/Linux?", etc. If you see a reoccurring problem being submitted that a guide/tutorial can solve let me know, I'll add it to the queue.

 

I'm also open to "information donations". If you have a guide/tutorial idea that I don't have confidence in doing myself but you don't want to take the time to make/submit if you're willing to provide me the raw bulk of the data I'll figure out the finer details (or ask for them) and compile them into something that should be easily comprehensive to a beginner. Of course I will credit you for all of the information you provided.

[Bacon soup]

Nice and simple, exactly what a beginner needs. Here is some additional software that I recommend:

 

For linux clients I recommend sshfs and fuse to remotely mount filesystems, and rsync to transfer files.

 

Next cloud is a private cloud software stack that syncs files between smart phones and computers. All the photos I take on my phone appear on my PC automatically, and all the pod casts I down load on my PC get automatically sent to my phone. It's awesome.

[Windows7ge]

1 hour ago, Bacon soup said:

For linux clients I recommend sshfs and fuse to remotely mount filesystems, and rsync to transfer files.

Like how Windows allows you to map drive letters to network drives this sounds like the Linux equivalent of that. Would I be correct? If so this is something I would very much like to explore and can add to the guide.

 

I did a guide on how to use rsync on Windows via WSL. I feel like going into how to backup the data here would warrant a guide of it's own. This guide is more about how to just get the one server setup initially and connecting to it.

 

1 hour ago, Bacon soup said:

Next cloud is a private cloud software stack that syncs files between smart phones and computers. All the photos I take on my phone appear on my PC automatically, and all the pod casts I down load on my PC get automatically sent to my phone. It's awesome.

Going into phones isn't really something I'm interested in. Desktops/Laptops/Networking/Servers maybe I've missed one but these provide enough work to write guides on. Personally I'm not that into cellphone apps.

[Bacon soup]

41 minutes ago, Windows7ge said:

Like how Windows allows you to map drive letters to network drives this sounds like the Linux equivalent of that. Would I be correct? If so this is something I would very much like to explore and can add to the guide.

 

you can mount the remote filesystem anywhere and it appears like a local folder. In linux you can mount almost anything to almost anything. By exporting ssh keys you can have network locations mount automatically. Before I used NextCloud, I had my desktop and laptop mount each other's home folders whenever I was home.

 

 

 

[Windows7ge]

53 minutes ago, Bacon soup said:

you can mount the remote filesystem anywhere and it appears like a local folder. In linux you can mount almost anything to almost anything. By exporting ssh keys you can have network locations mount automatically.

Ah, it uses the ssh protocol. This is a problem for my own setup as I use 10Gbit and SSH has far too much overhead. Do you know of a way to automatically mount a network drive while using SMB? I'm a little familiar with fstab for mounting local disks but I don't know if it can be used for network shares.

[Bacon soup]

4 minutes ago, Windows7ge said:

Ah, it uses the ssh protocol. This is a problem for my own setup as I use 10Gbit and SSH has far too much overhead. Do you know of a way to automatically mount a network drive while using SMB? I'm a little familiar with fstab for mounting local disks but I don't know if it can be used for network shares.

sorry i havnt needed SMB since I started using nextcloud. You can mount network locations in fstab. Archwiki should have something on it.

[Windows7ge]

5 minutes ago, Bacon soup said:

sorry i havnt needed SMB since I started using nextcloud. You can mount network locations in fstab. Archwiki should have something on it.

Yeah, if you don't use big files SSH is great. When you deal primarily with files that are 10~20GB SSH is a nightmare. So slow. SMB works quite a bit better. I haven't tested NFS yet because I'm working with cross-platform but for Linux server to Linux client I need to see how much better/worse it performs.

 

I'll add to this guide:

1. How to map network drives (Windows)
2. How to map network shares (linux - debian)

I will research sshfs & fuse but I have mixed feeling on weather or not it's worth adding as I've already included a Linux CLI & GUI method for connecting remotely.

[Jisagi]

Some additions (for debian servers):

• The latest debian 10 (buster) only uses zfs 0.7.x, but the backports already use 0.8.x, which has some really nice new feature alongside a lot of performance improvements and fixes. It might therefore be better to use the newer backport version, especially if you run something like an SSD storage array, because of the added TRIM support in 0.8.x. Full changelog HERE. A simple entry inside "/etc/apt/sources.list" with "deb <repo-link> buster-backports main contrib" (and deb-src) before installing zfs-dkms and zfsutils-linux will use 0.8.x instead of 0.7.x. You have to omit "contrib" from the normal repo, otherwise it might still use the older version and not the backports. I think you didnt mention to add contrib at all anywhere, which will result in apt saying it cannot locate the package, be it 0.7 or 0.8.
• When installing zfs, apt will open a notice with blue background that zfs and debian don't like eachother because licenses. Some people might not know why this is shown at all and it might scare them, despite it not being a "problem". It can safely be ignored.
• It might be a good addition to 3 (or 4), that zfs must be loaded manually as kernel module. After installing zfs, apt(-get) says "modprobe: FATAL: Module zfs not found in directory /lib/modules/4.x-x-x-amd64" and "ZFS modules are not loaded". A user must load it with "(/sbin/)modprobe zfs", otherwise the zfs service cannot be started.

[Windows7ge]

On 2/14/2020 at 1:55 PM, Jisagi said:

Some additions (for debian servers):

• The latest debian 10 (buster) only uses zfs 0.7.x, but the backports already use 0.8.x, which has some really nice new feature alongside a lot of performance improvements and fixes. It might therefore be better to use the newer backport version, especially if you run something like an SSD storage array, because of the added TRIM support in 0.8.x. Full changelog HERE. A simple entry inside "/etc/apt/sources.list" with "deb <repo-link> buster-backports main contrib" (and deb-src) before installing zfs-dkms and zfsutils-linux will use 0.8.x instead of 0.7.x. You have to omit "contrib" from the normal repo, otherwise it might still use the older version and not the backports. I think you didnt mention to add contrib at all anywhere, which will result in apt saying it cannot locate the package, be it 0.7 or 0.8.
• When installing zfs, apt will open a notice with blue background that zfs and debian don't like eachother because licenses. Some people might not know why this is shown at all and it might scare them, despite it not being a "problem". It can safely be ignored.
• It might be a good addition to 3 (or 4), that zfs must be loaded manually as kernel module. After installing zfs, apt(-get) says "modprobe: FATAL: Module zfs not found in directory /lib/modules/4.x-x-x-amd64" and "ZFS modules are not loaded". A user must load it with "(/sbin/)modprobe zfs", otherwise the zfs service cannot be started.

This is a little over my head but if I understand it correctly you're basically talking about that ZFS can be updated to a later version manually (not the default version in package manager for zfs-utils) and that the newer version has to be loaded into the system manually on startup for unknown reasons.

[Jisagi]

29 minutes ago, Windows7ge said:

This is a little over my head but if I understand it correctly you're basically talking about that ZFS can be updated to a later version manually (not the default version in package manager for zfs-utils) and that the newer version has to be loaded into the system manually on startup for unknown reasons.

Close, but no. I read your original post again and now I'm not so sure what OS we are talking about anymore. I was specifically talking about debian (not ubuntu!!), no matter which version of it. Same goes for zfs on debian, no matter which version of that either. debian and linux never liked eachother because of the nature of their licenses and oracle being part of the whole conversation. The short version, ANY zfs version on debian has to be loaded like I described it. You install it, load the kernel module with modprobe and then start the service as usual.

The new(er) version of zfs I suggested was just an additional thing I proposed, because it really has some neat new features some people might want to have, but it has nothing to do with the installation stuff from above.

 

EDIT: Here a pic of how that license "popup" looks like on debian. (Copypasta from linuxhint btw)

Edited February 15, 2020 by Jisagi

[Windows7ge]

11 minutes ago, Jisagi said:

The short version, ANY zfs version on debian has to be loaded like I described it. You install it, load the kernel module with modprobe and then start the service as usual.

Still over my head. For the application of this guide all I do is pull it from the repositories and it's ready to use. No manual loading of the kernel module. No manual starting of the service. No manual editing of any files so it loads at start-up.

 

If any of those are required the process of installing it does them automatically.

[Jisagi]

9 minutes ago, Windows7ge said:

Still over my head. For the application of this guide all I do is pull it from package manager and it's ready to use. No manual loading of the kernel module. No manual starting of the service. No manual editing of any files so it loads at start-up.

 

If any of those are required the process of installing it does them automatically.

I tested it yesterday again and no, it did NOT work out of the box for me. I had to go the "load it manually" and tbh, I can't see any other way on debian for it to work out of the box at all. If it would, there would be no problem between out good ol linus torvalds, oracle and all those open source licenses. Or rather, it doesn't work out of the box because of those things and maybe never will.

 

Test was done with the latest debian 10 netinstall as of yesterday.

 

EDIT: Adding this here again, I do not know how this is handled on ubuntu, only debian!

 

EDIT2: About to test this again with v0.7 as well, just to be sure.

 

EDIT3: Adding the "contrib" source on debina is a must though, no matter which zfs version. Otherwise it will not find neither find zfsutils-linux nor zfs-dkms

[Windows7ge]

6 minutes ago, Jisagi said:

I tested it yesterday again and no, it did NOT work out of the box for me. I had to go the "load it manually" and tbh, I can't see any other way on debian for it to work out of the box at all. If it would, there would be no problem between out good ol linus torvalds, oracle and all those open source licenses. Or rather, it doesn't work out of the box because of those things and maybe never will.

 

Test was done with the latest debian 10 netinstall as of yesterday.

 

EDIT: Adding this here again, I do not know how this is handled on ubuntu, only debian!

Then I would say in here lies the difference. Couple weeks ago I created a VM using Ubuntu Server 19.10. Installed ZFS (sudo apt install zfs-utils) and with no additional steps I had zpool and zfs commands at my disposal. Made my pools. Did some testing. Restarted the VM. Pool automatically remounted. I didn't have to do any extra work to install ZFS.

 

It's worth mentioning that because there are so many different distributions of Linux based on various different kernels (Debian/OpenSUSE/RHEL/etc) there's UNIX/ONIX and sub-categories each of which can operate on different principals (Ubuntu/Mint/Lubuntu/PopOS/etc.) that just because Ubuntu is based on Debian it doesn't necessarily mean that using the distro Ubuntu was based off of is going to result in the same exact setup process.

 

I'm still a greenhorn when it comes to the Linux world. I'm going to be uploading a new tutorial today or tomorrow. A guide to using GNU/Linux (Ubuntu) and Microsoft Windows 10 simultaneously with one keyboard and mouse though virtualization, GPU pass-though & Looking Glass.

 

The tutorial is going to be longer and even more complex than this one. So if you're interested at all you can watch out for that one.

[Jisagi]

Sure, I'll be watching out for that then. And yeah, debian seems to be fairly different to ubuntu then. Especially that license notice can be "frightening" for new users, who just follow the standard procedure of a guide.

 

Btw, which version of zfs-dkms does ubuntu install by default. Knowing ubuntu in general and compared to debian, it usually is on the more "up-to-date" side of things when it comes to new version of basically all included software. Is it already running 0.8.x or still 0.7.x? You can check by running something like "apt-cache policy zfs-dkms" to see the package managers version of said software package. This should work on ubuntu as well (I guess).

[Windows7ge]

Comes up as "0.8.1-1ubuntu14.3".

[Jisagi]

10 hours ago, Windows7ge said:

Comes up as "0.8.1-1ubuntu14.3".

Not bad, but not as "latest" as i would've thought ubuntu would include. debian includes 0.8.3 on the backports and the same version on the newer version of debian, which is the latest version since roughly a month now. 0.8.1 is from iirc June last year.

[WkdPaul]

Added to the Guides & Tutorials catalog