Input requested for a mini HTTP/HTTPS/SteamCache Server

Fishscene · August 21, 2019

Greetings!

I'm looking to build a cache server for home to help maximize my internet connection as well as a fun project for me to do. Here's the ideal server that I'm looking for:

Software

- cache HTTP (fairly easy to do)

- cache HTTPS (probably the most difficult part of this server build)

- cache steam game downloads

Hardware

- Low-power

- Ideally Able to deliver content at least half of gigabit speed. Preferably faster of course. (I'm open to dropping a 10 gigabit card)

- Ideally small

Internet connection 100mbit (Supposed to be 250, but that's another issue to work out with Comcast after a fence post was sunk into my cable line and they re-ran a new cable)

To head off questions about my home network setup, I'm building out with 10gigabit in mind. All my network cable is CAT6A, but my switches and NIC's are on gigabit, but I'm moving towards a 10gigabit LAN.

-------

I did some preliminary testing of the Steam cache server with Fedora Server on a 5-year old laptop with an SSD and I was quite disappointed that the download speeds off this laptop was less than half of what I was getting through my 100mbit Internet connection. I'm not really sure where the bottleneck was, but I was pleased that it was actually caching and working.

I've done HTTP caching in the past and am well aware of stale caching. I'm also aware that with HTTPS caching, I'll need to set up a certificate to have installed on my clients.

Here's where I could especially use some help:

- A good tutorial on HTTPS caching in squid on Fedora (I can usually adapt from Ubuntu instructions, though Arch/Gentoo is out of reach of my current skill set)

- Hardware. The laptop performance was dismal even with the SSD. I'm not sure where the bottleneck was, but I'd love to find a small form-factor PC that could have 1 or 2 SSD's (And more if needed to deliver faster) that can deliver at gigabit rates or faster (with a 10G card)

Thoughts?

Electronics Wizardy · August 21, 2019

What clients and how many? If you don't have a lot of clients, caching won't help much. Your browser already does a lot of caching of websites aswell.

With the laptop did you look at what part of the system was being used the most?

Do you have a rack or do you want this desktop to be a desktop? Id personally get a used dell or hp server, and run this in a vm with other servers, but you could also get a new nuc.

Fishscene · August 21, 2019

39 minutes ago, Electronics Wizardy said:

What clients and how many? If you don't have a lot of clients, caching won't help much. Your browser already does a lot of caching of websites aswell.

With the laptop did you look at what part of the system was being used the most?

Do you have a rack or do you want this desktop to be a desktop? Id personally get a used dell or hp server, and run this in a vm with other servers, but you could also get a new nuc.

3 Windows 10 clients.

Centralized cache management wouldn't necessarily be for speeding up the Internet (though load times can be split second and all household members would be contributing to the cache). There's a couple of reasons I have to web caching (HTTP/HTTPS):

1) Fun learning project.

2) Smoothing out the web experience when the bandwidth is maximized.

With the laptop, I just kind of figured there was a bottleneck somewhere and I needed better hardware. I didn't bother gathering performance data as it was really just a test of the software to see if I could get it working under Fedora - which was a success. I was always planning on purchasing a small server for this project - I had to re-purpose the laptop temporarily for testing the Steam Cache and I placed it back in to normal duties when the Steam Cache was a success. I can get the spec's if you really want them, but it'll take me a bit to spin up the steam cache and get some info to you.

I do have a small enclosed rack and I'm having problems with some of my equipment in there getting a bit too warm for my taste. As such, I haven't been looking at rackmount, but more of a small/mini computer that I can stick on top of my rack.

One of the items in my rack is an OLD (~15 years old) Thecus N7700 NAS that I'm planning on replacing with a Synology DS1618+ and stuff with 32GB of RAM to host Virtual Machines. However, finding stats on VM performance is difficult to come by and the NAS replacement won't take place for at least another year, so I figured I'd get best performance for the cache server out of a dedicated box.

I've been eyeballing the NUC, but again, no idea what kind of performance I'd get out of it as a few of them are Celeron processors and I'm unfamiliar with that lineup. The testing cache laptop had an i3.

Thanks for responding so fast! Let me know if you'd like to know anything else.

Electronics Wizardy · August 21, 2019

Just now, Fishscene said:

I'm planning on replacing with a Synology DS1618+ and stuff with 32GB of RAM to host Virtual Machines.

Id really stay away from these for vms, id get a nuc or simmilar and run a normal hypervisor like esxi, hyper-v, proxmox or others.

Just now, Fishscene said:

I've been eyeballing the NUC, but again, no idea what kind of performance I'd get out of it as a few of them are Celeron processors and I'm unfamiliar with that lineup. The testing cache laptop had an i3.

Many have the quad core i5s.

You can also get a used desktop like a optiplex 7020.

Id just run this on a vm with the rest of the services, no reason to give it its own box.

Just now, Fishscene said:

Smoothing out the web experience when the bandwidth is maximized.

I really don't think this will help here, You don't have many clients. I would focus on setting up qos and limiting bandwidth on your route first instead of caching with this setup here.

Just now, Fishscene said:

With the laptop, I just kind of figured there was a bottleneck somewhere and I needed better hardware. I didn't bother gathering performance data as it was really just a test of the software to see if I could get it working under Fedora - which was a success. I was always planning on purchasing a small server for this project - I had to re-purpose the laptop temporarily for testing the Steam Cache and I placed it back in to normal duties when the Steam Cache was a success. I can get the spec's if you really want them, but it'll take me a bit to spin up the steam cache and get some info to you.

Knowing what was limiting performance would really help here. Don't just buy faster hardware until you first know what the slow part is.