Steam Caching Tutorial

[jakkuh_t]

Heyo! Here's a little tutorial to accompany our "Downloading Games at 10 GIGABIT?" video.

 

I know it looks like a lot, but it's actually not that hard to setup.

 

This tutorial is designed around Ubuntu 18.04 Desktop, but should be adaptable to most Linux distros. We aren't going to go into detail about setting up your storage medium, so you will have to do your own research on that. Here's a link to a good article on how to setup a software RAID in Ubuntu if that's what you intend on using.

 

1. Setup a non-root user to issue docker commands.
   • If you are using 18.04 Desktop, you should already be using a non-root account w/ sudo access
   • Otherwise add a user, replacing "USERNAME" with your desired username while logged in as root.

     adduser USERNAME

   • If you wish to run docker commands via sudo, add the user to the sudo group:

     usermod -aG sudo USERNAME

   • OPTIONALLY: if you just want to allow the user to use docker commands:

     usermod -aG docker USERNAME

2. Ensure you have curl installed

   sudo apt install curl -y

3. Install Docker

   sudo curl -sSL https://get.docker.com/ | sh

   • Docker can be installed solely by your distro's package manage, but running the following command takes care of all dependencies and will ensure you end up with the newest version of docker
4. Verify Docker is functioning correctly.
   • Run the following command to check if docker installed and is functioning properly. It should output with a bunch of text hopefully including: "active (running)" in green. That means docker is working properly

     sudo systemctl status docker

     

5. OPTIONAL: If you intend to cache services other than just steam, like origin, or battle.net/blizzard, add the corresponding amount of IPs necessary.
   • If you just want to cache steam games, you don't need to add any additional IPs - steamcache will run just fine on the main system IP ASSUMING nothing else is already using ports 53 (dns forwarding), and 80 (caching service).
     • Otherwise, you need to determine the amount of additional IPs you will require. Add 1 additional IP per extra service you'd like to cache. For example, if you wish to cache steam and origin, you would need 1 additional IP on top of the existing IP of the system, for a total of 2 IP addresses.
     • For our long term setup we need 6 IPs total, as we will be caching everything EXCEPT for Uplay as that seems to break it for whatever reason.
       • 192.168.1.50 - steamcache-dns and steam-cache docker containers
       • 192.168.1.51 - origin-cache (Origin)
       • 192.168.1.52 - blizzard-cache (Battle.net)
       • 192.168.1.53 - riot-cache (Riot Games)
       • 192.168.1.54 - frontier-cache (Frontier Launchpad)
       • 192.168.1.55 - windows-cache (Windows Updates)
     • Make sure you CHANGE the example IPs to follow your subnet, some people's routers use "192.168.1.o", some are "192.168.0.x", it really depends. You also don't have to use .50-55, you can use any unused local addresses. I'd recommend checking the systems current IP address and going off of that (should be the first IP listed).

       hostname -I

     • MAKE SURE no other devices on your network are already using those IP addresses. Mobile apps like "Fing" can be useful to scan your network to find unused IPs (this can also be used to find out what your subnet is)
   • With your IPs now decided upon, it's time to set them in NetworkManager utility, run:

     sudo nmtui

   • Navigate into "IPv4 Connection">"Wired connection 1" (this might be named something else, so select the entry under "ethernet")
   • Set "IPv4 Configuration" to "Manual"
   • Enter each desired IP, with an "/32" appended to the end to indicate to the system that it is a single IP address range, like shown below
   • Set "Gateway" to the IP address of your router. This is "192.168.1.1" for us.
   • Set "DNS servers" to your desired DNS servers, if you don't know what this is use "1.1.1.1" and "8.8.8.8"
   • Exit out via <OK>, then <Back>, then Quit.
   • Restart networking by running:

     sudo systemctl restart networking

   • Check that the IP addition worked by running "ping ADDITIONALIP" on another computer on the same network. This would be "ping 192.168.1.51" or .52, .53, .54, or .55. If it doesn't respond to pings, try restarting the system. If it still does not work, make sure there are no IP conflicts and that you entered the settings properly.
6. Start the docker containers for each caching service
   • Depending on how many caching services you intend to run, you might have to run up to 8 full length commands to get all of the services running, thankfully you can just copypasta them from here and edit to suit your setup. Only run the caching container commands for services you want to use and then remove the IPs from the dns container command for the services you don't intend to cache.
   • Change the IPs of each module to your decided up IP address(es)
   • Change the 2 instances of "/home/USERNAME/cache/" after "-v" on each module to the location you intend to store the cached data. If the caching server is just a single drive system, the supplied location will work after changing "USERNAME" to the username of the user you are using (lol, get used).
      

     • Steam

       
       sudo docker run \
         --restart unless-stopped -d \
         --name steam-cache \
         -v /home/USERNAME/cache/steam/data:/data/cache \
         -v /home/USERNAME/cache/steam/logs:/data/logs \
         -p 192.168.1.50:80:80 \
         steamcache/generic:latest

     • Origin

       
       sudo docker run \
         --restart unless-stopped -d \
         --name origin-cache \
         -v /home/USERNAME/cache/origin/data:/data/cache \
         -v /home/USERNAME/cache/origin/logs:/data/logs \
         -p 192.168.1.51:80:80 \
         steamcache/generic:latest

     • Blizzard / Battle.net

       
       sudo docker run \
         --restart unless-stopped -d \
         --name blizzard-cache \
         -v /home/USERNAME/cache/blizzard/data:/data/cache \
         -v /home/USERNAME/cache/blizzard/logs:/data/logs \
         -p 192.168.1.52:80:80 \
         steamcache/generic:latest

     • Riot Games

       
       sudo docker run \
         --restart unless-stopped -d \
         --name riot-cache \
         -v /home/USERNAME/cache/riot/data:/data/cache \
         -v /home/USERNAME/cache/riot/logs:/data/logs \
         -p 192.168.1.53:80:80 \
         steamcache/generic:latest

     • Frontier Launchpad

       
       sudo docker run \
         --restart unless-stopped -d \
         --name frontier-cache \
         -v /home/USERNAME/cache/frontier/data:/data/cache \
         -v /home/USERNAME/cache/frontier/logs:/data/logs \
         -p 192.168.1.54:80:80 \
         steamcache/generic:latest

     • Windows

       
       docker run \
         --restart unless-stopped -d \
         --name windows-cache \
         -v /home/USERNAME/cache/windows/data:/data/cache \
         -v /home/USERNAME/cache/windows/logs:/data/logs \
         -p 192.168.1.55:80:80 \
         steamcache/generic:latest

1. Start the steamcache-dns docker container
   • This container takes incoming DNS requests and forwards any content that should be cached to the caching server, and any other requests to their intended location.
   • Remove the lines for services you aren't using, and make sure to change the IPs to the correct IP for each caching container.
   • Change the 192.168.1.50 after "-p" to the main IP of the server

     sudo docker run \
       --restart unless-stopped -d \
       --name steamcache-dns \
       -p 192.168.1.50:53:53/udp \
       -e UPSTREAM_DNS=1.1.1.1 \
       -e STEAMCACHE_IP=192.168.1.50 \
       -e ORIGINCACHE_IP=192.168.1.51 \
       -e BLIZZARDCACHE_IP=192.168.1.52 \
       -e RIOTCACHE_IP=192.168.1.53 \
       -e FRONTIERCACHE_IP=192.168.1.54 \
       -e WINDOWSCACHE_IP=192.168.1.55 \
       steamcache/steamcache-dns:latest

2. Start the sniproxy docker container
   • This container forwards HTTPS requests that the cache cannot read due to encryption straight along to the original destination, without it Origin has a lot of issues.

     docker run \
      --restart unless-stopped -d \
       --name sniproxy \
       -p 443:443 \
       steamcache/sniproxy:latest

3. Check the status of the containers
   • All of the containers should show up as "Running for X"

     sudo docker container ls

4. If everything is working, head over to a client computer and set the DNS
   • Navigation to Settings>Networking & Internet>Change adapter options>Right click your internet adapter>Properties>uncheck IPv6>Click IPv4 and then Properties
   • Check "Use the following DNS server addresses"
   • Set the "Preferred DNS server" to the IP of your steamcache-dns docker container, for us that is "192.168.1.50"
   • Set the "Alternate DNS server" to your DNS provider of choice, if you don't know what this is, use "1.1.1.1"
5. On the client computer open command prompt and run

   ipconfig /flushdns

6. Attempt to download a game!
   • You can monitor usage on the server via an app like "nload". On first download you should see a lot of traffic both incoming and outgoing, but it should pretty much be strictly outgoing when downloading already cached content

 

SUCCESSS!

 

Any Linux guru's out there feel free to comment any additional info you think would be helpful. I'm by no means an expert, but these are the steps we took to get it working.

 

Sources (useful if you're more interested):

https://steamcache.github.io/

https://github.com/steamcache/generic

https://hub.docker.com/r/steamcache/steamcache-dns/

 

Also, if you want to learn how to set this up raw dawg with a full manual install, Arstechnica has a great article on how to do that, along with a bunch more information related to the docker setup. Check it out here: https://arstechnica.com/gaming/2017/01/building-a-local-steam-caching-server-to-ease-the-bandwidth-blues/

 

 

[Tomohare]

Hi,

 

Maybe a docker-compose would save time on does copy paste (not tested ;3) :

version: "3"
services:

  steamcache-dns:
    restart: unless-stopped
    image: steamcache/steamcache-dns:latest
    environment:
      - UPSTREAM_DNS: "1.1.1.1"
      - STEAMCACHE_IP: "192.168.1.50"
      - ORIGINCACHE_IP: "192.168.1.51"
      - BLIZZARDCACHE_IP: "192.168.1.52"
      - RIOTCACHE_IP: "192.168.1.53"
      - FRONTIERCACHE_IP: "192.168.1.54"
      - WINDOWSCACHE_IP: "192.168.1.55"
    ports:
      - "53:53/udp"
    networks:
        vpcbr:
          ipv4_address: 192.168.1.50

  sniproxy:
    restart: unless-stopped
    image: steamcache/sniproxy:latest
    ports:
      - "443:443"

  steam-cache:
    restart: unless-stopped
    image: steamcache/generic:latest
    volumes:
        - cache/steam/data:/data/cache
        - cache/steam/logs:/data/logs
    ports:
      - "80:80"
    networks:
        vpcbr:
          ipv4_address: 192.168.1.50

  origin-cache:
    restart: unless-stopped
    image: steamcache/generic:latest
    volumes:
        - cache/origin/data:/data/cache
        - cache/origin/logs:/data/logs
    ports:
      - "80:80"
    networks:
        vpcbr:
          ipv4_address: 192.168.1.51

  blizzard-cache:
    restart: unless-stopped
    image: steamcache/generic:latest
    volumes:
        - cache/blizzard/data:/data/cache
        - cache/blizzard/logs:/data/logs
    ports:
      - "80:80"
    networks:
        vpcbr:
          ipv4_address: 192.168.1.52

  riot-cache:
    restart: unless-stopped
    image: steamcache/generic:latest
    volumes:
        - cache/riot/data:/data/cache
        - cache/riot/logs:/data/logs
    ports:
      - "80:80"
    networks:
        vpcbr:
          ipv4_address: 192.168.1.53

  fontier-cache:
    restart: unless-stopped
    image: steamcache/generic:latest
    volumes:
        - cache/fontier/data:/data/cache
        - cache/fontier/logs:/data/logs
    ports:
      - "80:80"
    networks:
        vpcbr:
          ipv4_address: 192.168.1.54

  windows-cache:
    restart: unless-stopped
    image: steamcache/generic:latest
    volumes:
        - cache/windows/data:/data/cache
        - cache/windows/logs:/data/logs
    ports:
      - "80:80"
    networks:
        vpcbr:
          ipv4_address: 192.168.1.55

 

> docker-compose up -d

[BrianAnim]

Hey Jake, I made this tutorial awhile back for people who might want a good video tutorial on the steam portion:

 

[Larklen]

Wonder if anyone else is planning to use this on a Windows machine. I have one of my home servers set up using Windows 10 Pro and have taken a quick stab at getting this setup to no avail - doesn't seem to be too much in the way of guides/other people setting it up so I suppose I'll have to fiddle around with it. If anyone else has experience, I'd love to poke a brain or two!

[deanrodger7]

I tried setting something like this up awhile ago, but couldn't get it to work properly. So I left it, until a new feature in Windows came out. The Windows Subsystem for Linux. So I tried the steps listed on the ArsTechnica website using the DIY method. And to my surprise it worked. After testing it some more. I then created a simple script to install bind9 and nginx, plus setting up all the need files. The scrtipt only works with networks with 24 bit subnet mask. You might want to test the performance difference between Linux and Windows. I attached the script for you to look at. Although I've only got Steam working.

steamcache.sh

[deanrodger7]

8 minutes ago, Larklen said:

Wonder if anyone else is planning to use this on a Windows machine. I have one of my home servers set up using Windows 10 Pro and have taken a quick stab at getting this setup to no avail - doesn't seem to be too much in the way of guides/other people setting it up so I suppose I'll have to fiddle around with it. If anyone else has experience, I'd love to poke a brain or two!

I've got this working using the Windows Subsystem for Linux and using native windows exe of nginx and bind9. If you need any help setting this up, I'll be glad to help.

[Cornelicorn]

10 minutes ago, Larklen said:

Wonder if anyone else is planning to use this on a Windows machine. I have one of my home servers set up using Windows 10 Pro and have taken a quick stab at getting this setup to no avail - doesn't seem to be too much in the way of guides/other people setting it up so I suppose I'll have to fiddle around with it. If anyone else has experience, I'd love to poke a brain or two!

It should not be very difficult: Install docker and check if its working using https://docs.docker.com/docker-for-windows/#shared-drives then just edit the commands 

docker run \
  --restart unless-stopped -d \
  --name steam-cache \
  -v {Path of Steamcache-cache-folder on the host drive}:/data/cache \
  -v {Path of Steamcache-log-folder on the host drive}:/data/logs \
  -p {Main IP for Steam Cache}:80:80 \
  steamcache/generic:latest

 

Adjust the other commands accordingly, you can configure multiple IPs by going into the Control Panel -- Network and Internet -- Network and Sharing Center -- Change adapter settings -- right click on your Network Interface and click on Properties -- select Internet Protocol Version 4 and choose use the following IP address -- enter your main IP of your system -- enter your preferred DNS-Servers -- click on andvanced to configure multiple IPs: just click add to add another one, if you don't know what to enter as Subnet mask just enter 255.255.255.0. As you said this is one of your Home Servers let the Steam Cache run on a different IP if you are already hosting another DNS Server or Webserver.

 

Additional note to the Path for the folders: You probably have to configure them in the docker application as a shared folder, but this should work in the newest version if your User has full access to the directories

[Larklen]

Hey guys! Appreciate the responses! I've reviewed the Ars Technica page as well as re-read the instructions listed above by jakkuh_t as well as you Cornelicorn. As far as I can tell (besides the fact it isn't working as intended), I am following the steps properly - I believe my issue listed in the 'steamcache/steamcache-dns:latest: image which when I run docker ps - it shows the 'STATUS' constantly as 'Restarting (1) X second(s) ago' - so with this image constantly restarting, it'll make sense why I can't resolve anything on my local PC.

 

Here is the 'docker ps' result:

 

CONTAINER ID        IMAGE                              COMMAND                  CREATED             STATUS                        PORTS                     NAMES
9d481bae8265        steamcache/steamcache-dns:latest   "bash /scripts/boots…"   14 seconds ago      Restarting (1) 1 second ago                             steamcache-dns
107f95346619        steamcache/steamcache:latest       "/scripts/bootstrap.…"   About an hour ago   Up About an hour              0.0.0.0:80->80/tcp        steamcache
f80025f0c99c        steamcache/sniproxy:latest         "sniproxy -f"            About an hour ago   Up About an hour              0.0.0.0:443->443/tcp      sniproxy
da9d20fd9ae5        steamcache/generic:latest          "/scripts/bootstrap.…"   2 hours ago         Up 2 hours                    192.168.1.14:80->80/tcp   windows-cache
faf6db2fab9a        steamcache/generic:latest          "/scripts/bootstrap.…"   2 hours ago         Up 2 hours                    192.168.1.13:80->80/tcp   frontier-cache
e968dae6364a        steamcache/generic:latest          "/scripts/bootstrap.…"   2 hours ago         Up 2 hours                    192.168.1.12:80->80/tcp   blizzard-cache
59c463a13cd6        steamcache/generic:latest          "/scripts/bootstrap.…"   2 hours ago         Up 2 hours                    192.168.1.11:80->80/tcp   steam-cache

This is the command run to start the image - note: The internal IP Address of this PC is 192.168.10, I have tried this command with both that and 0.0.0.0 (just to test).

 

docker run  --restart unless-stopped -d  --name steamcache-dns  -p 192.168.1.10:53:53/udp  -e UPSTREAM_DNS=1.1.1.1  -e STEAMCACHE_IP=192.168.1.11  -e BLIZZARDCACHE_IP=192.168.1.12 -e FRONTIERCACHE_IP=192.168.1.13 -e WINDOWSCACHE_IP=192.168.1.14 steamcache/steamcache-dns:latest

 

I have confirmed (as best as I can without it working) that the drives are being shared with Docker. The IP range of 192.168.1.10 --> 192.168.1.14 can be reached by ping so they have been set up, as well as I have disabled IPv6 on both computers.

 

Here is the ipconfig result if it'll help any:

 

Ethernet adapter Ethernet:

   Connection-specific DNS Suffix  . :
   IPv4 Address. . . . . . . . . . . : 192.168.1.10
   Subnet Mask . . . . . . . . . . . : 255.255.255.0
   IPv4 Address. . . . . . . . . . . : 192.168.1.11
   Subnet Mask . . . . . . . . . . . : 255.255.255.0
   IPv4 Address. . . . . . . . . . . : 192.168.1.12
   Subnet Mask . . . . . . . . . . . : 255.255.255.0
   IPv4 Address. . . . . . . . . . . : 192.168.1.13
   Subnet Mask . . . . . . . . . . . : 255.255.255.0
   IPv4 Address. . . . . . . . . . . : 192.168.1.14
   Subnet Mask . . . . . . . . . . . : 255.255.255.0
   Default Gateway . . . . . . . . . : 192.168.1.254

 

 

I'm still digging and trying stuff on my end in hopes I can resolve this myself, but there are always smarter people out there than I and based on what I can see, the DNS image is the core issue.

[Cornelicorn]

6 minutes ago, Larklen said:

Hey guys! Appreciate the responses! I've reviewed the Ars Technica page as well as re-read the instructions listed above by jakkuh_t as well as you Cornelicorn. As far as I can tell (besides the fact it isn't working as intended), I am following the steps properly - I believe my issue listed in the 'steamcache/steamcache-dns:latest: image which when I run docker ps - it shows the 'STATUS' constantly as 'Restarting (1) X second(s) ago' - so with this image constantly restarting, it'll make sense why I can't resolve anything on my local PC.

 

Here is the 'docker ps' result:

 

CONTAINER ID        IMAGE                              COMMAND                  CREATED             STATUS                        PORTS                     NAMES
9d481bae8265        steamcache/steamcache-dns:latest   "bash /scripts/boots…"   14 seconds ago      Restarting (1) 1 second ago                             steamcache-dns
107f95346619        steamcache/steamcache:latest       "/scripts/bootstrap.…"   About an hour ago   Up About an hour              0.0.0.0:80->80/tcp        steamcache
f80025f0c99c        steamcache/sniproxy:latest         "sniproxy -f"            About an hour ago   Up About an hour              0.0.0.0:443->443/tcp      sniproxy
da9d20fd9ae5        steamcache/generic:latest          "/scripts/bootstrap.…"   2 hours ago         Up 2 hours                    192.168.1.14:80->80/tcp   windows-cache
faf6db2fab9a        steamcache/generic:latest          "/scripts/bootstrap.…"   2 hours ago         Up 2 hours                    192.168.1.13:80->80/tcp   frontier-cache
e968dae6364a        steamcache/generic:latest          "/scripts/bootstrap.…"   2 hours ago         Up 2 hours                    192.168.1.12:80->80/tcp   blizzard-cache
59c463a13cd6        steamcache/generic:latest          "/scripts/bootstrap.…"   2 hours ago         Up 2 hours                    192.168.1.11:80->80/tcp   steam-cache

This is the command run to start the image - note: The internal IP Address of this PC is 192.168.10, I have tried this command with both that and 0.0.0.0 (just to test).

 

docker run  --restart unless-stopped -d  --name steamcache-dns  -p 192.168.1.10:53:53/udp  -e UPSTREAM_DNS=1.1.1.1  -e STEAMCACHE_IP=192.168.1.11  -e BLIZZARDCACHE_IP=192.168.1.12 -e FRONTIERCACHE_IP=192.168.1.13 -e WINDOWSCACHE_IP=192.168.1.14 steamcache/steamcache-dns:latest

 

I have confirmed (as best as I can without it working) that the drives are being shared with Docker. The IP range of 192.168.1.10 --> 192.168.1.14 can be reached by ping so they have been set up, as well as I have disabled IPv6 on both computers.

 

Here is the ipconfig result if it'll help any:

 

Ethernet adapter Ethernet:

   Connection-specific DNS Suffix  . :
   IPv4 Address. . . . . . . . . . . : 192.168.1.10
   Subnet Mask . . . . . . . . . . . : 255.255.255.0
   IPv4 Address. . . . . . . . . . . : 192.168.1.11
   Subnet Mask . . . . . . . . . . . : 255.255.255.0
   IPv4 Address. . . . . . . . . . . : 192.168.1.12
   Subnet Mask . . . . . . . . . . . : 255.255.255.0
   IPv4 Address. . . . . . . . . . . : 192.168.1.13
   Subnet Mask . . . . . . . . . . . : 255.255.255.0
   IPv4 Address. . . . . . . . . . . : 192.168.1.14
   Subnet Mask . . . . . . . . . . . : 255.255.255.0
   Default Gateway . . . . . . . . . : 192.168.1.254

 

 

I'm still digging and trying stuff on my end in hopes I can resolve this myself, but there are always smarter people out there than I and based on what I can see, the DNS image is the core issue.

I know this may sound like the typical questions for less tech-savy people, but: did you check docker with hello world and seriously: did you try turning it off and on again? and at last even if it is reporting as being restarted, just try to lookup any dns from that server by running "ipconfig/flushdns" and then "nslookup google.de 192.168.1.10". And i would stick using the local IPs and not 0.0.0.0 or at least use 127.0.0.1.

 

Oh and by the way what does "docker --version" say?

[Cornelicorn]

1 minute ago, Cornelicorn said:

I know this may sound like the typical questions for less tech-savy people, but: did you check docker with hello world and seriously: did you try turning it off and on again? and at last even if it is reporting as being restarted, just try to lookup any dns from that server by running "ipconfig/flushdns" and then "nslookup google.de 192.168.1.10". And i would stick using the local IPs and not 0.0.0.0 or at least use 127.0.0.1.

 

Oh and by the way what does "docker --version" say?

And it would help if you would post the command you used to start the steamcache-dns

[Larklen]

12 minutes ago, Cornelicorn said:

I know this may sound like the typical questions for less tech-savy people, but: did you check docker with hello world and seriously: did you try turning it off and on again? and at last even if it is reporting as being restarted, just try to lookup any dns from that server by running "ipconfig/flushdns" and then "nslookup google.de 192.168.1.10". And i would stick using the local IPs and not 0.0.0.0 or at least use 127.0.0.1.

 

Oh and by the way what does "docker --version" say?

 

And it would help if you would post the command you used to start the steamcache-dns

Hey!

 

Yup, the hello world worked and I did the whole off/on earlier as well as again before I wrote this (just to be certain. Definitely a good question, since that solves ~90% of issues  )

 

Version result: Docker version 18.06.0-ce, build 0ffa825

 

This is the result from running nslookup on the machine with docker.

>nslookup google.com 192.168.1.10
DNS request timed out.
    timeout was 2 seconds.
Server:  UnKnown
Address:  192.168.1.10

DNS request timed out.
    timeout was 2 seconds.
DNS request timed out.
    timeout was 2 seconds.
DNS request timed out.
    timeout was 2 seconds.
DNS request timed out.
    timeout was 2 seconds.
*** Request to UnKnown timed-out

 

And this is the command I used to start the steamcache-dns image:

docker run  --restart unless-stopped -d  --name steamcache-dns  -p 192.168.1.10:53:53/udp  -e UPSTREAM_DNS=1.1.1.1  -e STEAMCACHE_IP=192.168.1.11  -e BLIZZARDCACHE_IP=192.168.1.12 -e FRONTIERCACHE_IP=192.168.1.13 -e WINDOWSCACHE_IP=192.168.1.14 steamcache/steamcache-dns:latest

Greatly appreciate your assistance here  I'm willing to best the problem lies with me, my knowledge and past experience with DNS is akin to essentially nothing.

[Cornelicorn]

11 minutes ago, Larklen said:

Hey!

 

Yup, the hello world worked and I did the whole off/on earlier as well as again before I wrote this (just to be certain. Definitely a good question, since that solves ~90% of issues  )

 

Version result: Docker version 18.06.0-ce, build 0ffa825

 

This is the result from running nslookup on the machine with docker.

>nslookup google.com 192.168.1.10
DNS request timed out.
    timeout was 2 seconds.
Server:  UnKnown
Address:  192.168.1.10

DNS request timed out.
    timeout was 2 seconds.
DNS request timed out.
    timeout was 2 seconds.
DNS request timed out.
    timeout was 2 seconds.
DNS request timed out.
    timeout was 2 seconds.
*** Request to UnKnown timed-out

 

And this is the command I used to start the steamcache-dns image:

docker run  --restart unless-stopped -d  --name steamcache-dns  -p 192.168.1.10:53:53/udp  -e UPSTREAM_DNS=1.1.1.1  -e STEAMCACHE_IP=192.168.1.11  -e BLIZZARDCACHE_IP=192.168.1.12 -e FRONTIERCACHE_IP=192.168.1.13 -e WINDOWSCACHE_IP=192.168.1.14 steamcache/steamcache-dns:latest

Greatly appreciate your assistance here  I'm willing to best the problem lies with me, my knowledge and past experience with DNS is akin to essentially nothing.

Lets get this command a bit smaller:

docker run  --restart unless-stopped -d  --name steamcache-dns  -p 192.168.1.10:53:53/udp  -e UPSTREAM_DNS=1.1.1.1  -e STEAMCACHE_IP=192.168.1.11  -e BLIZZARDCACHE_IP=192.168.1.12 -e FRONTIERCACHE_IP=192.168.1.13 -e WINDOWSCACHE_IP=192.168.1.14 steamcache/steamcache-dns:latest

goes to

 

docker run -f --name steamcache-dns -p 192.168.1.10:53:53/udp -e UPSTREAM_DNS=1.1.1.1 -e STEAMCACHE_IP=192.168.1.11 steamcache/steamcache-dns:latest

 

This should run the container in the foreground, thereby enabling console output. And we just remove the other services for now. Have you checked if you opened port 53 on your firewall?

Edited August 21, 2018 by Cornelicorn
Port 5 typo instead of Port 53

[Larklen]

1 hour ago, Cornelicorn said:

Lets get this command a bit smaller:

docker run  --restart unless-stopped -d  --name steamcache-dns  -p 192.168.1.10:53:53/udp  -e UPSTREAM_DNS=1.1.1.1  -e STEAMCACHE_IP=192.168.1.11  -e BLIZZARDCACHE_IP=192.168.1.12 -e FRONTIERCACHE_IP=192.168.1.13 -e WINDOWSCACHE_IP=192.168.1.14 steamcache/steamcache-dns:latest

goes to

 

docker run -f --name steamcache-dns -p 192.168.1.10:53:53/udp -e UPSTREAM_DNS=1.1.1.1 -e STEAMCACHE_IP=192.168.1.11 steamcache/steamcache-dns:latest

 

This should run the container in the foreground, thereby enabling console output. And we just remove the other services for now. Have you checked if you opened port 53 on your firewall?

I had to remove the '-f' as it returned an error, but it seems that command worked! I got a response that I never did before. Curious to why the last command wasn't working, but this is a good step in the right direction! Not to get this working for Blizzard/Windows cache as well hah

 

Edit: Got it for most all the caches, only one that gave an error was trying to create a windows cache, but I'm not overly concerned about that one. This was the final command I ran:

docker run --name steamcache-dns -p 192.168.1.10:53:53/udp -e UPSTREAM_DNS=1.1.1.1 -e STEAMCACHE_IP=192.168.1.11 -e BLIZZARDCACHE_IP=192.168.1.12 -e FRONTIERCACHE_IP=192.168.1.13 steamcache/steamcache-dns:latest

 

Edited August 21, 2018 by Larklen
Added edit

[BrianAnim]

We use Hyper-V inside windows 10 to do ours.

[Cornelicorn]

6 hours ago, Larklen said:

I had to remove the '-f' as it returned an error, but it seems that command worked! I got a response that I never did before. Curious to why the last command wasn't working, but this is a good step in the right direction! Not to get this working for Blizzard/Windows cache as well hah

 

Edit: Got it for most all the caches, only one that gave an error was trying to create a windows cache, but I'm not overly concerned about that one. This was the final command I ran:

docker run --name steamcache-dns -p 192.168.1.10:53:53/udp -e UPSTREAM_DNS=1.1.1.1 -e STEAMCACHE_IP=192.168.1.11 -e BLIZZARDCACHE_IP=192.168.1.12 -e FRONTIERCACHE_IP=192.168.1.13 steamcache/steamcache-dns:latest

 

Then try to add 

--restart unless-stopped -d

this makes sure it gets restarted again if the container fails for any reason. And it seams they removed support for Windows Updates as there is no mention except on the dnsmasq branch on their GitHub Page which probably is outdated since it wasn't edited 7 days ago but a year ago.

Glad you got it working

[sirikon]

Just saw the video and was wondering... The fact that the caching server works just like this, without any advanced configuration on the end machines (using Steam to download the games), isn't itself a security problem?

This seems to work because the requests made to download steam games are using HTTP, not HTTPS. This means that anyone could make a MitM attack and replace the game we're downloading with some malicious software.

Or, as the DNS configuration was made on the entire network/computer, any information downloaded using http could be affected.

The difference here is that we're just trusting the server we just built... but anyone could do this in a Lan Party or Cybercafe without us noticing at all and doing some bad stuff.

I think that Steam and many other providers should enforce HTTPS always, including game (software) downloads. This way, this server cache wouldn't work without some configuration on the machines using Steam (installing a trusted certificate provided by the Lan Party host) but the security overall would improve.

[Abromberek]

i keep getting error's trying to do this

 

alex@Caching01W:~$ sudo docker run \
> 
"docker run" requires at least 1 argument.
See 'docker run --help'.

Usage:  docker run [OPTIONS] IMAGE [COMMAND] [ARG...]

Run a command in a new container
alex@Caching01W:~$   --restart unless-stopped -d \
> 
--restart: command not found
alex@Caching01W:~$   --name steam-cache \
> 
--name: command not found
alex@Caching01W:~$   -v /home/alex/cache/steam/data:/data/cache \
> 
-v: command not found
alex@Caching01W:~$   -v /home/alex/cache/steam/logs:/data/logs \
> 
-v: command not found
alex@Caching01W:~$   -p 10.200.17.1:80:80 \
> 
-p: command not found
 

[Larklen]

@Cornelicorn I'll throw that in there when I get home! I'm willing to bet the inclusion of the windows-cache in the original statement is where it was screwing up, since its no longer supported.

 

@Abromberek cut the command into a single line and try it again  Using the example listed in OP, it'd be something like this:

 

sudo docker run --restart unless-stopped -d --name steam-cache -v /home/USERNAME/cache/steam/data:/data/cache -v /home/USERNAME/cache/steam/logs:/data/logs -p 192.168.1.50:80:80 steamcache/generic:latest

 

[Androidz]

I have tried this on three VM's with no success. I first started with CentOS as that is my flavor, no mas. So then I decided to try exactly what the tutorial says and used 18.04 desktop, that also failed. I raged quit and moved on to today.....

Today I decided to try again as I thought I may have missed something in the tutorial which may be incorrect. It says you require an IP for each cache service, but if only caching one service you can "skip". So I believe what the tutorial means is that you need an IP (separate from host) for each cache, regardless of using one cache service or multiple. Is this correct?

So I setup a new 18.04 LTS host with multiple IP's and again I run into the same issue and that is steamcache-dns constantly restarting. I believe the issue is related to port 53, so I tried to change the port however steamcache-dns always wants port 53.

I use DNS Resolver in pfsense which uses port 53 so I believe that is where the problem is, however I haven't tried altering pfsense at this time.

pfsense:
The port used for responding to DNS queries. It should normally be left blank unless another service needs to bind to TCP/UDP port 53.

Docker:
sudo docker run \
> --restart unless-stopped -d \
> --name steamcache-dns \
> -p 192.168.1.26:55:55/udp \ <- This is where I tried binding to port 55 instead, however no matter what it wants to use port 53
> -e UPSTREAM_DNS=192.168.1.1 \
> -e WINDOWSCACHE_IP=192.168.1.27 \
> steamcache/steamcache-dns:latest

sudo docker container ls
CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES
8ec9096f75da steamcache/steamcache-dns:latest "bash /scripts/boots…" 3 seconds ago Up 1 second 192.168.1.26:53->53/udp steamcache-dns
1f4d9897f9d8 steamcache/sniproxy:latest "sniproxy -f" 21 minutes ago Up 18 minutes 0.0.0.0:443->443/tcp sniproxy
58ee1b56c65a steamcache/generic:latest "/scripts/bootstrap.…" 27 minutes ago Up 18 minutes 192.168.1.27:80->80/tcp windows-cache

5 seconds later....
8ec9096f75da steamcache/steamcache-dns:latest "bash /scripts/boots…" 6 seconds ago Restarting (1) Less than a second ago steamcache-dns


Questions:
1. Is there anyway to bind steamcache-dns to something other than 53?
2. Is opening port 53 required on firewall? This is not mentioned in the tutorial.
3a. Anyone with pfsense using DNS Resolver (53) try running this?
3b. Did you switch the pfsense DNS Resolve port?

[Larklen]

@Androidz Try removing the portion that is enabling a windows-cache, in my attempts we found out that windows-cache is no longer supported and when I removed that, it began to work as expected - I would check Cornelicorn's last post, he has my quote in it with some info as well, it shows the command I used to in the end to have it functioning. 

 

Hopefully this helps, I don't have direct experience with pfsense or any linux distro's so if the issue persists, hopefully someone with more knowledge than I can help!

[Unieveth]

I have been working on this for 2 hours now and I'm stuck now.  Every time I get to the start the steamcache-dns docker container part it gets stuck restarting itself.  I've dug through the logs and it shows this: 

     _                                      _                       _
    | |                                    | |                     | |
 ___| |_ ___  __ _ _ __ ___   ___ __ _  ___| |__   ___   _ __   ___| |_
/ __| __/ _ \/ _` | '_ ` _ \ / __/ _` |/ __| '_ \ / _ \ | '_ \ / _ \ __|
\__ \ ||  __/ (_| | | | | | | (_| (_| | (__| | | |  __/_| | | |  __/ |_
|___/\__\___|\__,_|_| |_| |_|\___\__,_|\___|_| |_|\___(_)_| |_|\___|\__|


Bootstrapping DNS from https://github.com/uklans/cache-domains
Creating apple template
Creating arenanet template
Creating blizzard template
Creating daybreak template
Creating frontier template
Creating gog template
Creating hirez template
Creating minecraft template
Creating nexusmods template
Creating nintendo template
Creating origin template
Creating riot template
Creating rockstar template
Creating sony template
Creating steam template
Creating uplay template
Creating twitch template
Creating wargaming template
Creating wsus template
Creating xboxlive template

 ---

Enabling WINDOWS on IP 192.168.86.82
creating /etc/bind/cache/windows.db
sed: /etc/bind/cache/windows.db: No such file or directory

It always crashes on the enabling windows part, since it can't seem to find a windows.db file.  I have attached my text file that I have been using to edit the commands for my server and tracking errors and other solutions I have found on here.  I am using a Linux server with Ubuntu 18.04

Cache Setup.txt

Edited August 21, 2018 by Unieveth
Added server info

[Androidz]

@Larklen - Thank you for the reply. You are correct, without Windows cache steamcache-dns works fine. Unfortunately my goal was caching windows updates and nothing else.

@Unieveth - See Larklen's reply. Looks like Windows cache is broken. Perhaps someone can find a fix?

[Unieveth]

I have it working to a point where the DNS is not crashing, but now the issue is that I see no traffic going through the server when I'm downloading a game from steam.  It still seems to be bypassing the server.  I have changed the DNS settings on my local PC to test it and it's not going through the server.  Here is an image of the settings along with putty with nload running: https://i.imgur.com/mcqwyOA.png I have also flushed the DNS on my system so unless I'm missing something it should be working.  I have also checked the stream cache and it has 1 item in it: https://i.imgur.com/zIhwXQq.png

 

Fixed it

Edited August 24, 2018 by Unieveth
Fixed the issue

[Androidz]

I have steam cache running but its not actually caching. Not going to spend anymore time on it since my main goal was Windows Updates. If you end up getting that working please let me know.

Thank you

[Jelsie]

Has anyone got this working with Windows Update?