Jump to content

Steam Caching Tutorial

jakkuh_t

I've followed the tutorial but i can't get it working. The steamcache-dns container keeps restarting for some reason. 
 

sudo docker ps
CONTAINER ID        IMAGE                              COMMAND                  CREATED             STATUS              PORTS                       NAMES
7de874b45916        steamcache/steamcache-dns:latest   "bash /scripts/boots…"   32 minutes ago      Up 7 seconds        192.168.10.107:53->53/udp   steamcache-dns
daa65b6a84b9        steamcache/generic:latest          "/scripts/bootstrap.…"   33 minutes ago      Up 33 minutes       192.168.10.151:80->80/tcp   windows-cacheworks
1a5f93dd5da6        steamcache/generic:latest          "/scripts/bootstrap.…"   33 minutes ago      Up 33 minutes       192.168.10.150:80->80/tcp   steam-cacheworks2

Help would be appreciated, thanks in advance

Link to comment
Share on other sites

Link to post
Share on other sites

Not sure what's going on with mine, it's caching and working as it should - except download performance is total butt. I just can't figure out where it's being held up, I'm suspecting a CPU bottleneck but I'm not sure why. It's been assigned two cores of a dual L5639 system - increasing core count doesn't improve the performance.

Download speed without cache server (maxes out internet connection):

dlwhithout.png.68e51a61247efa0981727864b00ebb21.png

Download speed with cache server (before game has been cached, it is creating the cache):

dlwith.png.51f5a422c4991fb003a4550e558e8fbb.png

Disk speed available to the cache server:

diskspeed.png.71255f73c619d1ee059a48559650efbc.png

CPU usage of the cache server during download (creating cache) - they stay at low percentage:

cpu.png.3a6d17ee29adb74346dda250ff3254aa.png

CPU usage as reported by ESX - the server is virtualized and on a 10G network:

esxcpu.png.c6ce7db52188b1997828ce783f07e738.png

Really odd flatline happening which is indicative of a bottleneck. But on the OS of the caching server it reports low CPU usage. I'm unfamiliar with Docker so I wonder if there's some kind of tweak that needs to be applied but maybe someone else has some input?

 

 

 

Link to comment
Share on other sites

Link to post
Share on other sites

24 minutes ago, dolledollson said:

I've followed the tutorial but i can't get it working. The steamcache-dns container keeps restarting for some reason. 
 


sudo docker ps
CONTAINER ID        IMAGE                              COMMAND                  CREATED             STATUS              PORTS                       NAMES
7de874b45916        steamcache/steamcache-dns:latest   "bash /scripts/boots…"   32 minutes ago      Up 7 seconds        192.168.10.107:53->53/udp   steamcache-dns
daa65b6a84b9        steamcache/generic:latest          "/scripts/bootstrap.…"   33 minutes ago      Up 33 minutes       192.168.10.151:80->80/tcp   windows-cacheworks
1a5f93dd5da6        steamcache/generic:latest          "/scripts/bootstrap.…"   33 minutes ago      Up 33 minutes       192.168.10.150:80->80/tcp   steam-cacheworks2

Help would be appreciated, thanks in advance

Check the logs. Can't do much without them.

docker logs steamcache-dns

 

Link to comment
Share on other sites

Link to post
Share on other sites

9 hours ago, dolledollson said:

I've followed the tutorial but i can't get it working. The steamcache-dns container keeps restarting for some reason. 
 


sudo docker ps
CONTAINER ID        IMAGE                              COMMAND                  CREATED             STATUS              PORTS                       NAMES
7de874b45916        steamcache/steamcache-dns:latest   "bash /scripts/boots…"   32 minutes ago      Up 7 seconds        192.168.10.107:53->53/udp   steamcache-dns
daa65b6a84b9        steamcache/generic:latest          "/scripts/bootstrap.…"   33 minutes ago      Up 33 minutes       192.168.10.151:80->80/tcp   windows-cacheworks
1a5f93dd5da6        steamcache/generic:latest          "/scripts/bootstrap.…"   33 minutes ago      Up 33 minutes       192.168.10.150:80->80/tcp   steam-cacheworks2

Help would be appreciated, thanks in advance

 The Problem is the Windows Cache, seems like they removed support or bugged it in one of the many releases they did a week ago. You can read the first comments on this post for further information.

Link to comment
Share on other sites

Link to post
Share on other sites

Hello All,

 

In response to the issue with the windows update cache that appears to be bricked.

 

TL:DR; Replace  -e WINDOWSCACHE_IP=192.168.1.55 \  with -e WSUSCACHE_IP=192.168.1.55 \  @jakkuh_t

 

 

Now for the long explanation:

 

When running and debugging the steamcache-dns image I noticed that zoning files were created based on profiles that are fetched from this url: https://raw.githubusercontent.com/uklans/cache-domains/master/cache_domains.json

 

In the bootscript of the docker image it will take all parameters ending with CACHE_IP and use the prefix of this string to match it to an existing profile in the URL above.

This also means that currently there are a lot more services available that could be cached then just the ones mentioned in the tutorial.

 

So to build further upon the example of the video.

 

This

sudo docker run \
  --restart unless-stopped -d \
  --name steamcache-dns \
  -p 192.168.1.50:53:53/udp \
  -e UPSTREAM_DNS=1.1.1.1 \
  -e STEAMCACHE_IP=192.168.1.50 \
  -e ORIGINCACHE_IP=192.168.1.51 \
  -e BLIZZARDCACHE_IP=192.168.1.52 \
  -e RIOTCACHE_IP=192.168.1.53 \
  -e FRONTIERCACHE_IP=192.168.1.54 \
  -e WINDOWSCACHE_IP=192.168.1.55 \
  steamcache/steamcache-dns:latest

 

becomes

 

sudo docker run \
  --restart unless-stopped -d \
  --name steamcache-dns \
  -p 192.168.1.50:53:53/udp \
  -e UPSTREAM_DNS=1.1.1.1 \
  -e STEAMCACHE_IP=192.168.1.50 \
  -e ORIGINCACHE_IP=192.168.1.51 \
  -e BLIZZARDCACHE_IP=192.168.1.52 \
  -e RIOTCACHE_IP=192.168.1.53 \
  -e FRONTIERCACHE_IP=192.168.1.54 \
  -e WSUSCACHE_IP=192.168.1.55 \
  steamcache/steamcache-dns:latest

I hope this helps :)

 

Have a nice day!

 

By the way, when checking up on the history of the template file I noticed that there never was a "WINDOWS" template to begin with. Maybe an other version was used in the video. (https://github.com/uklans/cache-domains/commits/master/cache_domains.json)

 

 

image.png.94d67cf0c49162989e89fff338f3924e.png

Link to comment
Share on other sites

Link to post
Share on other sites

How does this work? Does it even work on slow internet. Do i need a seperate computer to set it up? Lol new here so how do you set up linux

Link to comment
Share on other sites

Link to post
Share on other sites

20 hours ago, Jelsie said:

Has anyone got this working with Windows Update?

This would be called a WSUS server. Windows update is a different animal because of the differences in hardware and OS versions inside a network.

Link to comment
Share on other sites

Link to post
Share on other sites

38 minutes ago, grimmjoww578 said:

How does this work? Does it even work on slow internet. Do i need a seperate computer to set it up? Lol new here so how do you set up linux

First you WILL need a machine dedicated to being the cache server. The only real advantage it has is in the bandwidth saving for most people. It just means that instead of downloading and item multiple times that after the first it is cached. The second benefit is that on subsequent installs you would be limited to your LAN performance and not that of your ISP.

Link to comment
Share on other sites

Link to post
Share on other sites

On 21.8.2018 at 11:30 PM, Unieveth said:

I have it working to a point where the DNS is not crashing, but now the issue is that I see no traffic going through the server when I'm downloading a game from steam.  It still seems to be bypassing the server.  I have changed the DNS settings on my local PC to test it and it's not going through the server.  Here is an image of the settings along with putty with nload running: https://i.imgur.com/mcqwyOA.png I have also flushed the DNS on my system so unless I'm missing something it should be working.  I have also checked the stream cache and it has 1 item in it: https://i.imgur.com/zIhwXQq.png

I've got the same problem but i have zero files in the cache. I got it to crashing by fixing the Windows Update thing but it won't cache steam games at all.

Link to comment
Share on other sites

Link to post
Share on other sites

UnRAID (2 Gamers, 1 CPU) users can use the single image docker container 'SteamCacheBundle" ( mlebjerg/steamcachebundle:latest ).

Link to comment
Share on other sites

Link to post
Share on other sites

Curious to know if anyone has actually got this to to work 100%, my cached downloads work great but non-cached downloads are still no good, usually topping around 10MB/s but eventually slowing down to under 1MB/s within minutes. I've tried Ubuntu 18.04, 16.04, Centos 7-1804, tried changing the storage from mdadm to btrfs, tried changing the docker network driver from bridge to macvlan. No improvement, on the cache-build download the cpu/disk/memory utilization is all low. Someone had similar issues - second post here: (https://github.com/steamcache/steamcache/issues/15) I don't doubt someone has got this working but with low hardware utilization across the board I suspect I'm running into a software or docker bug.

Link to comment
Share on other sites

Link to post
Share on other sites

8 hours ago, Emtu said:

Curious to know if anyone has actually got this to to work 100%, my cached downloads work great but non-cached downloads are still no good, usually topping around 10MB/s but eventually slowing down to under 1MB/s within minutes. I've tried Ubuntu 18.04, 16.04, Centos 7-1804, tried changing the storage from mdadm to btrfs, tried changing the docker network driver from bridge to macvlan. No improvement, on the cache-build download the cpu/disk/memory utilization is all low. Someone had similar issues - second post here: (https://github.com/steamcache/steamcache/issues/15) I don't doubt someone has got this working but with low hardware utilization across the board I suspect I'm running into a software or docker bug.

Not sure if this could be the problem but I had issues when setting up the single image version in UnRAID. The forwarding DNS didn't seem to like anything other than 1.1.1.1 ...what made mine more problematic is the fact my virtual machines are all using a Windows Domain Controller... after initially making a mistake typing in the docker IP addresses while enjoying a cold beer, I had problems getting anything in the cache and then getting really slow speeds... so... not sure if it helps but make sure your Windows computer can resolve DNS

#Power Shell (Windows):
Resolve-DnsName steamcache.cs.steampowered.com -server X.X.X.X
#Also
Resolve-DnsName google.com -server X.X.X.X
#Replacing X.X.X.X with what you set as your primary DNS server

Edit: Although related to the UnRAID docker container, this post may help others: https://squishedmooo.com/the-new-and-easier-all-in-one-steamcache/

Link to comment
Share on other sites

Link to post
Share on other sites

2 hours ago, Deadpan110 said:

Not sure if this could be the problem but I had issues when setting up the single image version in UnRAID. The forwarding DNS didn't seem to like anything other than 1.1.1.1 ...what made mine more problematic is the fact my virtual machines are all using a Windows Domain Controller... after initially making a mistake typing in the docker IP addresses while enjoying a cold beer, I had problems getting anything in the cache and then getting really slow speeds... so... not sure if it helps but make sure your Windows computer can resolve DNS


#Power Shell (Windows):
Resolve-DnsName steamcache.cs.steampowered.com -server X.X.X.X
#Also
Resolve-DnsName google.com -server X.X.X.X
#Replacing X.X.X.X with what you set as your primary DNS server

Edit: Although related to the UnRAID docker container, this post may help others: https://squishedmooo.com/the-new-and-easier-all-in-one-steamcache/

I am trying to use the steamcachebundle with my QNAP NAS but the resolve DNS command just returns an error. So I assume something went wrong. 
How can I find out what exactly went wrong ? 

I think the issue is that it is portforwarding see image here. So right now the IP of my NAS is 192.168.1.200
 

123qweqweqweqwe.PNG

Link to comment
Share on other sites

Link to post
Share on other sites

This is UnRAID Docker... I am using a privileged bridge to provide the ports... I have not used your setup (unfortunately ...however I am curious) ...pics below...

Capture.PNG

Link to comment
Share on other sites

Link to post
Share on other sites

24 minutes ago, Deadpan110 said:

This is UnRAID Docker... I am using a privileged bridge to provide the ports... I have not used your setup (unfortunately ...however I am curious) ...pics below...

Capture.PNG

It looks quite similar I mean I can set a custom IP but for me on QNAP there is no field where I can set the LANCACHE_IP any idea how I could set it anyways? 
Maybe it is not working because I can not set the lancache ip.

 

Link to comment
Share on other sites

Link to post
Share on other sites

First of all,

hello LTT community :)

 

Got mine running on a Synology DS918+ (1x 6TB WD RED), inside a Ubuntu 18.04 VM (Virtual Machine Manager), I assigned 3 Cores and 2GB of Ram.

My internet speed is 50Mbps so usually I get around 6.3MB/s.

 

I am currently caching 6 Services, here are some of the numbers i can achieve with this setup :)

 

First time download (Not cached):

Steam - 5.9MB/s (Constant)

Origin - 5.4MB/s (Constant)

Uplay - 6.3MB/s (btw. I had no problem with Uplay at all)

Rockstar - 6.3MB/s (With Uplay fastest uncached download)

Blizzard - 2MB/s (Capped for me at exact 2MB/s or sometimes even lower,..... WoW took sooooo long^^)

Riot - 2MB/s (Capped for me at exact 2MB/s all the time)

 

When cached:

Steam - 30MB/s most of the time (some games like CS:S do 50MB/s)

Origin - 30-50MB/s

Uplay - 30MB/s

Rockstar - When downloading the launcher from the website I get aroud 40-50MB/s. (No cached game yet, I am currently caching GTA V so I will report back later)

Blizzard - 30-40MB/s

Riot - 50MB/s

 

At the end I need to say, this results are awesome for a NAS with an Intel Celeron J3455 Quad-Core CPU and 4GB of RAM xD

Link to comment
Share on other sites

Link to post
Share on other sites

On 8/21/2018 at 6:12 AM, jakkuh_t said:

sudo docker run \ --restart unless-stopped -d \ --name steamcache-dns \ -p 192.168.1.50:53:53/udp \ -e UPSTREAM_DNS=1.1.1.1 \ -e STEAMCACHE_IP=192.168.1.50 \ -e ORIGINCACHE_IP=192.168.1.51 \ -e BLIZZARDCACHE_IP=192.168.1.52 \ -e RIOTCACHE_IP=192.168.1.53 \ -e FRONTIERCACHE_IP=192.168.1.54 \ -e WINDOWSCACHE_IP=192.168.1.55 \ steamcache/steamcache-dns:latest

I Just removed everything apart from the steam cache settings and im not having any trouble.

Its working very well.

 

I am running Ubuntu Desktop 18.04 on a Hyper-v server.

My observation is it likes CPU cores, ive given mine 12 threads just for a laugh and utilization is still around 50% when a client is receiving cached content.

Transfer speeds are around 70 to 80megs a second. Making my poor 2600k sweat alot haha.

Server is a Dell Poweredge T420, 64gig of ram and 2 xeon E5-2440 cpu's with 8 15k SAS drives in Raid 5.

 

Dumb question, im assuming you need to have multiple DNS IPs for each service on the client end?

e.g first dns .50 for steam

then .51 for Origin?

Link to comment
Share on other sites

Link to post
Share on other sites

I keep getting this error 

dc6953ac8957dbef3660fc94dfbe44b547e007489eeb2db064bdb4f60c2d16e8
docker: Error response from daemon: driver failed programming external connectivity on endpoint steam-cache-dns (89976a48e27885f1474bb98a5f3342911eba0d5df0cf7b3bec0cf7d4d9c6e007): Error starting userland proxy: listen udp 192.168.1.100:53: bind: cannot assign requested address.

and this one 

docker logs steamcache-dns
Got permission denied while trying to connect to the Docker daemon socket at unix:///var/run/docker.sock: Get http://%2Fvar%2Frun%2Fdocker.sock/v1.38/containers/steamcache-dns/logs?stderr=1&stdout=1&tail=all: dial unix /var/run/docker.sock: connect: permission denied
HELP!!!

Link to comment
Share on other sites

Link to post
Share on other sites

I have it working to the point where I can see the data coming through my cache server.  When I try and download a game I just installed then uninstalled it still pulls the game data from steam servers and not from the cache.  Anything I'm missing that would cause it not to pull from the cache?

 

Link to comment
Share on other sites

Link to post
Share on other sites

I have some spare hardware lying around. Thinking of trying this. A few questions, should I provide a static IP in my routers DHCP leases for the steamcache, origin cache, and windows cache? Second, should I change the DNS settings on my router to the separate cache addresses as primary then something like 1.1.1.1 for secondary? That way I don't have to changes the DNS settings on all my individual PCs? Or would that mess things up for the router? And do I need all the various caches listed as DNS, or just one? (I just noticed linus says to just configure it in the router, so yes?)

 

Edit: one more question, is it possible to aggregate/team two gigabit ports to get 2gbps? (my switch supports it) If so, how with this setup? Or is that a huge can of worms?

Link to comment
Share on other sites

Link to post
Share on other sites

I was able as well to set it up. But normally I have a download speed of 45-50 MB/s but with steamcache the inital download is at max. 8-9 MB/s

What reason could that have?

Link to comment
Share on other sites

Link to post
Share on other sites

12 hours ago, salexes said:

It looks quite similar I mean I can set a custom IP but for me on QNAP there is no field where I can set the LANCACHE_IP any idea how I could set it anyways? 
Maybe it is not working because I can not set the lancache ip.

 

This is how my UnRAID SteamCacheBundle is set:

docker run -d --name='SteamCacheBundle' --net='br0' --ip='10.0.0.253' --privileged=true -e TZ="Australia/Adelaide" -e HOST_OS="unRAID" -e 'LANCACHE_IP'='10.0.0.253' -e 'TCP_PORT_80'='80' -e 'CACHE_MEM_SIZE'='2048m' -e 'CACHE_DISK_SIZE'='100g' -e 'CACHE_MAX_AGE'='150d' -e 'TCP_PORT_443'='443' -e 'UDP_PORT_53'='53' -e 'UPSTREAM_DNS'='1.1.1.1' -e 'USE_GENERIC_CACHE'='true' -e 'DISABLE_FRONTIER'='' -e 'DISABLE_ORIGIN'='' -e 'DISABLE_RIOT'='' -e 'DISABLE_STEAM'='' -e 'DISABLE_UPLAY'='' -e 'DISABLE_BLIZZARD'='' -e 'DISABLE_WINDOWS'='' -v '/mnt/user/gen-cache':'/data/cache':'rw' -v '/mnt/user/appdata/gen-cache/logs':'/data/logs':'rw' 'mlebjerg/steamcachebundle:latest'
Link to comment
Share on other sites

Link to post
Share on other sites

On 8/21/2018 at 3:10 PM, sirikon said:

This seems to work because the requests made to download steam games are using HTTP, not HTTPS. This means that anyone could make a MitM attack and replace the game we're downloading with some malicious software.

Not it they're digitally signed.

Link to comment
Share on other sites

Link to post
Share on other sites

So if I understand correctly, I'd have to change my DNS on my windows machines depending on which cache server I want to pull from in the list?  For instance if i have it set to 192.168.1.50 (Or whatever IP I used) to pull from steamcache, If i want windows cached updates I'd have to change  my DNS server to 192.168.1.55 to pull from the windowscache docker for cached updates???

 

 Sounds like a lot of tedious work if you want to use it for multiple things at once.  Or am I missing something?

 

  -e STEAMCACHE_IP=192.168.1.50 \
  -e ORIGINCACHE_IP=192.168.1.51 \
  -e BLIZZARDCACHE_IP=192.168.1.52 \
  -e RIOTCACHE_IP=192.168.1.53 \
  -e FRONTIERCACHE_IP=192.168.1.54 \
  -e WINDOWSCACHE_IP=192.168.1.55 \
  steamcache/steamcache-dns:latest
Link to comment
Share on other sites

Link to post
Share on other sites

Create an account or sign in to comment

You need to be a member in order to leave a comment

Create an account

Sign up for a new account in our community. It's easy!

Register a new account

Sign in

Already have an account? Sign in here.

Sign In Now


×