Setting up a local Steam cache

[AshleyAshes]

So, yeah.  I have a server I've mentioned in previous thread.  It may be a Windows 10 Pro server but that doesn't mean it can't run Docker images with the right software.  I've been eyeballing the idea of running a dedicated local Steam Cache on my network that my Steam client can then hit up when re-downloading previously cached Steam games.   I dunno about you guys but there's no such thing as a 'quick reinstall' of GTA5 and pulling it off the local cache at 1gbps would be a lot nicer than the 30mbps I pay for from TekSavvy.

 

https://github.com/steamcache/steamcache

 

This is what I've been eyeing but I'm wondering up if anyone has any experience with it.  I figure it'd be best to give it a dedicated 2.5" HDD rather than using the server's pooled storage.  One downside is that I'll need to use the HOSTS file to insert the location of the cache for my PCs, and that'd mean that the HOSTS file would have to be modified if ever accessing Steam while off my own network.  I could run my own DNS server too, but that means a whole machine or VM within a machine dedicated to DNS for what amounts to basically ONE task.  Also, if my server ever goes down, my entire local DNS is down.  I have an Asus AC66U router but as far as I know, it itself has no features for just 'patching' one little new thing for local clients as they hit their DNS.

 

But a worthwhile discussion, no?

[Electronics Wizardy]

you can install steam games on a external drive and copy the files. It will use all the files you copy and download any missing

[AshleyAshes]

Just now, Electronics Wizardy said:

you can install steam games on a external drive and copy the files. It will use all the files you copy and download any missing

Where's the fun in that!?

[Electronics Wizardy]

Just now, AshleyAshes said:

Where's the fun in that!?

You could also just install the games on the server and use it as a file server.

[AshleyAshes]

Just now, Electronics Wizardy said:

You could also just install the games on the server and use it as a file server.

That would be... Terrible.

 

It also in no way negates the issue of a game being uninstalled from it's installation location and needing to be reinstalled.

[Electronics Wizardy]

1 minute ago, AshleyAshes said:

That would be... Terrible.

 

It also in no way negates the issue of a game being uninstalled from it's installation location and needing to be reinstalled.

You don't need to reinstall it. You just point it to your hdd and hit play. Its already been installed.

[AshleyAshes]

Just now, Electronics Wizardy said:

You don't need to reinstall it. You just point it to your hdd and hit play. Its already been installed.

Sometimes uninstalls become necessary to fix software issues (Looking at YOU GTA5) and a lot of games REALLY can't have their files accessed by multiple computers concurrently.  ...Most really.  Not to mention the terribleness of going from a locally installed SSD to a remote server over a SMB share.  Your idea just adds more moving parts for no purpose other than to slow down access time.  It's also boring and unfun.  I kinda wanna to DO The cache thing, because it'd be a fun and interesting thing to do.

[leadeater]

What you can do is keep a copy of all your games on a NAS, as mentioned, but not use them for playing the game. When you need to reinstall copy the files in to the steam apps folder and then verify the game content, on next run this will also trigger the installation scripts to fix any issues that were the reason for the reinstall.

 

Not much different to the steam cache idea but either way the steam cache container looks like the better idea.

 

As for DNS you can set a primary DNS server (a local VM or container) and a secondary DNS server (router/ISP) so if your local one goes down everything will still work.

[tt2468]

I'm probably one of maybe 3 people on this forum that have actually tried the steamcache docker images. Unless you are setting up for a LAN and NEED it (like me), don't waste your time. The docker images are great when they work, but take me hours of troubleshooting every time I have another LAN. You are willing to go try it out. They do work, but you gotta expect a VERY bumpy road.

[AshleyAshes]

The DNS issue I think I've found the way to make it work.  The Asus AC66U, while there's nothing in it's GUI, it's firmware is BusyBox Linux.  So I should be able to modify the HOSTS in the router so that any attempt to access steam was redirected to the local Steam cache.  I'd just have to do it by commandline in BusyBox.

[dendryganarren]

I plan on running this at a LAN for around 80 persons soon, and so far all my testing have been successful.
In a couple of days I will test caching pvp.net, origin and others using the same docker file and some DNS magic.

 

EDIT:
Here is a short conclusion of my work until now:
I run the cache on two Intel 750 SSDs in RAID 0 so that i can saturate 2*10Gb/s links.
While testing i got up to 790Gb/s until the SSD that the game was installed to bottle necked. Since my switch refuses to work with 10Gbit I have began looking into using LAGG to serve content off my 6*1Gbit ethernet ports.

Edited October 16, 2016 by dendryganarren

[dendryganarren]

I actually ended up using the generic cache instead of steam cache, since the latter refused to work with League of Legends.
After a bit of fiddling with docker and the hosts file (I will be running pfsense at the event, but at home I still run the ISP provided one) I managed to get both steam and LoL to work over a single NIC.
Next goal: LAGG

[Deadline]

Hey guys

 

 

I have been playing with steam cache servers on and off for quite some time now, I have previously setup the method others seemed to have used in this thread just for fun and geek out at steam downloading at gigabit speeds :D. The method i previously used was using nginx on ubunu using the guide from this site https://blog.multiplay.co.uk/2014/04/lancache-dynamically-caching-game-installs-at-lans-using-nginx/ 

 

Now the way this type of steam cache works is by using DNS redirects to a local cache server but i have found the biggest issue is keeping the DNS host redirects list up to date since steam has many different content servers, often changes or adds new ones sometimes on different domains often offloading them onto ISPs which for me was the biggest issue with this older method of caching. I'm not sure but I'm guessing this is the issue tt2468 is talking about.

 

I have recently found this website http://opensourcelan.com/blog/2016/07/01/steam-cdn/ (explanation) https://github.com/OpenSourceLAN/steam-hijack (github files). Now this looks like a much better way to cache steam as it actually interrupts the way steam requests the steam content servers and redirects it to use your own cache server. I haven't tried it yet but it seems like a great idea not having to constantly update the dns redirects, though it does mention on the site that it does add more overhead and is more complex but i think the extra complexity and overhead out weigh not having to keep updating dns entries. 

 

Would be great if those who do really need this kinda steam cache or just those who like to tinker like me try this way of steam caching and share their thoughts, results or any other useful info.

 

 

[aubryscully]

20 hours ago, Deadline said:

Hey guys

 

 

I have been playing with steam cache servers on and off for quite some time now, I have previously setup the method others seemed to have used in this thread just for fun and geek out at steam downloading at gigabit speeds :D. The method i previously used was using nginx on ubunu using the guide from this site https://blog.multiplay.co.uk/2014/04/lancache-dynamically-caching-game-installs-at-lans-using-nginx/ 

 

Now the way this type of steam cache works is by using DNS redirects to a local cache server but i have found the biggest issue is keeping the DNS host redirects list up to date since steam has many different content servers, often changes or adds new ones sometimes on different domains often offloading them onto ISPs which for me was the biggest issue with this older method of caching. I'm not sure but I'm guessing this is the issue tt2468 is talking about.

 

I have recently found this website http://opensourcelan.com/blog/2016/07/01/steam-cdn/ (explanation) https://github.com/OpenSourceLAN/steam-hijack (github files). Now this looks like a much better way to cache steam as it actually interrupts the way steam requests the steam content servers and redirects it to use your own cache server. I haven't tried it yet but it seems like a great idea not having to constantly update the dns redirects, though it does mention on the site that it does add more overhead and is more complex but i think the extra complexity and overhead out weigh not having to keep updating dns entries. 

 

Would be great if those who do really need this kinda steam cache or just those who like to tinker like me try this way of steam caching and share their thoughts, results or any other useful info.

 

 

@jmart604 @Slick Do yall do something like this for benchmarking already?  Seems like it could save some time constantly downloading games on different machines

[AshleyAshes]

Honestly, I found an alternative... I got a raise and then got 250mbit internet. ^_^;