Jump to content

Steam Caching Tutorial

jakkuh_t
51 minutes ago, bashbro said:

5b8d421eed0af_ScreenShot2018-09-03at10_11_59AM.png.9a0a5c9936c015ead340d66497af3890.png

 

This does not look like the way Linus did it on the video. Preferred DNS Server was the server itself 192 blah blah blah. 

I forgot to mention this is the server settings (cache server)

Link to comment
Share on other sites

Link to post
Share on other sites

On 27-8-2018 at 1:36 AM, Androidz said:

@Dred88 - You would point to steamcache-dns IP.

To further explain this:

There are 2 types of services being started:
 - One type is the actual caching containers. They are per service (Steam, Origin, ...)
 - The other type is a container that takes the incoming requests, and then either forwards them to the caching containers or just forwards towards the internet if no container is found for the service.

You need to point your Windows machines to the second type.

That time I saved Linus' WiFi pass from appearing on YouTube: 

A sudden Linus re-appears : http://linustechtips.com/main/topic/390793-important-dailymotion-account-still-active/

Link to comment
Share on other sites

Link to post
Share on other sites

Solution if  everything runs okey but it's not caching

the solution it was the firewall I disabled it and everything is working fine

Link to comment
Share on other sites

Link to post
Share on other sites

i got data coming through the server, but it does not pull any data from the cache folder.
how do i fix this?

i followed the guide step by step, but it didn't work, then i changed some things according to other people in the post and actually got Data to come through, I opened all the IP's, turned off the firewall [on the server], ports are forwarded, everything should be right, just wanna make sure Steam and Battle.net works for now, then i can always add more.

these are the commands i use:

 

docker run --restart unless-stopped -d --name steam-cache -v /cache/steam/data:/data/cache -v /cache/steam/logs:/data/logs -p 192.168.0.210:80:80 -e CACHE_MAX_AGE=60d steamcache/generic:latest

docker run --restart unless-stopped -d --name blizzard-cache -v /cache/battlenet/data:/data/cache -v /cache/battlenet/logs:/data/logs -p 192.168.0.211:80:80 -e CACHE_MAX_AGE=60d steamcache/generic:latest
docker run --restart unless-stopped --name steamcache-dns -p 192.168.0.210:53:53/udp -e UPSTREAM_DNS=1.1.1.1 -e STEAMCACHE_IP=192.168.0.210 -e BLIZZARDCACHE_IP=192.168.0.211 steamcache/steamcache-dns:latest

 

Link to comment
Share on other sites

Link to post
Share on other sites

Got everything working (i think) but when i type in 

sudo docker exec steamcache /scripts/watchlog.sh

i get: 

#Watching logs...

and when i download a game, i get no output from the watchlog...

DNS-Server on Client is the server-ip.

 

 

 

Link to comment
Share on other sites

Link to post
Share on other sites

On 9/1/2018 at 6:15 AM, Shadowman94 said:

Yeah i got same problem with steamcache... but the main problem is... when i want download a game (e.g. cs:go, dota 2, AOE2:HD ), the steam is start downloading with https protocol.. (what is not able to cache, only http), but with Battlenet app, windows update, uplay is works fine.


I'm having this issue as well. If I stop sniproxy, steam will use 80 and it works fine. If sniproxy is running, steam switches to 443 and won't cache.

Link to comment
Share on other sites

Link to post
Share on other sites

On 8/21/2018 at 4:12 AM, jakkuh_t said:

Otherwise, you need to determine the amount of additional IPs you will require. Add 1 additional IP per extra service you'd like to cache. For example, if you wish to cache steam and origin, you would need 1 additional IP on top of the existing IP of the system, for a total of 2 IP addresses.

Uhh, silly question. By saying "add 1 additional IP" did he mean add it to the DNS configuration (as per his example) or do we have to reconfigure the networking config of Ubuntu for this? And a follow up question, running docker files for the other game cache docker images shouldnt be a problem right? 

Link to comment
Share on other sites

Link to post
Share on other sites

On 8/21/2018 at 5:20 PM, Jelsie said:

Has anyone got this working with Windows Update?

This would be called a WSUS server. Windows update is a different animal because of the differences in hardware and OS versions inside a network.

Also the caching is nice if you have data caps or a slow download speed, but honestly the slowest part of the process is how steam decompresses and installs games while they are being downloaded. This means if you have a slower hard-drive, one that is fragmented, or closing in on capacity limits.. then this can REALLY hamper this setup.

 

If you are doing something like this... you might be better off running a raid 0 setup and grabbing the items straight from the steam folder on that machine. If you can afford a 10Gbe setup then you can do do something similar to my setup. I had a lab full of computers that nephews and kids play games on. I was using a ton of space on each machines SSD for the games they play... I moved all of the common games to a server running gbe and a could SSD's in raid 0. I then mapped drives and locations to these items on the other machines. So when they launch a game they are pulling it from the machine with the SSD's. The ssd setup provides about 1200-1350 read/write. Being an SSD there is no seek time so that isn't a concern. Then being on 10 Gbe is have about 1250 maximum transfer rate over the wire... which is actually more like 1200 ish. Which means that while these games load a little slower than if they were using the local SSD, they still launch much faster than a 7200 rpm drive. So instead of a 6-8 second launch it might be more like 10-12 now.

Link to comment
Share on other sites

Link to post
Share on other sites

I am not able to get the data to actually cache to the server. I followed the instructions and pointed DNS to it. I have attempted turing off sniproxy  and I still got nothing. Everything is passing through the server fine tho. I can give any info needed just ask.

 

Link to comment
Share on other sites

Link to post
Share on other sites

21 hours ago, Lucas319315 said:

I am not able to get the data to actually cache to the server. I followed the instructions and pointed DNS to it. I have attempted turing off sniproxy  and I still got nothing. Everything is passing through the server fine tho. I can give any info needed just ask.

 

I actually figured it out after reading old comments multiple times. You have to do every suggestion. do not do it then undo it that was my issue. I could not get 

sudo docker exec steamcache /scripts/watchlog.sh to work properly.
Link to comment
Share on other sites

Link to post
Share on other sites

Hello, Someone has the configuration for the launcher Epic games for fortnite?
thank you :)
Link to comment
Share on other sites

Link to post
Share on other sites

containers are not always running when i start the server but a restart will solve the problem, any suggestions what is the cause of the problem.

 

Thank you.

Link to comment
Share on other sites

Link to post
Share on other sites

  • 2 weeks later...

I did everything you did - nothing more, nothing less, but when I run

On 8/20/2018 at 10:12 PM, jakkuh_t said:

sudo docker container ls

The only output I am getting is:

CONTAINER ID        IMAGE                        COMMAND                  CREATED             STATUS              PORTS                  NAMES
645670f6079c        steamcache/sniproxy:latest   "sniproxy -f -c /etc…"   About an hour ago   Up 24 minutes       0.0.0.0:443->443/tcp   sniproxy

And I did it for Steam, Battle.net and Windows Update!

When I am downloading anything I am staying at normal speeds even when downloading twice, what should cache the files.

I am linux newbie and I dont know any further. Help would be appreciated very much!

Link to comment
Share on other sites

Link to post
Share on other sites

I am getting this error when adding the containers pls help:

docker: Error response from daemon: driver failed programming external connectivity on endpoint steam-cache (b7d7fb5758441f8a7f986d78457aab07d0d4f631cf3f47d1112565011f7a18ac): Error starting userland proxy: listen tcp 192.168.178.191:80: bind: cannot assign requested address.

 

Link to comment
Share on other sites

Link to post
Share on other sites

Hey!

I have created steamcache using your tutorial. After a lot of pain with linux (don't get angry linux fanobiz:) I'v finally made it work!
Some specs:
CPU:       Intel Celeron G3900 (Skylake 2x 2.8GHz)

RAM:      4GB DDR4 2133MHz

MOBO:   Asrock H110M-ITX

SSD:      64GB Adata 

HDD's:   2x WD Blue 1TB (7200RPM 64mb Cache)  

              1x Seagate Barracuda (7200RPM 64MB Cache)


I'v spent around 200 Euro's. I had 2 HDD's already so I had to buy just one HDD for 30 Euro (ebay) CPU 30 Euro (ebay), MOBO, RAM and Case are from online shop (100 Euro).
SD (20 Euro ebay). Rest, like Sata cables etc. were bought online. 

I'm pretty impressed with results. I'v created Raid 0 partition (3x 1TB HDD). System is on SSD. I think I lack some memory. CPU isn't speed monster either but gets the job done. Transfers will be different with different games. Some will achieve almost 80mb/s. Some "only" 40mb/s. With my shitty internet connection I really enjoy those speeds. Sure, 3 SSD's or more would be overkill but as you can see such steam cache server can be done with just under 200 Euros. Nice! 
I think that speeds over 100mb/s will need SSD to work. I think it's HDD limitation but I might be wrong.

Steam / Origin / Blizzard / Riot Games work perfect. I will try to add Epic Games Launcher, EvE Online Launcher and Oculus soft. No idea if this will works with Oculus app, would be nice to have VR Games alredy in cache!

 

Some screenshots with transfers. I will update as soon I will try to add container for Oculus. I hope this will work! :)

 

79 mb s.jpg

40mb s.jpg

Link to comment
Share on other sites

Link to post
Share on other sites

Hi, so i have a debian linux based nas (omv).

 

I want to use 1 of my drives for this. But is it posible just from command prompt in debian and can i set it up to use 1 drive?

 

And i can mannage everything now from web ui can that still work. And does 5 min spindown still work?

Edited by andre7455
Link to comment
Share on other sites

Link to post
Share on other sites

  • 3 weeks later...
On 8/30/2018 at 10:10 AM, Shadowman94 said:

Try to change slice 1m; to slice 10m; in config in blizzard container.

how do i do this? i need the commands and stuff. thanks

 

my battle net downloads are painfully slow

Link to comment
Share on other sites

Link to post
Share on other sites

also my steamcache isnt caching. im assuming because its going to the comcast CDN... no fix for this yet?

Link to comment
Share on other sites

Link to post
Share on other sites

  • 2 weeks later...

Since I saw the video on youtube, I've been going back and forth whether or not to do this cache-machine. I'm leaning towards it though :)

The parts that I understand are:

1. If storage is large enough, you can save all your games on server, rather than to re-download them again from the internet.
2. You can have multiple cache set to different clients on cache server (windows, steam, etc).
3. The IP's set to specific cache are only on the cache server, not on the windows machine, so your windows machine(s) just downloads locally (no need to set ip on windows machine to download on specific cache IP).

 

Where my understanding is lacking is, how does it work with game/windows updates?
 

When I start my PC up and subsequently start a game, sometimes it starts downloading an update (client or game update). Will this update be pre-downloaded on the cache server then?

And with Windows, the updates will be pre-downloaded on the server, so I have to, periodically, update my windows manually?

Writing all of this makes me feel like I already know the answers. :)

Link to comment
Share on other sites

Link to post
Share on other sites

I am thinking of adding a cache docker for steam on my home server. Not sure if it's the best solution or best way to do it. 

 

I don't play much since I have a young family and my connections is not really fast and limited on steam due to providers. 

 

I am annoyed with waiting for updates mostly or download when slow. 

 

I play some games like rocketleague on my living room computer, but having to wait 30 minutes for updates is not possible and annoying and computer is turned off when not in use. 

 

If I had a steam cache, is it possible to force or keep some of the games updates or downloaded all the time or on specific time of the day (night)? 

 

Or does I need to run a second system (physic or virtual) to first load them? 

Or is there another solution other than running backup games? 

Link to comment
Share on other sites

Link to post
Share on other sites

On 11/8/2018 at 8:30 AM, Juunasv said:

Since I saw the video on youtube, I've been going back and forth whether or not to do this cache-machine. I'm leaning towards it though :)

The parts that I understand are:

1. If storage is large enough, you can save all your games on server, rather than to re-download them again from the internet.
2. You can have multiple cache set to different clients on cache server (windows, steam, etc).
3. The IP's set to specific cache are only on the cache server, not on the windows machine, so your windows machine(s) just downloads locally (no need to set ip on windows machine to download on specific cache IP).

 

Where my understanding is lacking is, how does it work with game/windows updates?
 

When I start my PC up and subsequently start a game, sometimes it starts downloading an update (client or game update). Will this update be pre-downloaded on the cache server then?

And with Windows, the updates will be pre-downloaded on the server, so I have to, periodically, update my windows manually?

Writing all of this makes me feel like I already know the answers. :)

Best thing is to change the DNS on your router, so instead of contacting the windows or steam server, it will contact your cache. If it's not present on the cache, the cache will download it and forward it to the client. So speed will be the same as directly downloading it from the server (windows or steam). 

 

The issue you might be facing is the same as me that I not sure if there is a walk around. You first need to have a computer to download it first time. I don't think you can force one game or update on the cache server directly. 

Otherwise it would be perfect for me. 

 

My other solution would be to have a temporary computer or virtual open to force the download/install but it would require the double quantity of space, one for that system and one for the cache... 

Link to comment
Share on other sites

Link to post
Share on other sites

Just discovered this tonight and I gotta say this is pretty cool software. I made a new virtual machine with ubuntu and ran the Docker commands in terminal. It was all surprisingly easy to setup. I had to change virtual box's network adapter settings to "bridge mode" instead of NAT. But that was just about the only tweaking I had to do. Very cool app!

 

I'm running my VM from a Samsung T5 SSD with 250GB of space reserved for caching (for now- just to test). One thing I'm noticing is when caching Xbox One game downloads, the initial download is super slow. The 1st time I get speeds like ~50mbps and then subsequent downloads are at 250mbps. Is there any way around this? Without caching, my download speeds are ~250mbps. I get that overhead is to be expected, but the initial download is like one-fifth my normal speeds. Is this normal or am I doing something wrong?

 

Running my caching server in Ubuntu (newest ver) through a VM (VirtualBox, newest ver). 2.8ghz i7 7700HQ with the VM getting 2-cores and 4GB of RAM. Speedtest from within the VM shows 400mbps down. On host (Windows 10) machine it's 850mbps.

Link to comment
Share on other sites

Link to post
Share on other sites

2 hours ago, GoTeamScotch said:

 

Running my caching server in Ubuntu (newest ver) through a VM (VirtualBox, newest ver). 2.8ghz i7 7700HQ with the VM getting 2-cores and 4GB of RAM. Speedtest from within the VM shows 400mbps down. On host (Windows 10) machine it's 850mbps.

 

Wait... you're running containers inside of a virtual box VM on ubuntu machine?

 

Or... the steam cache is running as VMs on the ubuntu system?

 

 

Can Anybody Link A Virtual Machine while I go download some RAM?

 

Link to comment
Share on other sites

Link to post
Share on other sites

3 hours ago, unijab said:

 

Wait... you're running containers inside of a virtual box VM on ubuntu machine?

 

Or... the steam cache is running as VMs on the ubuntu system?

 

 

My PC is running Windows as its main OS. I'm running Ubuntu inside of a virtual machine within Windows. Ubuntu is running Docker and the steamcache containers.

Link to comment
Share on other sites

Link to post
Share on other sites

Create an account or sign in to comment

You need to be a member in order to leave a comment

Create an account

Sign up for a new account in our community. It's easy!

Register a new account

Sign in

Already have an account? Sign in here.

Sign In Now


×