Steam Caching Tutorial

[Poet129]

Any way to add HTTPS Cache to lancache?

[brwainer]

1 hour ago, Poet129 said:

Any way to add HTTPS Cache to lancache?

HTTPS can’t be cached by design. You would need to make a new global certificate and import it into every client device. Even if you think you are willing to do this, it is not a good idea. If you want to know why then research it yourself.

[Poet129]

18 minutes ago, brwainer said:

HTTPS can’t be cached by design. You would need to make a new global certificate and import it into every client device. Even if you think you are willing to do this, it is not a good idea. If you want to know why then research it yourself.

I know the huge security risk but I wanted to do this for my slow Internet connection for my family.

[brwainer]

31 minutes ago, Poet129 said:

I know the huge security risk but I wanted to do this for my slow Internet connection for my family.

My only recommendation is don’t try to do it with lancache (meaning making modifications to their containers) because then you can’t easily update it. You can do research on adding it to squid. I have used squid on PFSense in conjunction with lancache before, to cache anything from general HTTP that I could (lancache only caches the specific servers). I set Squid up on pfsense as a transparent proxy and told it to not cache anything from the lancache server’s IP (to make sure it isn’t double cached). In theory you should be able to do the same, but add HTTPS to squid. 

[Poet129]

1 hour ago, brwainer said:

My only recommendation is don’t try to do it with lancache (meaning making modifications to their containers) because then you can’t easily update it. You can do research on adding it to squid. I have used squid on PFSense in conjunction with lancache before, to cache anything from general HTTP that I could (lancache only caches the specific servers). I set Squid up on pfsense as a transparent proxy and told it to not cache anything from the lancache server’s IP (to make sure it isn’t double cached). In theory you should be able to do the same, but add HTTPS to squid. 

Sounds great but I couldn't get squid to work. Cache wise it was entirely empty... probably just a config error though.

[brwainer]

1 hour ago, Poet129 said:

Sounds great but I couldn't get squid to work. Cache wise it was entirely empty... probably just a config error though.

I had no issues setting up squid caching following the pfsense/netgate instructions for it, and I don't have any knowledge of it beyond following those instructions. But I did see that very little was being cached even when throwing a small LAN party at it (plus the lancache, which did have quite a lot in it)

[Fishscene]

As of right now, all instructions for lancache no longer work or are out of date unless they are referencing the instructions found at: https://lancache.net/

 

Specifically:

 

git clone https://github.com/lancachenet/docker-compose lancache
cd lancache
nano .env ## Modify to suit your environment
docker-compose up -d

 

In both my VM and physical server, after setting this up, I noticed some odd behavior... steam downloads were incredibly inconsistent - whether the game was cached or not.

For example, I'd be downloading a game and then it would slow down drastically and pretty much stop all together. Then after a couple of minutes, it would pick up speed, but never to the original download speed. Last night, I made a huge breakthrough:

1) Windows 10

2) Settings

3) Update & Security

4) Windows Security

5) Virus & threat protection

6) Virus & threat protection settings > Manage settings

7) Disable "Cloud-delivered protection"

This one simple trick sped up my steam downloads to near gigabit speed for the entire download. (Server uses a single SSD for caching).

 

But wait, there's another problem.

DNS resolution failures have been occurring randomly. It's like something gets hosed up or overloaded and lookups fail randomly. I'm still testing, but here's my original DNS chain/hierarchy:

My Computer > lancache > piHole > Gateway Router > 1.1.1.1

 

What I'm currently testing:

My computer > piHole > lancache > Gateway Router > 1.1.1.1

 

So far, it looks promising, but I haven't yet figured out how to test DNS upstream lookups. It's still likely the PiHole is just returning its cached DNS entry and forward lookups are still broken on the lancache while it is being used.

 

[Sameer Akhtari]

I done every step and for steam only ...
but there is a problem , using my laptop (which is working on wifi ) steam downloads are not working through cache server ...!
Is that because laptop is connected to wifi and I setup for Ethernet on setp5 (sudo nmtui...) ????
Also I checked the Cache folder on server (after game was downloaded on client pc ) the folder was empty with no files .
[ I haven't checked using Ethernet cable on laptop cause there is only 1 working cable which is currently connected on Cache Server PC and because of corona lockdown  markets are closed  ]
Any one help plz..

[brwainer]

7 minutes ago, Sameer Akhtari said:

I done every step and for steam only ...
but there is a problem , using my laptop (which is working on wifi ) steam downloads are not working through cache server ...!
Is that because laptop is connected to wifi and I setup for Ethernet on setp5 (sudo nmtui...) ????
Also I checked the Cache folder on server (after game was downloaded on client pc ) the folder was empty with no files .
[ I haven't checked using Ethernet cable on laptop cause there is only 1 working cable which is currently connected on Cache Server PC and because of corona lockdown  markets are closed  ]
Any one help plz..

 

On 3/16/2020 at 12:53 AM, brwainer said:

This guide is out of date / not the most recommended way to set this up anymore. There is a much easier container to use now that doesn't require adding more IPs or a lot of the fuss you have. I understand if you don't have time/resources to update the guide, but I think it would be good to add a note at the top of the post to the effect of "This guide is outdated, please see http://lancache.net/docs/ (If you need help with getting a Linux system with Docker, then you can follow the instructions below up to step 4."

Please see my post above, because the instructions as they are in the first post really aren't correct anymore - Steam has made changes, and only specific setup which the instructions here sets up doesn't work anymore. The developers changed over to a new project called LanCache and it is much easier to set up now. I recommend you start over - although you should be able to keep you docker server as-is, its mainly the containers which you need to remove.

[bbull]

On 4/7/2020 at 9:03 PM, brwainer said:

I had no issues setting up squid caching following the pfsense/netgate instructions for it, and I don't have any knowledge of it beyond following those instructions. But I did see that very little was being cached even when throwing a small LAN party at it (plus the lancache, which did have quite a lot in it)

Squid works, but you need to configure /usr/local/libexec/squid/storeid_file_rewrite to collapse the identical chunks from multiple CDNs to local names. Enabling caching of dynamic content is required of course.

 

ex. /usr/local/etc/squid/storeid_file_rewrite.conf

 

^http:\/\/.*steam(?:content|powered)\.com\/depot\/([^?]*)   http://valveN.steamcontent.com.squid.internal/depot/$1
^http:\/\/.*cdn\.gog\.com\/([^?]*)   http://gog.com.squid.internal/$1
^http:\/\/.*blizzard.com\/tpr\/([^?]*)   http://blizzard.com.squid.internal/tpr/$1

 

[Fishscene]

Hey all, a bit of an update:

 

I've noticed an oddity when using lancache, specifically for Warframe.

 

The problem:

After setting up lancache, I noticed when I was in my lander, I'd be ok for a bit, then I'd drop to about 5fps and it would stay like that forever until I ran a mission. The mission would always play flawlessly, but when I returned to my lander, there was a 75% chance I'd immediately drop back to 5fps. If it didn't, it was only a matter of minutes before it would do so.

 

The solution:

I checked logs on lancache and noticed DNS lookup timeouts for "content.warframe.com". Doesn't make sense because other domains weren't having issues and all other DNS servers up the chain could resolve it just fine. So I manually added the IP address to my PiHole under "Local DNS Records" on the webconfig (If you don't see this, upgrade to the latest available update). Here's my entry:

content.warframe.com 205.185.216.42

 

I've been testing this for a long time now and have had 0 issues. I have not reported this upstream to lancache.

-----------------

I also wanted to mention I've had great success in arranging my DNS hierarchy in this way:

Computer -> PiHole -> lancache -> Router -> DNS_Server_on_the_Internet

 

This specific arrangement allows me to add and remove the lancache server from the DNS hierarchy without affecting my network clients at all. I just go to the PiHole and point it to my router and Voila! I can update/restart/test/do whatever with the lancache server.

[Soul Eather]

Are people still in need of a tutorial?

Cuz i found a way to install lancache+dns+sisproxi+pihole only using docker on Synology. No VM, no proxy, no conflict on 80 and 443.... the good stuf

[Poet129]

52 minutes ago, Soul Eather said:

Are people still in need of a tutorial?

Cuz i found a way to install lancache+dns+sisproxi+pihole only using docker on Synology. No VM, no proxy, no conflict on 80 and 443.... the good stuf

That would be cool.

[Soul Eather]

I wrote my first tutorial here if someone si interested.

Here are the general steps to make this work.

[Soul Eather]

now i need anime
 

[ageekhere]

maybe QoS3 "Secure Caching in HTTPS Based on Fine-Grained Trust Delegation" Will help in the future https://www.hindawi.com/journals/scn/2019/3107543/#conclusion-and-future-work

[Kitsep]

So, I had multiple paragraphs typed out about how I was having a HELL of a time getting it to work, even after correcting for the new caching service... turns out it was just a typo where I had a single - instead of two -- in front of name.

 

Once I figured that out, it works like a dream! (Probably a good idea to update the tutorial with the new service names though!)

 

- Kit

[GodAtum]

Hi all, I've setup everything but stuck on the last step. I cannot change my gaming PC's DNS because it already points to the DNS server I have (PFsense). What can i do?

[brwainer]

2 hours ago, GodAtum said:

Hi all, I've setup everything but stuck on the last step. I cannot change my gaming PC's DNS because it already points to the DNS server I have (PFsense). What can i do?

Why can’t you set the PC to point to the steam cache, and the steam cache to point to PFSense? The steam cache container has a setting for upstream DNS server.

[Hawk421]

Maybe this has already been asked,  but could someone potentially use this for NETFLIX? This would be great for apartment landlords, or ever at home caching idk? Anyone have some thoughts on this and the legality?

[brwainer]

2 hours ago, Hawk421 said:

Maybe this has already been asked,  but could someone potentially use this for NETFLIX? This would be great for apartment landlords, or ever at home caching idk? Anyone have some thoughts on this and the legality?

All the video streaming services use either HTTPS or QUIC, both of which have the content encrypted differently for each client and thus unable to be cached by a network device in the middle.

 

For large complexes (thousands of apartments) they can apply to have a Netflix server placed directly onto their network: https://openconnect.netflix.com/en/

“For offload efficiency and system scale, we require a minimum of 5 Gbps of peak Netflix traffic per deployment site”

[jacob_samd]

Nevermind. Found an update idea of this

[Klinkers]

Is there any update on install instructions for a Steam Cache Server On Ubuntu 16 / 18 / 20 ? as steamcache doesn't seem to exist now and lancache seems to not work properly either.

 

A step by step guide would be cool as I am a bit of a Linux Noob.

 

 

Even A windows Tutorial would be cool. I can use Win 7 - Win 10 on the machine i am wanting to use.