Search the Community

I have a spare server I'm not using. Is there a way to cache some of my streaming services (ie. HBO Max, Hulu, Netflix) for a rainy day? Especially if I am having an internet/power outage or slow internet (I have cable internet, so sometimes slows down during peak usage hours.) I know you can do this with services like Steam, Origin, & Ubisoft. I have my doubts, but it doesn't hurt to ask. Possibly something like an Edge-CDN? (Might have that wrong.) Thanks in advance for any advice!

I'm moving from unraid to proxmox, but need advice with the storage. 2 x 4TB HDD 1 x 1TB NVME 1 x 256GB NVME 1 x 128GB SSD I would like to use proxmox for VM (Windows 10 for gaming), docker containers and network storage. For VM & docker I need speed. There is mixed information about ZFS cache. There is also talk about ZFS & LVM cache. I would like to use 128 GB SSD for proxmox. 2 x 4TB HDD as mirror? But what about those two NVME-drives? What is the "best way" to use those so I have everthing backed up, but also run VM & docker from NVME, not from HDD. Thank you for any advice.

Hi, Not entirely sure if this is the right section to post this in so apologies if its not. I am wanting to improve the transfer speeds from my main system to my server, I have tested and founf unsurprisingly that transferring to the server SSD is far faster than to its HDDs. What im looking for is whether i can set up an SSD as a cache in a storage pool or a way to get the system to automatically transfer any files held on a specified SSD to the HDD. Any advice is greatly appreciated thanks

Hi, I'm looking at using/ upgrading a old optiplex with a SATA3 ssd, 4590, and 16gb ddr3 to try and set up a steam caching server, along with some other smaller servers (2-5 vinilla mc, SCP Secret Lab, etc) Mine main question is what my bottlenecks are going to be and how much it's going to affect performance of the cache. Ideally once I have that then i can start looking at what upgrades I should get and/or whether they are going to be worth the cost. I also don't know if this is the right place to be asking so please point me in the right direction if it's not Thank

My system currently has 2x 8TB HDDs, an unused SATA SSD and an NVMe boot drive. I want to use the SATA SSD as a cache. In my research so far there is a software called primocache. My question is are there any free alternatives? If not is primo cache worth the money? System Specs: Motherboard: B550 Aorus ELITE AX V2 CPU: R7 5800x GPU: Gigabyte Gaming 3080Ti RAM: 4x 16GB Team group Delta R @3200MHZ (XMP) Storage: 2x 8TB Seagate Ironwolf, 1x 256GB Gigabyte NVMe boot drive, 1x 256GB Samsung 860 EVO SATA SSD CPU Cooler: iCUE H115i ELITE CAPELLIX Liquid CPU Cooler Case fans: 3x Cooler Master MF120 PSU: Corsair RM 850w

hey we are setting up a game caching server and we are wondering what caching options you are recommending.

We hear quite often these days of developers and gamers complaining games aren't optimized for graphics hardware for AMD/Nvidia or are choking at the CPU whether by terrible single threaded implementation or by bad multithreaded code. In reality it's not very difficult to optimize C++ code, and game developers should frankly be ashamed they hide behind the excuses they currently give. Here I have supplied a rudimentary program which may not appear to be very useful or worthwhile to the average reader, but these specifically can be used as part of a CPU-based physics engine or Artificial Intelligence routine in the hands of someone who can manage data well. Here are a few ways to implement getting the sum of the elements of columns in a matrix. You may not find this particularly interesting, but for any mathematical/statistical reduction, it's best to try to get all of your data in matrix form, be it a 2D array or a flat buffer you can traverse by using multiplicative offsets. Furthermore, it's best you know how your programming language stores data sequentially. Going in the wrong order can have disastrous consequences on your runtime (yay foreshadowing). Trials run per code base: 6 CPU: Core I7 2600K stock clocks & settings, watercooled (no thermal throttling possible) RAM: dual-channel G.Skill Trident X 1866 (3x4GB) OS: Ubuntu 15.04 Vivid Vervet fully updated as of 5/14/15 Compiler: icpc (ICC) 15.0.3 20150407 Copyright © 1985-2015 Intel Corporation. All rights reserved. Compiler Flags: -Wall -Wextra -std=c++11 -g -Ofast dummy.cpp -o dummy Timing Specifications: /usr/bin/time -v ./dummy 115000 20000 Major (requiring disk I/O) page faults: 0 (very important that we're not hitting virtual memory on the hard drives here) CPU Usage: 99% for all valid runs Be aware due to SandyBridge being limited to AVX 128 (with MMX 256) and FMA2, you will see better performance in Haswell, but you will need a lot of RAM to see how it scales. I use up most of my available 10 GB, and the system has the last 2. Naive/First Instinct Serial Solution //some have noted I forgot to use a reference operator for my function, but adding it shaves only a single second in runtime, insignificant until later. Elapsed time average: 53.46 seconds Standard deviation: 0.27 seconds C++ Favorable Serial Solution Elapsed time average: 4.90 seconds (that's right, 10.91 x speedup with 1 tiny change) Standard deviation: 0.003 seconds And that's just by understanding how C++ stores multi-dimensional arrays: row major order, and how the translation lookahead buffer tries to keep spatially local data in the caches for your CPU. Traversing by column causes a huge number of cache misses which, as you can see, severely diminishes your performance. Feel free to experiment by adding columns, narrowing row width, expanding it, etc.. This second algorithm seems to be slightly obfuscated to a novice or someone who doesn't know the nitty gritty details of a language's implementation (COUGH* game programmers COUGH*). Believe it or not, this sort of naivete I found almost an exact copy of in the physics engine for the Unity Engine. You wouldn't believe how hard I had to fight to get it changed, including providing extensive evidence you can get here for yourselves. I could not believe how impossible it was to convince three programmers some upstart nobody found something they could vastly improve in their code. If I ever come across as an elitist, it's because I'm jaded by industry people who think stuck up college kids know nothing. Now, using Intel's crafted CilkPlus libraries (which run on AMD processors all the same if you compile with g++ or clang++ (g++ 4.9.2 has some CilkPlus bugs right now though)), we can do even better before going multi-threaded. To my knowledge Microsoft's Visual C/C++ compiler does not yet support CilkPlus at all. Good CilkPlus Solution (for g++ or clang++, must use -fcilkplus flag) Elapsed time average: 3.00 seconds (almost 40% speedup over good C++ algorithm) Standard deviation: 0.000 seconds Bad CilkPlus Solution (for g++ or clang++, must use -fcilkplus flag) Elapsed time average: 50.80 seconds (better than bad algorithm in raw C++, but terrible nonetheless) Standard deviation: 0.29 seconds CilkPlus even manages to get you something when your algorithm doesn't use your hardware resources optimally, but it doesn't get you too much. OpenMP and CilkPlus Combined Solution (export OMP_NUM_THREADS=4, compile using -fopenmp (and -fcilkplus if using g++ or clang++) Cpu Usage: 391% Elapsed time average: 1.19 seconds (more than 60% faster than the plain CilkPlus Solution. There is a barrier to scaling here determined by the critical section which must be resolved) Standard deviation: 0.0058 seconds ***Note using 2 threads gives 0.92 seconds elapsed time and lower (174%) CPU usage. This is a lesson that parallelization can yield diminishing returns and even lose you ground if you aren't careful*** Proposals: chunk-wise reduction from partials to total. Edit: no gains. Still confused. Sometimes it's the most obvious things that escape our sight when looking for ways to improve. Memory allocation takes a long time, and trying to do it in one single call was not appropriate for such a large structure. Having said that, reserving a single block of data which will do nothing other than point to the individual rows and then making calls to RAM to allocate much smaller pieces takes far less time. I still believe I'm hitting a bandwidth bottleneck considering increasing the thread count for this increases the runtime, especially given the threshold is 2 and not 3. I believe the increased run time would result from contention to the DDR3 bus, but I lack a hardware profiler to confirm this (there are various levels of profilers which can analyze software and hardware performance, and to pinpoint exactly where the delay is being generated would most likely require a hardware profiler. EDIT: OpenMP Solution was given further optimization, and scaling is now closer to expectation. However, there is more I have to think about. There's definitely still room for improvement.

I am currently running a FreeNAS system (Build 11.0-U2 (e417d8aa5)) with an Intel Pentium G3240, 16GB of RAM and 4 2TB hard drives. The hard drives are WD Reds. Two of the hard drives are in one mirrored volume, which is working fine. The other two hard drives are in another mirrored volume, but transfer speeds seem to be wonky. When transferring large numbers of files, the transfer speed starts off at around 5-10 MB/s, then tanks and stays at 0 KB/s for several minutes, then increases to around 40-60KB/s before starting over at about 5-10MB/s. This cycle continues throughout the entire transfer. I do not have any disk caching enabled, and my RX/TX buffers are set to their default settings. The NAS unit is connected to the local network via CAT6 ethernet, and the client computer can sustain transfer speeds of 350-400Mb/s speeds. Theoretically, I should be able to sustain 40-50MB/s transfer speeds, right?

I've noticed that LTT has some content on using Optane to accelerate large spinning disks for gaming. LMG have also covered some limitations of Optane, like the fact that the performance gains over slower SSDs are negligible for gamers, making it hard to justify its high price. You've also done a video on using Intel Optane for storage acceleration with AMD, which comes with some additional problems. All together, these videos highlight the following limitations of using SSDs as cache to accelerate access to spinning rust, which strike me as important: Chipset specificity: Intel's Optane storage acceleration requires an Intel CPU, and AMD's StoreMI requires specific AMD motherboards Size limitations: AMD's StoreMI only supports cache drives up to 256GB. But games are enormous these days! The Star Citizen example from the ‘fastest SSD for gaming’ video is especially potent: 200GB for a single game! This means that when it comes to the largest games, only a few could be accelerated in this way Prohibitive cost: Intel's Optane drives are VERY expensive per gigabyte, but their unique strengths have little to offer gamers qua gamers Drive specificity: to get this kind of storage acceleration on Intel, you have to use Optane Drive dedication: Intel's Optane storage acceleration option and AMD's StoreMI require you to dedicate an entire device to this. If you only have one SSD, you can't choose to set aside only a partition of it for SSD game caching, while using the rest, for example, as a boot drive Not all options covered have all of these limitations, but there are no options for accelerating HDDs using SSD caching covered by LTT which bear none of them. But Linux, thanks to its long presence in the server space, is lousy with powerful tiered storage solutions that share NONE of these limitations. Specifically, I can think of these storage options which support cache tiers with arbitrary devices off-hand: LVM bcache Stratis I think covering these options would fit well with much of LTT's other content on the subject. It would make it possible to, for example use a 1 TB SSD to cache on a 14 TB HDD for one fourth the price of a 480 GB Optane drive (although using such a large SSD may require some tuning for optimal performance), yielding a large enough cache for several of even the largest games in existence use ALL of a 480 GB Optane drive for caching on an AMD system use a single SSD for a blazing-fast OS experience and game caching on a budget Any of these could make for a nice, flashy headline related to previous LTT content. And I think the video also fits nicely with the more recent uptick in Linux coverage on the part of LMG and increased interest in Linux from LTT's gaming audience. While for the most part, videos up until this point have mostly focused on Linux overcoming being a relative newcomer to the mainstream desktop, this one could show how Linux's background on the server actually strengthens it for certain, special desktop use cases. That's pretty different, and I think the novelty could be fun. Plus, I think this kind of video, because it is relatively technical for Linux content, presents an opportunity to ramp up very cute dynamic of interaction between Anthony as Linux Guru with James as his Linux noob/straight man gamer that we saw in the last Linux gaming video. What do you think? Would anyone else like to see this?

Hello, first post here just wondering if anyone has any suggestions for configuring the drives in my gaming setup. The drives in question: 1x WD Black 256gb NvME SSD 2x WD Black 1tb HDD I currently have the entire SSD as boot drive and HDDs set up in raid0 (through mobo and drivers). The larger storage will be used solely as a games drive and I do not intend on installing any intensive applications on the SSD. My question is, should I just stick with the current configuration and move games to SSD for better performance or use caching/tiering to accelerate game load times? Current ideas are partitioning the SSD (60gb/120gb?) and using this to accelerate the HDDs. Alternatively, I could install Windows to the HDDs and use the entire SSD for caching. Would this give more leeway in terms of the number of separate games cached or just slow down boot? Finally, unsure which software to use for this. I am aware of AMDStoreMI (running on AMD platform so RST is out of the question) and Primocache. Thanks for reading and any help would be greatly appreciated.

Hey there! So, after several years, I finally decided to buy an SSD. I decided to go with Adata XPG Gammix S11 Pro 512 GB, which has TLC cells, pretty good read/write speeds, sufficient lifespan, and great reviews. I was planning to allocate about 100 GB for the OS, an undisclosed amount of data for HDD caching and leave the rest for programs that I use the most. Do you think this is a) possible, and b) worth it? If so, which caching utility would you recommend? Based on Linus' video, I was planning to use Primocache, but I'd like to know if there's a cheaper alternative. (I'd use AMD's StoreMI, but I have a B350 board with 1st gen Ryzen, so that's not an option for me) Thanks in advance!

Ive followed this guide (with the update monothlic / lancache-dnses) Im running 2 vms, pihole and the steam cache. Ive set the dns on the steamcache to point at the pihole. Im using the pihole dns on my gaming rig. But when i download games from steam or blizzard. Its not caching, the steamcache is doing 0. Is it no longer possible to use caching?

I saw the Techquickie "What Are DRAM-less SSDs" and realized that the ssd I just got the a400 Kingston 240GB does not have a DRAM cache. good thing is I didn't open it yet and I can upgrade it to a WD Green ssd for 10$ more, now the WD green also does not have a DRAM cache but it does have SLC caching so is worth he upgrade or is there no real difference between the WD Green and the a400.

I plan to use primocache to speed up my 6tb hard drive with a ssd but I'm not sure what size ssd I should get to do this nothing on the drrive is important just games I would just like to speed up load times in games thanks in advance.

Dear Linus, I have an idea that you can show in your YouTube video. Personally I am a real fan of linux and I would really like it if you want to demonstrate this project. I came up with this idea when I saw your steam caching server and immediately thought why no part 2? This is the idea. You have access to a few gigabit network cards. What happens when you put these cards in bonding mode and the steam caching server runs on that server? will downloading go faster? Try to put two network cards in bonding mode on the Linux client. I recommend using bonding mode 0 (Round-Robin), which ensures that all traffic passes over 2 network cards. I'm curious if you put both the server and the client in bonding, the speed actually doubled Some URL's to the bonding guide Look at this post: Bonding on ubuntu to learn more about bonding on Ubuntu Linux Sorry about my bad English

Hi, everyone. yesterday, I bought an 16GB Intel Optane Memory module to speed up my 1tb Seagate barracuda boot drive. But I had a question that google failed to answer. How exactly does Optane work? Google told me that Optane speed up a hdd by caching recent or the most used files, but what does caching means? Does it means the often used hdd files were copied to the Optane stick, so when I open the file on the hdd, I open the identical copy on the Optane instead? Or does it means that the Optane and hdd are merged into one drive, and often used files are moved to the Optane section (like an sshd)? also, how do I install the driver software? Can I uninstall Optane after it is successfully installed?

I have been getting random freezes on my computer ever since I built my new computer. Computer freezes totally and only way that it can be turned off is by holding the power button for couple seconds. I narrowed the problem down to Ramcache II software which comes with Asus Rog motherboard. I turned off ramcache for a month and the computer stopped freezing completely and then I turned it back on and it has now crashed every single day since. I ran memtest and it says that my memory is working fine, so could this problem be SSD related or should I use some other software for ram caching? I have tried allocating 2048MB and 4096MB on ram cache, but it crashed with both settings. I really want to use ram caching since it speeds up significantly many programs that I use. My specs: ROG STRIX B350-F GAMING AMD Ryzen 5 1600 NVIDIA GeForce GTX 1060 6GB HyperX Predator DDR4 3200Mhz 2x8GB (16-18-18-36) 240GB KINGSTON SUV400S37240G (SSD) Seasonic Prime 650 W Gold

Title says it all. I have a 6TB WD Black series HDD that's showing a hair under 1TB capacity, RAID is not currently configured. I have an Intel 545s SSD that isn't even showing up at all in disk manager. The ULTIMATE goal is to have the full 6TB capacity showing and use 64GB of the Intel SSD's capacity for SSD caching using Intel Rapid Storage technology. Any idea on how to get this up and running? Note - I'm aware that some SATA ports are disabled when an M.2 is install in this mobo which is the case. Intel 545s 512GB SSD WD Black 6TB HDD Asus Maximus IX Formula Motherboard I look forward to your responses and thanks in advance!!

So im kinda getting into the idea of home servers and stuff like lancaching etc. I have a couple of 250gb ssds, and a couple of 500gb harddrives. I was thinking about splitting one ssd and use it as caching for 2hdds, how would one do that? Ive seen some softwares. But is there another way to do it for free? What do you guys suggest?

Hi all, I was watching the ULTIMATE Build a Better $1500 Gaming PC Computer "How To" Guide and was interested when Linus said that he planned to dedicate 60GB of the SSD as cache for the HDD to boost performance. I was wondering if a similar thing can be done by dedicating a portion of m.2 storage (say, from a 512GB Samsung 950 Pro) as cache to speed up a SATA SSD JBOD (such as four 2TB Samsung 850 EVO's). If the answer is yes, I have a couple more questions: What kind of read/write gains (roughly) might I expect from a setup like this? Is there a hardware limit to the amount of m.2 space I can allocate for the cache, and is there a performance "sweet spot" as far as this goes? I don't want to set aside 100GB if it won't be efficient. Is there a relationship between the size of the JBOD and the effectiveness of the cache, i.e. will having a larger volume detract from the effectiveness of the cache? Thanks so much!

Hi. I've coded a minimal c++ web server, and now I wanna start adding more things and make it as smooth and perfect as possible. I'm starting with caching. Now, I can send the typical stuff in the response headers that'll make the browser give me back either a "If-None-Match" if I'm doing ETags or a If-Modified-Since if I'm using dates. Now what I'm asking is which one would be more optimized? I honestly don't know how to make ETags an easy task, it could be really useful in long-term caching for pages that won't be changed in a long period of time, but is it really worth it? I mean I see no really simple way of doing, either store it in file comments (which would go against the purpose, given you'd have to check metadata for that), I could make a global variable, like an array or some other object to store every single page's ETag, which would make it really hard to update pages if you're trying to access them as you're developing or editing, because you'd either have to change that variable manually or make something to check whenever the file was modified. This would be nice for pages that only got an update every once in a while and was highly requested, but any other case would make it clunky. Not mentioning long term caching can't really be that useful, I mean what are the chances you're gonna keep a page cached in a browser for say 1 or 2 months? Second Option: Last-Modified + If-Modified-Since + Checking File Metadata for last modified date every time. Which doesn't seem too much of a compromise until you start getting a lot of requests. I don't know at what point this becomes not-worth it compared to ETags? I figure it's still way better than sending the whole file every time, but that's not even really an option so.. I could just go ahead and code the database program, which would be optimized and would be able to handle ETags the best way possible, but that can take a while and I want to understand this first. Maybe I could do it sort of like a hybrid, I'd config each page as ET or MD and handle it all as such. But I need someone's opinion. I'm unexperienced, I don't know until what point storage can be powerful in order to handle looking into metadata a lot of times or if I should make it more scalable by handling those ETags to memory, or whether that'd happen at all if I did. Thank you.

Recently I've become more and more interested in home networking and looking for projects I can try etc. I know only a small amount about proxy server caching and it has interested me. I have a Chromebox (NUC style PC) and a 320GB external hard drive connected with decent read/write speeds and my network is gigabit. (I know the Chromebox setup isn't ideal but it works) I have 40mbps down so speeds would be increased greatly I assume. I've heard about Squid on Linux but can't find much information on how to configure it for what I would like to do. How would I go about making a server for caching on my network to make previously visited websites and steam library updates quicker? (My brother and I both have large steam libraries) ps. I'm also interested in blocking ads, could this be done using Squid? Help appreciated.

Every time I turn on my desktop I let it sit on the desktop for a while. I usually then open 3 apps: SAMP, Groove Music, and Firefox. I've noticed that recently these apps seem to be opening faster and faster every time I use the PC, and just now I opened all 3 of them with literally no delay between releasing my hand off the mouse and the program appearing on screen. Groove Music opened (with no blue loading screen) before the start menu had time to animate out . I know that the 5400RPM Samsung Spinpoint(Seagate) drive in my desktop isn't that fast (Maxes out at 65MB/s, Compared to the 7200RPM drives in my server maxing out at 110MB/s), so I'm kinda confused at why these apps are launching so fast. My college friend's system with a 3GHz Xeon(don't know model number), a 120GB Toshiba SSD and 32GB RAM, seems only marginally more responsive compared to my system. I disabled fast boot on his system since it most certainly doesn't make sense to use on an SSD. I have fast boot turned off and only have 4GB RAM (+ a 160GB drive used for pagefile until I can afford more ram), boot hdd is 2% fragmented. I also partitioned off the first 115GB of my boot hdd so I can easily upgrade to a 120GB SSD in the future. Does Windows 10 have some kind of predictive memory caching? (Linux has memory caching but it's not predictive, the kernel only caches a file in memory after it's been explicitly opened)

Do you need to change some settings in bios before caching using Intel Smart Response?

Hey! First Off i want to let you know how my internet setup works like for now at least okey, My fiber to media converter ---> comes the wan (cat 5) 25(up) 75(down) ----> then it goes into my Untangle firewall router server (has it static ip config) ----> then goes to my Tplink modem with open WiFi with custom firmware (openwrt) ---> then another cat 5 cable from that modem goes into my pc. So, that's my setup, i have a lot of people using my internet all the time we have 3 windows 10 pcs and some android devices and ios and just becacuse the open WiFi which i am doing coz i like to makes my internet download the same updates time to time so, that wasted my bandwidth which could be used by other people for other downloads. one of my friends told me that i can setup a web caching server which makes it so if someone downloads a specific file and another person on the same network ties to download it it downloads from the local hard drive i would do anything for the web cache setup i need it really bad i am kindof a noob into all this i would lov to know how to set this up thanks!