Kir_

I wish I had found this thread earlier. It has all the info I painstakingly gathered over days. I too have been looking into running LLMs on CPU for inference (on a budget), since it's much cheaper to add a bunch of RAM than GPUs. imo octa channel is where it starts to get interesting, since memory bandwidth seems to be the limiting factor. TR 3000 is obviously not a budget option. TR 1000/2000 is only quad channel. I have not yet found "cheap" used intel offerings. Epyc 7002 though offers octa channel, goes up to a theoretical 204.8 GB/s bandwidth and can be found on ebay at somewhat reasonable prices. An Epyc 7302 + Gigabyte MZ32-AR0 Motherboard can be had for ~400USD+customs. You'll need 3200MT/s ECC RAM for that, which isn't necessarily cheap, but maybe you could get away with slower single rank modules and overclocking. Taking a 70b Q4_K_M model at ~44GB RAM you should theoretically achieve 4.6 t/s, not sure how much less it would be in actuality. You'd need two 3090's to run this and context could eat up the remaining 4GB. Q5_K_M at ~51GB simply wouldn't fit. I'd consider 4t/s to still be acceptable. Not sure about time to first token. Now what I'm still wondering is, would using dual socket motherboard with 2x Epyc 7002 also double the bandwidth/can llama.cpp make use of it? In the end I'm not sure I want to go for it though. Adding in 8 sticks of 3200MT/s ECC RAM, cooler, case, psu etc. the "budget" machine quickly gets closer to 1k, which is a bit much for a project purely for fun.

Have you tried any stress testing like the AIDA64 stability test? What is your voltage set to and how high do your CPU temps go and is the Memory running at 3000MHz?

Alright, I rerun the tests. This time I manually set the voltage to 1.18 for all tests and set the CPU fan to always run at max. RAM was not fiddled with and ran at 2133MHz. At 3.2GHz 1.18v (according to HWMonitor max voltage was 1.175 and lowest was 1.119v) the temperature after 10 minutes of AIDA system stability test went up to 68°C and stayed there. At 3.7GHz 1.18v (according to HWMonitor max voltage was 1.175 and lowest was 1.112v) the temperature after 10 minutes of AIDA system stability test went up to 72°C and stayed there. I guess these results make more sense, although they aren't very positive. I've done this multiple times to make sure there are no weird results like last time and the results where always within +/- 1°C.

When I reset my bios settings, the CPU voltage was set to auto. I only noticed this once in Windows. I checked in hwmonitor what it was set to and it said 1.25. When I then went on to test the OC I figured it can't hurt to lower the voltage since I remembered that it runs stable at a lower voltage. My bad, I'll try again once I'm back home and set a fixed manual voltage to better compare. If it's not possible for the temps to be so different I really hope I just messed something up. Yeah, that is about how much I added. I watched tutorials before the last time just to make sure and was extra careful when installing the cooler again. Thanks for the help.

Alright, I have done a bunch of testing, took me quite some time. First, I reset my bios settings. The case fans were always running at max. I have a Nanoxia Deep Silence 3 case with 3 x 120 mm (max. 1300 RPM). Two in the front and one in the back. I disabled everything in the startup tab in the task manager and would let the PC just sit there for a few minutes with 0% (sometimes spikes to 1%) CPU activity. After that I did the AIDA64 system stability test. I did notice something weird, as I was unable to recreate the 41°C on idle at first. Here are my results. Stock Clocks, 1.25v (auto) Idle: 32°C, 1282RPM (CPU fan) Load: 61°C, 1880 RPM @3.7GHz, 1.18v Idle: 36°C, 1331 RPM Load: 71°C, 2626 RPM After this I reinstalled the cooler again and used a little less thermal paste since it seemed like I used a bit too much. The temperatures on idle dropped from 32°C to 30°C at stock and from 36°C to 33°C @3.7GHz. At load they dropped from 61 to 59°C at stock and from 71 to 69°C @3.7GHz. Temps seem a lot better, right? Still higher than what some other people report and my PC is anything but silent with every fan running at max at this point. This is when I set my RAM to run with 2996 instead of 2133 and suddenly all my temps increased. Stock *Either forgot to measure or misplaced. Can't redo right now* @3.7GHz, 1.18v Idle: 41°C, 1445 RPM Load: 79°, 2596 RPM (AIDA test failed too after 5 minutes with Warning: Hardware failure detected) I don't know what to think. It's a bit underwhelming. When you google temperatures for the 1700 OC the first result is this, which at this point sounds like it's too good to be true (51C peak @3.7GHz 1.3v) or I just got unlucky. <rant>3.7Ghz just seems to be the guaranteed OC for most people and I kind of doubt they do this at 80°C+ and a machine that sounds like a turbine.</rant>

I can't measure the room temp, but it's winter and probably nowhere close to 31°C. Isn't 70°C for stock really high? I'm using HWMonitor. Also tried HWInfo. Temps were the same. And yeah, I'm thankful for any input!

I did screw it in until it wouldn't go any further. Haha, yeah. I did plug in the fan cable and the cooler is very audibly spinning at full load.

Hello, I just installed a Ryzen 7 1700 with the Wraith Cooler on a Asus Crosshair VI Hero and I was a bit surprised by the temperatures. With an open case, I get 41°C on idle, which seems a bit high? When overclocking to 3.8GHz at 1.25v the temperature exceeds 83°C at full load in a closed case with case fans on maximum. This temperatures seem a bit high, especially when comparing to this list, although you can't tell what cooler people are using. It still seems a bit high, considering the relatively low voltage. <65° seems to be pretty common. Can a better cooler make that much of a difference or did I just get a bad chip? I have already reseated the cooler and also replaced the thermal paste with Artic Silver 5, but that didn't help.