General Intel HEDT Xeon/i7 Discussion

[Camo5]

Thanks for the update you guys are very informative

[bimmerman]

So, it's been a while since Project BadDecisions was noodled on, but I've made some progress. Also bought some shiny bits (GPUs), some more shiny bits (EK, Heatkiller bits), and even more (socket 2011 2P Supermicro 4 gpu board, cpus, ram, cooler...)

 

Anyway. On to the update!

 

Since the SR2 uses a server chipset with some fuckery, it DOES support ECC. However, getting more than 4GB/slot to work has historically been an enormous pain in the ass. However....most posts on the subject on the internet are circa 2010-2014 era, and things have changed. Why would I need more than 2 CPU x 6 slot/CPU x 4GB/slot = 48 GB total? Well, for those who don't know/remember, the point of this project is to run 4 gaming VMs simultaneously, and the rule of thumb at present is to run 16GB per player, plus some for overhead for the hypervisor. That math....doesn't work with only 48GB total system memory. The best I had configured was 11 GB / player with 1 GB per VM allocated to the hypervisor. It technically works, especially for games like Left 4 Dead (which, realistically, is what this'll be used for primarily in my friend group).

 

So, because the other part of this project is to really push the SR-2 to the limit, I figured why not try for the elusive 96GB. Going over 48GB requires (according to internet, I can't prove/disprove) ECC memory, and supposedly specific kinds of it.

 

I....yolo'd it and bought a Supermicro motherboard with CPUs and 128GB of 1333 DDR3 ECC RAM for less than the price of just 96 GB ECC on ebay. Upside, I have another test bed if the SR2 just fails and I need more cores, since that SM board will tackle up to 12 cores/socket and has 4x x16 3.0 slots at double spacing....so it's technically the better way to go. If yall remember the Supermicro board I had bought before, this is the same thing just for socket 2011 (x79 era stuff). Anyway, that's boring, and can't overclock, and was intended to be a way to get ram.

 

So, I popped out 12 sticks of the ECC goodness and got tinkering. I can't speak to how much of a pain this would be with the earlier production stuff (think X5650, 70s, etc), but the internet says those are much more challenging, especially ES/QS versions, to get this to work. YMMV. For me, with X5675s, this was stupid easy. Step the first: put in 6 sticks, 3 per CPU, and then manually set the timings, speed, and voltage in BIOS. The key here is to manually set the Command Rate to 2. Then power off, install sticks, and BOOM 96 GB. The only critical thing is that you get dual rank 8GB modules (2Rx4). Results are iffy on 16GB or other rank combinations, despite the CPUs' compatibility with 16GB dimms and other ranks...in other motherboards. The SR-2 is a weirdo.

 

 

It went shockingly smoothly, once I typed in the correct timing in the correct spot. Clearing CMOS on this thing is a pain in the ass, especially since not all settings get saved in profiles. But hey, here's the proof! 96 GB, running nice and happy at 1333 MHz, at the timings the memory manufacturer's data sheet specified (Samsung, in this case).

 

 

Eagle eyed readers will see the next part of the update.

 

Previously I had been using rando GPUs I and friends had lying around. The plan all along has been to run 4 GPUs, but here comes SR-2 pain in the ass part the second-- due to the janky way the chipset on this era stuff is laid out, the maximum SATA speed is 3GB/s, and there is almost no USB3 on the board....and what there is, is sharing bandwidth with the singular SATA 3 (6GB/s) controller, all of which is set behind a singular PCIe 2.0 lane (5GB/s bandwidth). In simple terms, for intense IO, routing through the cipset will bottleneck. Intense IO may be a stretch for four simultaneous gaming VMs, but maybe not-- 4+ SSD drives plus USB connectivity. Simpler math says that'll suck if I can't figure out a solution....which I did: I need to run a PCIe USB controller card, with 4x controllers so that I can pass each controller to separate VMs (enabling hot plug and downstream hubs and things, which is nice), plus an HBA PCIe card to bypass the chipset entirely for my SSDs.

 

So, that's pretty sweet! One problem...the SR2, despite being a 10 slot form factor, only has 7 slots, and GPUs are dual slot. Well shit.

 

Oh, wait, reference design 1080ti can be water blocked to be single slot. And I already have one....

 

Hmmmmmmmmm

 

Yep. Right when the internet was losing its mind over the 3080's performance, prior to realizing you can't actually buy them, I scooped up 3 additional 1080ti that also came with water blocks. Looking back, 20x0 series may have been smarter to buy, as they include a USB controller on the damn thing, but that wouldn't have solved the HBA issue-- if you can use chipset storage, that's the way to go IMO). Anyway, while I'm waiting on radiators to arrive....and free time... I lent one of the 1080Ti to a buddy, borrowed his 1080 in exchange, and threw this together (my main rig is keeping its 1080Ti until it gets upgraded). Right now I have the following setup: 1080TI in slots 1 and 2, HBA card slot 3, 1080ti slots 4 and 5, USB3 controller card slot 6, 1080 non-ti slot 7/8. Since I don't have water cooling fully done yet, this spacing lets the cards breathe without requiring me to tear down my other rig, which is nice.

 

Pr0nz:

 

 

Lastly, before calling it for the evening, I ran a set of Cinebench R15 and R20 for @CommanderAlex's comparison sake (spoiler....10980XE cleans house)

 

 

 

There's no way to spin this. The performance of stock X58 era stuff....is not great. I'll do some more thorough benchmarking in games (and VMs) before I really start tweaking knobs with overclocking, but....while my X5675 scaled Cinebench perf pretty much linearly with OC percent (~148 single core @ 4.5ghz in R15), that's still not great.

 

Next update will likely be a while, but next step is to play with OC as-is before ordering all the water cooling parts, and do a dry (ha!) run with friends to see whether performance of this era stuff is really going to work as gaming VMs where each player is running 2c/4t. Next update may be some performance figures and/or a for sale post-- I don't think overclocking the system is going to make up for only having 2c/4t per player and all the latency from the PCIe multiplexer chips due to the janky-ass PCIe layout (from a virtualization perspective, it's fine in normal use). If that's the case, I'm heavily leaning towards selling it all as a unit to someone who really wants to tinker with the SR2 rather than trying to do some super niche virtualization thing like I am....it's really just not the right set up for my use case. If interested, shoot me a PM haha.

 

I know what my old X5675 would do at its 4.5 GHz limit, which was honestly really good performance in modern-ish titles at higher resolutions, so there's a chance this works. It's really just a question of will it work at 2c/4t (at...likely closer to 4/4.2) AND will the unraid hypervisor be able to run things with 2c/CPU allocated. More details on getting the SR2 working with virtualization in a follow up post, because it's super non-trivial due to the aforementioned NF200 PCIe multiplexer layout. Seriously, to the best of my internet sleuthing, only one guy has done this before and published how he did it, and it's not a small job to work around the multiplexers. But it CAN be done, and I've gotten it to work already in testing, so more details to come.

 

Finally, as I'm writing this and running FAH, the 2x stock X5675s, 2x 1080ti, and 1x 1080 apparently pull enough juice to make my UPS scream bloody murder and trip the overload protection. Fcking Awesome.

[Pasi123]

Cinebench R23 on a X5670 @ 4.4GHz (5464 pts)

 

X5677 4c/8t @ stock (2996 pts)

[bimmerman]

Whelp.

 

I've been tinkering further with the SR-2.

 

My board and my 96GB Ram do not get along above 145 BCLK. It will post with 48GB just fine, but 96 isn't happening no matter what I've tried, which is a bummer but not the end of the world.

 

However, this kinda is: I wanted to get a baseline of 2 core + 4 thread performance with GPU before I go much further with this platform. The SR-2 has physical jumpers that let you disable/enable CPU and memory banks, and in the BIOS you can tell it how many cores per socket you want it to use. I also set my display GPU to a PCIe slot that was knocked down to PCIe 2.0 x8-- because of the USB 3.0 card and LSi HBA for SSDs, at least two of the VMs will be running their 1080ti at PCIe 2.0 x8 (aka, 3.0 x4) speed. Combining the 2c/4t CPU, at +500mhz over stock, with a 2.0x8 GPU, is going to be the at-best upper bound for the VM configuration.

 

And....performance is not good. I'm getting 38 fps in Shadow of the Tomb Raider at 1080p High. On a 1080ti. With all 12 cores and 24 threads, at STOCK speeds, with a full bandwidth 1080ti, it was pulling down 90+. At 1080p Lowest setting, it poops out 42 fps. It's crazily CPU bottlenecked with 2c/4t, the benchmark tool says there is 0% GPU bound. With the same 2c/4t/8x setup, Doom (2016) is doing ~55-70 range consistently, which is a positive result.

 

So.....where does that leave the project? Realistically, it's game over. I can overclock it a bit more and get another 500MHz out of it without worrying about my total system power budget, but 500 MHz isn't going to magically double framerates. I also can't NOT run at least one GPU at x8 link speed because there are not enough USB expansion ports otherwise.

 

Long way of saying, the SR2 build is over-- my use case of 4x virtualized simultaneous gamers is just not feasible with this motherboard, and it's time to cut my losses and sell off the parts. With all 12c/24t it is a very entertaining platform, especially if you like overclocking and tweaking...I just can't justify keeping it when I have no real use for it.

 

So.

 

Who wants an SR-2 + ram + noctua coolers + CPUs + chipset watercooling parts? And case to fit it (900D)? Send PMs if interested. The board works, it overclocks, it's great....just not for the task I want to use it for. I'm likely going to also sell off my other supermicro 2p boards as well (X8DTG-QF bare, X9DRG-QF + 2x 8c Xeon + 128gb ram + coolers) and just move to a zen 3 threadripper or X299 setup.

[Slayer3032]

Well seeing everyone posting these R23 benchmarks all over the internet, I was curious what I'd get in R23 so I started off doing a R15 run and updating CPU-Z/HWInfo. But then X58 decided to do weird X58 things despite not having touched my bios in almost a year.

 

I noticed something very strange, my x26 turbo multiplier was being used for some reason even though this thing has ran at 180x25@1.425v for the last two years? I once had a "bug" where I scored 1089 very early on in trying to dial in a stable overclock where it boosted to 4.8ghz and made a great score activating the x26 multiplier but nothing I have ever done would "enable" it.

 

 

I guess I'm 4.68ghz on air now despite 181x25 being unstable, as well as 200x23 and other 4.6ghz overclocks I've tried in the past.

 

 

Everything seems mostly the same, the scores are normal(1040/137-139 in R15).

 

I'm curious if a recent Windows update did something or if a reboot will "fix it". HWInfo is also pretty sure my base clock swings from 176-183 while CPU-Z seems far more normal. I'm leaning towards "random bug" which will be fixed by a reboot. After a reboot, it still boosts to 4,680mhz and I haven't noticed any instability as of recent. I really wonder what this X5675 would do if I had a good board to put it in.

[Slayer3032]

Rebuilt my home server into a new case and oh man is this thing way too fancy. This would mark it hitting just about the $300 mark ignoring storage. The old case was getting noisier and noisier but it seems some of the noise was the Zotac GT1030 which manages to be at least twice as loud the Fractal 140mm case fans, Scythe 1200rpm fans and hard drives. I got the board to boot without the GPU so there's no reason to leave it in there.

 

The Define 7 is really an amazing case, I'm so used to Corsair cases and garbage where you have to use multiple people/limbs or both of your forearms and your body weight just to get the side panels on. The side panels are just so damn good. The Scout and my Antec 900 were the only cases I've built in where the side panels actually didn't fight you. Its so well thought out and if you buy a second hard drive cage, you can fit 13 3.5" drives into it without even breaking into the 2-3 extra locations where you can bolt drives to. The P6T isn't happy that I'm not giving the fan hub a PWM fan as everything is 3 pin, it still seems to give the hub pwm out and somehow this thing still doesn't break 50c forcing it to transcode enough movies to likely flood my gigabit lan. Idle temps pretty close the same but slightly higher, but it used to be fairly loud with mostly molex fans and nothing but some foam to dampen sound out the front of the case.

 

My complaints would be that Fractal still overtightens screws as reviewers suggested. The marketing is confusing saying it comes with 4 hdd trays in the accessory kit, picturing it with no trays in the hard drive cage but in fine text in a random image saying it comes with 6 and the cage is populated in box. Nobody seems to take a decent picture of the gray color, its really more of a lighter gray than it comes off in pictures if it's in a bright room. All the reviewers seem to review the black ones which like never goes on sale. I wish they included a few more free zipties. Oh and most of the parts also seem really sensitive to getting finger prints.

 

I spent too many hours messing with the case and trying out different things, so by the time I was done I just wanted to get it plugged in so I didn't really get a chance to take halfway decent pictures.

 

[the pudding]

Just snagged an asrock X79 extreme4-M for £120 on the gumtree machine, pretty chuffed with that, not sure what cpu I'm going to chuck at it, 3930K's are cheap now, 3960X's aren't super expensive either, though I could splurge on a 4930K too, not all too sure, though I will need to grab a different kit of ram so that I'm not stuck on 16gb lul

[the pudding]

On a roll lately it seems, just snagged a rampage 4 extreme with a 3930K and 4x8gb of ram for 180, not going to complain at that  

[CommanderAlex]

24 minutes ago, the pudding said:

On a roll lately it seems, just snagged a rampage 4 extreme with a 3930K and 4x8gb of ram for 180, not going to complain at that  

That's very reasonable for the price... Someone in my area was selling a 4960X with 4x8GB sticks and a old beast of a graphics card (R9 295 X2) for like $500. The board and CPU were cheaper than individual pricing. 

[the pudding]

17 minutes ago, CommanderAlex said:

That's very reasonable for the price... Someone in my area was selling a 4960X with 4x8GB sticks and a old beast of a graphics card (R9 295 X2) for like $500. The board and CPU were cheaper than individual pricing. 

Yeah, a rampage 4 extreme and a 3930K round these parts would cost about 200 quid by itself, so for those and the 32gb of ram for 190 shipped, I can't complain with that

[Skunch]

Alright y'all, THE TIME TO OVERCLOCK HAS COME.

 

I just got my be quiet! Dark Rock Pro 4, it is rated to dissipate 250 watts of heat. Probably overkill, but I want to use it should I decide to upgrade in the future.

 

My current set up:

Asus Rampage III Formula

i7-980X or Xeon X5690

Cooler Master Hyper 212 (loud AF, only dissipates about 140-150 watts)

24GB DDR3 1600mhz Kingston HyperX

ASUS GTX 1070

512GB Samsung 950 PRO (boot and applications/games)

1TB WD NVMe Sata SSD (fast storage for non-application files

2GB WD Green (ew) for long term storage, (movies, archives, etc.)

Asmedia 106x SATA/RAID Controller PCI-E 2X

M-Audio Audiophile 2496 PCI (this card has windows 10 native drivers and IT ROCKS)

Seasonic 750W 80+ Titanium PSU

 

My plan is to swap out the venerable but loud hyper 212 for the Dark Rock Pro 4, but I have reached an existential crisis:

Do I try to overclock the 980X with it's unlocked multiplier, or do I install the Xeon X5690?

 

I'm looking to set things up and forget about it until I'm ready to upgrade to a new platform, which option would y'all pick?

 

I've never overclocked before, so I will be asking for help with that as well. Hoping for at least 4Ghz all core.

 

Thanks!

[CommanderAlex]

 

12 minutes ago, Skunch said:

Alright y'all, THE TIME TO OVERCLOCK HAS COME.

 

I just got my be quiet! Dark Rock Pro 4, it is rated to dissipate 250 watts of heat. Probably overkill, but I want to use it should I decide to upgrade in the future.

 

My current set up:

Asus Rampage III Formula

i7-980X or Xeon X5690

Cooler Master Hyper 212 (loud AF, only dissipates about 140-150 watts)

24GB DDR3 1600mhz Kingston HyperX

ASUS GTX 1070

512GB Samsung 950 PRO (boot and applications/games)

1TB WD NVMe Sata SSD (fast storage for non-application files

2GB WD Green (ew) for long term storage, (movies, archives, etc.)

Asmedia 106x SATA/RAID Controller PCI-E 2X

M-Audio Audiophile 2496 PCI (this card has windows 10 native drivers and IT ROCKS)

Seasonic 750W 80+ Titanium PSU

 

My plan is to swap out the venerable but loud hyper 212 for the Dark Rock Pro 4, but I have reached an existential crisis:

Do I try to overclock the 980X with it's unlocked multiplier, or do I install the Xeon X5690?

 

I'm looking to set things up and forget about it until I'm ready to upgrade to a new platform, which option would y'all pick?

 

I've never overclocked before, so I will be asking for help with that as well. Hoping for at least 4Ghz all core.

 

Thanks!

 

~Snipped photos~

 

Happy to see some activity here haha, thought this thread died..

 

I'd go with the 980X and see how far you can go with it. 

 

Depends on the platform and budget and what you want to be doing with it, unless its a hobby. 

[Skunch]

9 minutes ago, SpiderMan said:

 

Happy to see some activity here haha, thought this thread died..

 

I'd go with the 980X and see how far you can go with it. 

 

Depends on the platform and budget and what you want to be doing with it, unless its a hobby. 

It's my main rig, i use it primarily for gaming. I'm looking for the most stable overclock possible, and I've read lots in this thread about people loving the Xeons. I know that's because they are cheap, but they are also higher quality silicon compared to desktop chips.

[CommanderAlex]

Just now, Skunch said:

It's my main rig, i use it primarily for gaming. I'm looking for the most stable overclock possible, and I've read lots in this thread about people loving the Xeons. I know that's because they are cheap, but they are also higher quality silicon compared to desktop chips.

While that is true with Xeons, you are just limited with bumping up the BLCK to achieve any overclock whatsoever on these chips since the multiplier is locked. 

[Skunch]

11 minutes ago, SpiderMan said:

While that is true with Xeons, you are just limited with bumping up the BLCK to achieve any overclock whatsoever on these chips since the multiplier is locked. 

sounds like for me (a beginner overclocker) the unlocked multiplier will make things a lot easier for me.

[CommanderAlex]

1 minute ago, Skunch said:

sounds like for me (a beginner overclocker) the unlocked multiplier will make things a lot easier for me.

Yeah, I don't know anything about overclocking a Xeon. Only own a couple in cMPs. 

[Skunch]

3 minutes ago, SpiderMan said:

Yeah, I don't know anything about overclocking a Xeon. Only own a couple in cMPs. 

well then, sounds like I'm gonna stick with the 980X. if it gets cooked, I'll move on to the Xeon.

[Skunch]

this thing is stupid huge I love it!

[Skunch]

and it's all back together!! i wanna give a shout out to the single voltage controlled noctua fan I found in my workshop.

 

all in all, I've got two be quiet! pure wings 2 fans pulling air in the front (pwm), a random 240mm molex powered fan on top that I think came with this thermaltake case, a noctua voltage controlled fan pushing air out, 2 80mm cooler master pentium 2 fans forcing air through the chipsets, and the dark rock pro 4, with a 120mm and 135mm silent wings fans.

 

you could say that my computer blows...

 

time to make sure it boots, and then it's time to OVERCLOCK.

[Skunch]

aaaaand it wont post now! fun!!!!

[Skunch]

how could I be so foolish? I should have known that I'm supposed to boot first without my pci sound card, then install it.

/s

 

lol 11 year old hardware

 

looks like I'm good to go. I checked the overclock thread, and couldn't find anything about the 980x. I'd love to contribute, any suggestions where I should start?

[Pasi123]

My newest HEDT system. I'm probably going to buy a Xeon E5-2690 8c/16t for this system, this is one of the older versions of Lenovo ThinkStation S30 which don't support Ivy Bridge.

I might also buy another 4x16GB 1600MHz kit to get 128GB RAM total which should be decent for Chrome.

 

[Skunch]

sounds dope!

Edited February 2, 2021 by Skunch
I'm dum

[Skunch]

3 hours ago, Pasi123 said:

My newest HEDT system. I'm probably going to buy a Xeon E5-2690 8c/16t for this system, this is one of the older versions of Lenovo ThinkStation S30 which don't support Ivy Bridge.

I might also buy another 4x16GB 1600MHz kit to get 128GB RAM total which should be decent for Chrome.

 

also that chrome burn, shots fired lol

[Skunch]

Welp, I looked up that old video of linus overclocking a 980x from 10 years ago and used that info.

 

4.138Ghz all core with 1.35v.

 

average temp while running cinebench R15 is 55 degrees C, one core barely gets up to 61 degrees C.

I get about 930 points!

 

 

I'm gonna try cinebench R20 now!

*edit*

1832 points! not bad for an 11 year old CPU!

Edited February 3, 2021 by Skunch
posterity