Epyc Naples Warning

[jmcguire525]

@leadeater I found a quote from you a while back suggesting to stay away from Eypc Naples... 

Quote

However I'd still advise against first gen EPYC because of that memory locality problem with one of the dies.

 

Can you further explain this?  I'm planning a build using a 7401p and would like to know the full extent of the issue before finalizing my plans.

[HQuan]

EPYC CPU? It's good, why are that guy saying like that?

EPYC CPU is fricking good, I have used it before!

[Somerandomtechyboi]

If you wanna know more about epyc then look around in sth since those guys dabble around with these epyc cpus, even overclocking them

 

 

Naples is kinda trash, sure you can overclock it but itll still get decimated by zen2 epyc

 

Id suggest getting an epyc rome instead, if you wanna overclock get one that ends in 05, iirc its oem or something if it ends in 05

 

Though at 200$ its def a good deal if you need that many cores, though unless you wanna get a supermicro the boards are really expensive so that can kill the deal

 

For something with many cores and decent freq i7 7960x 16 core are going for ~360$ on ebay, though boards are ~200$ instead of 360$ for the cheapest supermicro boards you can get for sp3

 

Also even though 7960x is old, 14nm has been rather stagnant anyways so the performance will still be decent

 

 

Its either a crap ton of cores or alot of cores but they are also pretty fast, though x299 is hedt and sp3 is server so server stuff youll get extra things like remote system access or whatever

 

Though youll prob need watercooling and also delid ihs and swap tim for liquid metal if you want any sort of decent performance on a 7960x, cause swapping tim on those cpus seem to yeild around 15-20c lower temps looking at youtubers that delid these cpus

[leadeater]

5 hours ago, jmcguire525 said:

Can you further explain this?  I'm planning a build using a 7401p and would like to know the full extent of the issue before finalizing my plans.

Depends what you are actually going to use it for. EPYC 7001 series is 4 SoC dies on a single package and each one of those has it's own memory controller on it so each SoC die is locally connected to only 2 channels of memory and the other 6 are remote and have to be accessed through the other SoC dies.

 

Whether or not this is a problem is completely down to the application and use case. They are still great CPUs but they can be quite bad for certain things and CPUs with much less cores can easily outperform them.

 

5 hours ago, HQuan said:

EPYC CPU? It's good, why are that guy saying like that?

EPYC CPU is fricking good, I have used it before!

Because the first gen is terrible for VM hosts (for VMs with more than 8 vCPUs assigned) and many database workloads. There are many used Xeons that will easily outperform greater core count EPYC first gen CPUs, never just blindly buy and always check actual performance.

 

First gen is nothing like second and third gen where there are those memory locality and inter die bandwidth problems, at least not nearly as much.

 

 

[leadeater]

3 hours ago, Somerandomtechyboi said:

Id suggest getting an epyc rome instead, if you wanna overclock get one that ends in 05, iirc its oem or something if it ends in 05

 

Though at 200$ its def a good deal if you need that many cores, though unless you wanna get a supermicro the boards are really expensive so that can kill the deal

@jmcguire525FYI for both of you AMD EPYC CPUs vendor lock on first time boot to the system vendor it is installed in to, so if the CPU was originally in an HPE server then you can only ever use it in an HPE server/motherboard. There isn't a way to bypass that (without bios hex editing and vendor code faking or w/e, just guessing because I'd never try it).

 

So if you buy a used EPYC CPU always ask what system it came from, I think @Windows7gehas already been burned by this? Someone around here has anyway.

[Somerandomtechyboi]

20 minutes ago, leadeater said:

@jmcguire525FYI for both of you AMD EPYC CPUs vendor lock on first time boot to the system vendor it is installed in to, so if the CPU was originally in an HPE server then you can only ever use it in an HPE server/motherboard. There isn't a way to bypass that (without bios hex editing and vendor code faking or w/e, just guessing because I'd never try it).

 

So if you buy a used EPYC CPU always ask what system it came from, I think @Windows7gehas already been burned by this? Someone around here has anyway.

I think thats on epyc rome and newer, naples for all i know doesnt seem to have this vendor lock, and also seems like the only one that even uses vendor lock is dell

[leadeater]

6 minutes ago, Somerandomtechyboi said:

I think thats on epyc rome and newer, naples for all i know doesnt seem to have this vendor lock, and also seems like the only one that even uses vendor lock is dell

No first gen EPYC 7001 can also be vendor locked. If one is lucky then supermicro would be the only major vendor to not be doing it and I wouldn't count on them not doing it, HPE/Lenovo/Dell etc all are.

 

Have a read through this: 

 

[Somerandomtechyboi]

5 minutes ago, leadeater said:

No first gen EPYC 7001 can also be vendor locked. If one is lucky then supermicro would be the only major vendor to not be doing it and I wouldn't count on them not doing it, HPE/Lenovo/Dell etc all are.

 

Have a read through this: 

 

Naples already sucks with single core, now just gotta add this extra pain in the ass and youve got a flaming pile of garbage

 

Unless i can get an 05 aka oem epyc rome which is not vendor locked then i will never ever consider epyc, unless i need ludicrous amounts of cpu power, and yes if i ever get my hands on epyc i will f around with trying to overclock it cause haha 64 core server proccessor go brrrrrrrrrrrr

 

 

Welp now i just gotta hope that in the next few years x299 boards will fall to rock bottom prices if i decide to skip 1366

[Windows7ge]

34 minutes ago, leadeater said:

So if you buy a used EPYC CPU always ask what system it came from, I think @Windows7gehas already been burned by this? Someone around here has anyway.

Yeah that was me. Got it off eBay. I don't think the seller knew what he was selling though because even though I explicitly asked if it was vendor locked before buying what showed up was vendor locked.

 

Then after a return ship and refund they updated the listing to reflect that it was indeed vendor locked.

 

Just bought another one though from a vendor that states it's not HPE/Dell vendor locked so here's hoping to better luck this time. If you're wondering what the warning signs are just expect no POST. If you have Debug 7 segment display for error codes don't expect the error code to reflect useful information.

[leadeater]

4 minutes ago, Windows7ge said:

Yeah that was me. Got it off eBay. I don't think the seller knew what he was selling though because even though I explicitly asked if it was vendor locked before buying what showed up was vendor locked.

 

Then after a return ship and refund they updated the listing to reflect that it was indeed vendor locked.

 

Just bought another one though from a vendor that states it's not HPE/Dell vendor locked so here's hoping to better luck this time. If you're wondering what the warning signs are just expect no POST. If you have Debug 7 segment display for error codes don't expect the error code to reflect useful information.

Well when our 7713 servers come out of service in the next 5 years or so I'll send you the CPU so you can look at them and cry, unless you source an HPE server to use them lol

[leadeater]

@Windows7ge Good news!

 

Quote

HPE does not use the same security technique that Dell is using for a BIOS hardware root of trust. HPE does not burn, fuse, or permanently store our public key into AMD processors which ship with our products. HPE uses a unique approach to authenticate our BIOS and BMC firmware: HPE fuses our hardware – or silicon – root of trust into our own BMC silicon to ensure only authenticated firmware is executed.  Thus, while we implement a hardware root of trust for our BIOS and BMC firmware, the processors that ship with our servers are not locked to our platforms. (Source: HPE)

 

[jmcguire525]

1 hour ago, leadeater said:

EPYC 7001 series is 4 SoC dies on a single package and each one of those has it's own memory controller on it so each SoC die is locally connected to only 2 channels of memory and the other 6 are remote and have to be accessed through the other SoC dies.

Ok, that makes sense.  I actually went away from the idea of doing a dual Xeon build for a similar reason.  I'll have to read up on it more but it sounds like it is very similar to a dual cpu board and how each are sharing memory bandwidth.

For VM's with 8 or fewer vCPUs there should be no issue as long as the proper cores are selected (up to 4 physical per die)?  How would the system know to assign the proper channels of memory if the only option is to select the amount?  (Even more complicated is the fact I was looking at the ROME6U-2L2T which only has 6 channels)

[leadeater]

17 minutes ago, jmcguire525 said:

Ok, that makes sense.  I actually went away from the idea of doing a dual Xeon build for a similar reason.  I'll have to read up on it more but it sounds like it is very similar to a dual cpu board and how each are sharing memory bandwidth.

For dual socket systems this is actually much less of a problem, it's pretty rare to have an issue. I have but that was from people violating VM sizing best practices and assigning more vCPUs to a VM that are on a single socket so the VM was NUMA node spanning which is not good almost always.

 

17 minutes ago, jmcguire525 said:

For VM's with 8 or fewer vCPUs there should be no issue as long as the proper cores are selected (up to 4 physical per die)?  How would the system know to assign the proper channels of memory if the only option is to select the amount?  (Even more complicated is the fact I was looking at the ROME6U-2L2T which only has 6 channels)

A CPU and system has a NUMA domain and IOMMU map that the OS uses to assign threads to cores so for something like ESXi it knows which cores belong to which socket and also down to which sub-NUMA domain which is each SoC die in a 7001 CPU. So if you create a VM with 8 vCPUs ESXi will always try and assign those threads to 8 physical cores that live in a single die. All hypervisors will do this. Bare metal OS installs will also utilize this information too to correctly assign threads to CPU cores.

 

EPYC 2nd and 3rd gen also actually have sub-NUMA domains in the IOD and you can selectively configure how and whether or not to present this to the running OS. This is because internally to the IOD the memory controllers are zoned grouped controlling two DDR channels, it's not 1 large memory controller it's actually 4 of them. NPS4 mode means 4 sub-NUMA domains presented to the OS, NPS2 means 2 sub-NUMA domains presented to the OS and NPS1 means 1 sub-NUMA domain presented to the OS. This is per socket btw, so on our AMD 7713 servers were can have up to 8 NUMA nodes showing in the OS depending on how we have configured the BIOS.

[jmcguire525]

21 minutes ago, leadeater said:

All hypervisors will do this. Bare metal OS installs will also utilize this information too to correctly assign threads to CPU cores.

I understand that, what trips me up is how the memory itself would be assigned so that the VM could interact ONLY with the 2 channels locally connected.

[leadeater]

44 minutes ago, jmcguire525 said:

I understand that, what trips me up is how the memory itself would be assigned so that the VM could interact ONLY with the 2 channels locally connected.

Because each memory region is owned by a NUMA domain/node. NUMA is actually about memory not CPU cores it just so happens that in x86 servers these are directly related however the number of cores per domain can scale independently of memory capacity.

 

By scheduling a VM CPU thread to a CPU core you are  by nature assigning it to a NUMA domain/node because a core must belong to one, since the hypervisor knows what CPU cores belong to each NUMA domain/node it'll pick physical cores that belong to the same one on a per VM basis.

 

However it's not specifically limited to acting this way, if the CPU cores are too busy it may have to schedule some of the virtual cores on a remote core to the other ones. The other situation is if you assign too much ram to the VM, say you have 128GB of ram with 64GB on each socket and you give a single VM 65GB of memory. This would mean 64GB of memory would reside in Node 0 and 1GB would reside in Node 1 (the other CPU) however if the number of cores is fewer than the single socket they will all be scheduled on Node 0 and remote memory calls across the CPU QPI link to the other CPU will be done for any memory access in the 1GB region of memory that resides in Node 1.

 

So the key is really that NUMA is about memory not CPU cores so you break down how much memory belongs to each NUMA domain/node and that is your size that you should not exceed for a single VM. This guidance most applies to EPYC 1st Gen and Intel multi socket systems, EPYC 2nd & 3rd Gen are fine to span sub-NUMA domains/nodes within a single socket but like Intel multi socket systems don't do it across sockets.

 

Here is a NUMA map of a dual 7713 server running Linux

node 0 cpus: 0 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 128 129 130 131 132 133 134 135 136 137 138 139 140 141 142 143 144 145 146 147 148 149 150 151 152 153 154 155 156 157 158 159 160 161 162 163 164 165 166 167 168 169 170 171 172 173 174 175 176 177 178 179 180 181 182 183 184 185 186 187 188 189 190 191
node 0 size: 515803 MB
node 0 free: 497848 MB
node 1 cpus: 64 65 66 67 68 69 70 71 72 73 74 75 76 77 78 79 80 81 82 83 84 85 86 87 88 89 90 91 92 93 94 95 96 97 98 99 100 101 102 103 104 105 106 107 108 109 110 111 112 113 114 115 116 117 118 119 120 121 122 123 124 125 126 127 192 193 194 195 196 197 198 199 200 201 202 203 204 205 206 207 208 209 210 211 212 213 214 215 216 217 218 219 220 221 222 223 224 225 226 227 228 229 230 231 232 233 234 235 236 237 238 239 240 241 242 243 244 245 246 247 248 249 250 251 252 253 254 255
node 1 size: 516037 MB
node 1 free: 504818 MB
node distances:
node   0   1
  0:  10  32
  1:  32  10

 

[Windows7ge]

11 hours ago, leadeater said:

Depends what you are actually going to use it for. EPYC 7001 series is 4 SoC dies on a single package and each one of those has it's own memory controller on it so each SoC die is locally connected to only 2 channels of memory and the other 6 are remote and have to be accessed through the other SoC dies.

 

Because the first gen is terrible for VM hosts (for VMs with more than 8 vCPUs assigned) and many database workloads.

What if I set one up using ZFS V2.0 FreeBSD 13.0 and passed 20 SSDs to it?

 

So far the behavior is weird. Network transfers peg the CPU 100%. Didn't do that native on Linux...

 

10 hours ago, leadeater said:

Well when our 7713 servers come out of service in the next 5 years or so I'll send you the CPU so you can look at them and cry, unless you source an HPE server to use them lol

If you seriously did that I would frame it on my wall to admire.

 

I've been told HPE has some rather dubious practices with their servers that make it so you can't even get BIOS updates without a HPE account or Service Plan so that's a real turn off...

 

9 hours ago, leadeater said:

Good news!

I would put it in one of my servers without hesitation. Heck. Might even turn that into my main hypervisor server if the memory issues are fixed. Add 0.5TB of RAM and I'll only ever need one server to run absolutely everything.

 

8 hours ago, leadeater said:

Here is a NUMA map of a dual 7713 server running Linux

[leadeater]

3 hours ago, Windows7ge said:

I've been told HPE has some rather dubious practices with their servers that make it so you can't even get BIOS updates without a HPE account or Service Plan so that's a real turn off...

Drivers and firmware are all free access, you need an account for the Service Pack for ProLiant (SPP) ISO which is every single firmware and driver all one one ISO. You also need an account for the System ROM firmware but to be honest you basically don't have to update that anyway, these are most always CPU microcode updates which you can do other ways if you reallly want them.

 

Also it's not like it isn't possible to acquire these 

[Windows7ge]

2 minutes ago, leadeater said:

Drivers and firmware are all free access, you need an account for the Service Pack for ProLiant (SPP) ISO which is every single firmware and driver all one one ISO. You also need an account for the System ROM firmware but to be honest you basically don't have to update that anyway.

 

Also it's not like it isn't possible to acquire these 

That's alright. I prefer the standard platform approach anyhow. ASRock Rack & Supermicro (if I can't avoid it).

 

Speaking of. I'm also eyeing up the Xeon Platinum 8176. A pair of them wouldn't make a bad VM server upgrade I don't think.

[jmcguire525]

10 hours ago, Windows7ge said:

What if I set one up using ZFS V2.0 FreeBSD 13.0 and passed 20 SSDs to it?

 

So far the behavior is weird. Network transfers peg the CPU 100%. Didn't do that native on Linux...

Have you tried Truenas Core (Freenas) to see if they have any optimizations that solve the issue?  I plan on using Truenas Scale with HDDs and an all SSD pool for VM's and Docker.  Just snagged a 7401p for $105 tha isn't vendor locked (seller just upgraded his personal machine).  Couldn't pass that up and will upgrade to Rome or Milan when the prices drop.  Going to use a ROMED6U-2L2T board which is confirmed to support Naples with v1.0 bios, so it will be ready when I decide to upgrade.

[Windows7ge]

13 hours ago, jmcguire525 said:

Have you tried Truenas Core (Freenas) to see if they have any optimizations that solve the issue?  I plan on using Truenas Scale with HDDs and an all SSD pool for VM's and Docker.  Just snagged a 7401p for $105 tha isn't vendor locked (seller just upgraded his personal machine).  Couldn't pass that up and will upgrade to Rome or Milan when the prices drop.  Going to use a ROMED6U-2L2T board which is confirmed to support Naples with v1.0 bios, so it will be ready when I decide to upgrade.

I opted for FreeBSD 13.0 because of my innate fascination with CLI based management. I could test TrueNAS CORE I used to run FreeNAS years ago and it worked fine but I'd rather take to the FreeBSD forums and ask about how to optimize the system myself.

 

$105 for a 24-core? Nice. Although all of the aforementioned arguments about Naples performance are valid it does come down to what you're looking to do & your budget. Naples is a cheap way of getting massive I/O. 64 PCI_e lanes. If you have a highly parallel workload that cam be distributed across multiple chiplets and your RAM is physically configured in the correct slots it's a fine platform to have fun with. Plus since it's a CPU limitation you can just drop a ROME chip in down the road when prices are a little more reasonable.

[jmcguire525]

1 hour ago, Windows7ge said:

If you have a highly parallel workload that cam be distributed across multiple chiplets and your RAM is physically configured in the correct slots it's a fine platform to have fun with.

I'll be running a few VMs and various docker containers, but my main need for compute power is for Plex.  I want to do away with having separate libraries for 4K HDR and 1080p SDR where there are duplicates of each.  $105 spent on the 7401p is much less than the cost of storage going forward.  With Plex HDR to SDR Tone Mapping it should do the job.  Do you know how well that will be handled since the CPU cores will not be split the same way they would with VMs?

[Windows7ge]

10 minutes ago, jmcguire525 said:

I'll be running a few VMs and various docker containers, but my main need for compute power is for Plex.  I want to do away with having separate libraries for 4K HDR and 1080p SDR where there are duplicates of each.  $105 spent on the 7401p is much less than the cost of storage going forward.  With Plex HDR to SDR Tone Mapping it should do the job.  Do you know how well that will be handled since the CPU cores will not be split the same way they would with VMs?

I'm not very savvy with things related to PLEX but it would be my expectation that it would benefit from faster cores rather than more cores unless you have 20-odd simultaneous streams going all at once. I'd love to hear how it goes for you though and how many cores do show utilization for a single stream. I don't expect that many though.

 

EPYC is much better for a highly parallel workloads rather than singular tasks like PLEX. I like to use BOINC as an example. I already have a EPYC 7601 running PROXMOX and BOINC running World Community Grid in a LXC Container.

 

Native Linux vs Linux Container the performance is indiscernible for my workloads. There is overhead in VM's so avoid it if you can or pin CPU cores so everything stays on the same NUMA node. You can also investigate Huge Pages. These can increase the VM performance a fair bit.

[jmcguire525]

21 minutes ago, Windows7ge said:

I'm not very savvy with things related to PLEX but it would be my expectation that it would benefit from faster cores rather than more cores unless you have 20-odd simultaneous streams going all at once. I'd love to hear how it goes for you though and how many cores do show utilization for a single stream.

From what I've read, Plex transcoding is multi-threaded (except for VC1 video) one user said a single stream tops out at ~8 threads.  I regularly have 8-10 simultaneous streams going, so I expect it will work well for my use case