AMD's Datacenter and AI premiere - New Bergamo CPUs, MI300s and other stuff

[igormp]

Summary

AMD just had a conference announcing the release of some of their new products, such as new Epyc Bergamo with up to 128 Zen 4c cores and the MI300A APUs with up to 192GB of HBM3 memory.

 

Quotes

Quote

AMD is set to share a ton of new updates it has been working on since the last time we were in San Francisco in November 2022, and we are on the ground once again to keep you up to date with all the announcements.

We expect some exciting news about the development the company's data center portfolio, AI technology advancements and so much more, so stay tuned to our live blog below for all the latest updates...

 

 

My thoughts

Bergamo is a nice option for really dense racks that don't require much cache or excessive single-threaded performance, like most of webservers or virtualized environmets.

 

The MI300 lineup does look interesting, but aren't looking that impressive to me when compared against Nvidia's GH chip, with way more total memory and better software compatibility.

 

They also had some small talk about their DPUs and FPGAs, and also came with some of the creators of some ML frameworks (such as Pytorch and Huggingfaces_ to showcase their partnership to allow those kind of workloads to run on their hardware stack.

 

Sources

https://www.techradar.com/news/live/amd-data-center-and-ai-technology-premiere-2023-all-the-announcements-and-updates-live-from-san-francisco

 

 

[Athan Immortal]

37 minutes ago, igormp said:

The MI300 lineup does look interesting, but aren't looking that impressive to me when compared against Nvidia's GH chip, with way more total memory and better software compatibility.

This as I understand it is the main problem. So many deployments already on Cuda of some sort, or just ready for Nvidia. The savings that AMD offer put it in a weird spot where someone who has the money to buy these could probably stretch to the Nvidia option anyway and save retraining or reprogramming their workflows.

 

I think they may find a home in some new super computers built specifically with these in mind at huge scale where the saving would scale, but I don't know.

[igormp]

2 minutes ago, Athan Immortal said:

This as I understand it is the main problem. So many deployments already on Cuda of some sort, or just ready for Nvidia. The savings that AMD offer put it in a weird spot where someone who has the money to buy these could probably stretch to the Nvidia option anyway and save retraining or reprogramming their workflows.

 

I think they may find a home in some new super computers built specifically with these in mind at huge scale where the saving would scale, but I don't know.

I mean, they only showcased inference and not training, so you could go on with training on a regular Nvidia cluster and then using that saved model on an AMD platform to do inferences at lower TCO, but I don't think that's a viable scenario anyway.

 

For HPC stuff it seems to be already going on, see Frontier and Lumi.

[leadeater]

2 hours ago, igormp said:

For HPC stuff it seems to be already going on, see Frontier and Lumi.

I think or at least hope there will be decent flow on from these and general support and maturity will come out of that making regular market usage more viable. That's basically how it happened for Nvidia at the beginning. GPUs in servers wasn't an overnight and one generation affair, but AMD is WAY late to that party in a meaningful way,

[StDragon]

6 hours ago, leadeater said:

but AMD is WAY late to that party in a meaningful way.

Linus said it himself: there's no such thing as a bad product, only a bad price. While that's not universally true, it certainly is in this case. If AMD prices their offerings accordingly, that will take market share from Nvidia and will force a price war with cloud providers on a hardware buying spree. This could boost R&D further for AMD.

[igormp]

4 minutes ago, StDragon said:

Linus said it himself: there's no such thing as a bad product, only a bad price. While that's not universally true, it certainly is in this case. If AMD prices their offerings accordingly, that will take market share from Nvidia and will force a price war with cloud providers on a hardware buying spree. This could boost R&D further for AMD.

That doesn't apply to this case, because even if AMD were to price their products at a third of nvidia's, the developer cost to get stuff running on their platform wouldn't would far outweigh those savings.

[leadeater]

1 hour ago, StDragon said:

Linus said it himself: there's no such thing as a bad product, only a bad price. While that's not universally true, it certainly is in this case. If AMD prices their offerings accordingly, that will take market share from Nvidia and will force a price war with cloud providers on a hardware buying spree. This could boost R&D further for AMD.

I'm still reluctant to try them, they may be cheap but it's also not what any of our academics are asking for. And when we can get Nvidia cards at academic prices then well... the difference isn't so much between Nvidia and AMD.

[AlTech]

5 hours ago, igormp said:

That doesn't apply to this case, because even if AMD were to price their products at a third of nvidia's,

AMD would never go for that. Server and Workstation is where almost all the big bucks come from.

5 hours ago, igormp said:

the developer cost to get stuff running on their platform wouldn't would far outweigh those savings.

For some companies they don't care about the dev cost.

 

MLID has said Amazon doesn't care, they'll make their own software for the AMD MI Instinct products.

 

For everybody else that does care, AMD's working on the software ecosystem and has hired a ton of QA/testing people but the software ecosystem will take time to materialise.

 

Essentially they're gonna unify all 3 of their main software initiatives under one software package and branding: GPU Open, ROCm, and their other optimisations as well; to create an alternative to Cuda.

[leadeater]

5 minutes ago, AluminiumTech said:

For everybody else that does care, AMD's working on the software ecosystem and has hired a ton of QA/testing people but the software ecosystem will take time to materialise.

 

Essentially they're gonna unify all 3 of their main software initiatives under one software package and branding: GPU Open, ROCm, and their other optimisations as well; to create an alternative to Cuda.

Problem is that has been said before and under different older brand naming than ROCm. I only have hope this time because notable users in the top 500/top 10 supercomputer list are going to be using AMD Instinct. That gives me real hope unlike "we're trying" statements from AMD that have been said before.

 

ROCm is a 2016 era band and software stack, it's had a long time to go nowhere.

[igormp]

57 minutes ago, AluminiumTech said:

For some companies they don't care about the dev cost.

Such as...?

57 minutes ago, AluminiumTech said:

MLID has said Amazon doesn't care

Do you have any credible source?

59 minutes ago, AluminiumTech said:

they'll make their own software for the AMD MI Instinct products.

Once again, such as...? AWS only has instinct products for VDI. They do build custom software, but that's meant for their inferentia and graviton products, which lies on the other extreme of doing stuff in-house.

1 hour ago, AluminiumTech said:

For everybody else that does care, AMD's working on the software ecosystem and has hired a ton of QA/testing people but the software ecosystem will take time to materialise.

And that's great! But they're still far behind and not really useful not any ML workstation or even inference server as of today.

[AlTech]

Just now, igormp said:

Such as...?

Amazon and also Microsoft.

 

Microsoft did commit to working with AMD on a GPU Datacenter Partnership cos Microsoft don't want Nvidia to dominate AI and Datacenter.

Just now, igormp said:

Do you have any credible source?

MLId is a credible source.

Just now, igormp said:

Once again, such as...? AWS only has instinct products for VDI. They do build custom software, but that's meant for their inferentia and graviton products, which lies on the other extreme of doing stuff in-house.

I expect we'll see more MI cards in Azure servers in the future cos of the MS + AMD Partnership.

 

If AMD merges their software into 1 ecosystem then there'll be a lot more adoption because Nvidia can't keep up with demand for Grace Hopper and Hopper and the lead times are getting a bit silly.

Just now, igormp said:

And that's great! But they're still far behind and not really useful not any ML workstation or even inference server as of today.

MI Instinct cards are meant for inferencing (among other things) and they can he useful. It's just that their usefulness can be outweighed by the developer effort required to support both Cuda and ROCm/AMD's solutions.

[igormp]

20 minutes ago, AluminiumTech said:

Amazon and also Microsoft.

Microsoft is full on Nvidia for anything related to AI.

20 minutes ago, AluminiumTech said:

Microsoft did commit to working with AMD on a GPU Datacenter Partnership cos Microsoft don't want Nvidia to dominate AI and Datacenter.

During the presentation in the thread both AWS and Azure have only shown CPU related stuff.

In this scenario is makes no sense to offer AMD GPUs. I would not rent a machine with an AMD GPU to have headaches training my models.

22 minutes ago, AluminiumTech said:

I expect we'll see more MI cards in Azure servers in the future cos of the MS + AMD Partnership.

 

Not in the near future, but I hope this eventually happens, although I have more hopes for Intel.

23 minutes ago, AluminiumTech said:

If AMD merges their software into 1 ecosystem then there'll be a lot more adoption because Nvidia can't keep up with demand for Grace Hopper and Hopper and the lead times are getting a bit silly.

That makes no sense. ROCm is their compute ecosystem and it sucks.

Lead times for Nvidia's x100 products has always been long, and those are long because people really want those and are not really looking into AMD's lineup.

25 minutes ago, AluminiumTech said:

MI Instinct cards are meant for inferencing (among other things) and they can he useful. It's just that their usefulness can be outweighed by the developer effort required to support both Cuda and ROCm/AMD's solutions.

As someone who has tried to work with it, it sucks ass, it's awful to even get running. Don't belive me? Even Geohot gave up on them for his new startup:

https://news.ycombinator.com/item?id=36189705

[leadeater]

4 minutes ago, igormp said:

That makes no sense. ROCm is their compute ecosystem and it sucks.

Intel's oneAPI supports any compute device so could well end up using that for AMD GPUs if Intel kicks off that in a big way soon.

[igormp]

3 minutes ago, leadeater said:

Intel's oneAPI supports any compute device so could well end up using that for AMD GPUs if Intel kicks off that in a big way soon.

That's only in theory. In practice intel's extensions to tensorflow and pytorch only work with their own GPUs afaik, I haven't seen any reports of those also working with other GPUs or accelerators.

[AlTech]

13 minutes ago, igormp said:

Microsoft is full on Nvidia for anything related to AI.

During the presentation in the thread both AWS and Azure have only shown CPU related stuff.

In this scenario is makes no sense to offer AMD GPUs. I would not rent a machine with an AMD GPU to have headaches training my models.

Not in the near future, but I hope this eventually happens, although I have more hopes for Intel.

That makes no sense. ROCm is their compute ecosystem and it sucks.

Well no because their ecosystem is split up and fractured. That's one of the main problems right now.

13 minutes ago, igormp said:

Lead times for Nvidia's x100 products has always been long, and those are long because people really want those and are not really looking into AMD's lineup.

Some of those companies would buy AMD if their software ecosystem was unified.

 

Even more companies would buy AMD if AMD presented an alternative to Cuda that isn't OpenCL.

 

AMD's HIP solution (which works on both Nvidia and AMD GPUs) is supposed to be an alternative but I'm guessing not a lot of people know about it and it probably needs improving to be on par with Cuda.

 

OT: AMD's hardware though is great. It's just software holding them back.

13 minutes ago, igormp said:

As someone who has tried to work with it, it sucks ass, it's awful to even get running. Don't belive me? Even Geohot gave up on them for his new startup:

https://news.ycombinator.com/item?id=36189705

 

[leadeater]

25 minutes ago, igormp said:

That's only in theory. In practice intel's extensions to tensorflow and pytorch only work with their own GPUs afaik, I haven't seen any reports of those also working with other GPUs or accelerators.

Yea, I don't even know how mature it is for Intel hardware? Do you happen to know? I just know it exists and "can" support any.

 

Edit:

Oh and what I mean is I would bet on Intel oneAPI with AMD GPUs working before AMD ROCm lol

[igormp]

14 minutes ago, AluminiumTech said:

Well no because their ecosystem is split up and fractured. That's one of the main problems right now.

34 minutes ago, igormp said:

I don't think that's an issue at all. For compute all of their efforts are now centered around ROCm, which is also what was presented in this thread's presentation.

16 minutes ago, AluminiumTech said:

Some of those companies would buy AMD if their software ecosystem was unified.

 

Even more companies would buy AMD if AMD presented an alternative to Cuda that isn't OpenCL.

 

AMD's HIP solution (which works on both Nvidia and AMD GPUs) is supposed to be an alternative but I'm guessing not a lot of people know about it and it probably needs improving to be on par with Cuda.

 

OT: AMD's hardware though is great. It's just software holding them back.

Again, being unified isn't a problem, working and having good performance is the actual issue.

 

CUDA is an entire ecosystem, so the direct competitor to it is indeed ROCm. HIP would be the equivalent to CUDA as a C++ programming library, and converting CUDA code to HIP can be easily done in an afternoon, even for complex codebases. 

Most people in this industry are aware of HIP/ROCm, but it just doesn't work properly. I take that you may have no experience with it from what you're talking about.

 

Yeah, the hardware is great but is useless without a proper software ecosystem, and that's something that Nvidia has been putting a lot of money for almost 20 years now.

 

4 minutes ago, leadeater said:

Yea, I don't even know how mature it is for Intel hardware? Do you happen to know? I just know it exists and "can" support any.

 

Edit:

Oh and what I mean is I would bet on Intel oneAPI with AMD GPUs working before AMD ROCm lol

I wanted to buy an ARC GPU just to try some stuff out

But from what I've seen, support is already as good, if not (marginally) better than ROCm for consumer stuff. You can run tensorflow and pytorch stuff without many headaches, and getting it to run is WAY easier than ROCm (which is not impressive since ROCm basically has no official support for consumer GPUs).

 

Oh, right, got what you meant. Yeah, oneAPI is already close to ROCm in a way shorter timeframe. AMD just should help Intel at this point, but I doubt they'd do so given that they have their own FPGA companies competing in a similar space.

[porina]

9 minutes ago, leadeater said:

Oh and what I mean is I would bet on Intel oneAPI with AMD GPUs working before AMD ROCm lol

I haven't looked closely at it but I did find it amusing that Intel going open with OneAPI was more of an AMD-like move. Go open to try and get wider adoption, as a new closed system is going to be more challenging to get going. Is ROCm closed?

 

Intel do have a two prongs attack going on:

Transitioning CUDA code to Intel hardware: https://www.intel.com/content/www/us/en/docs/dpcpp-compatibility-tool/get-started-guide/2023-1/overview.html

Getting OneAPI working on nvidia hardware: https://codeplay.com/solutions/oneapi/for-cuda/

I see historic reports Intel intended to acquire Codeplay, but I haven't found anywhere that said it completed. It is still listed as a private company in the UK.

 

[igormp]

3 minutes ago, porina said:

Is ROCm closed?

It's not, but it's also not that welcoming to external contributors.

4 minutes ago, porina said:

I see historic reports Intel intended to acquire Codeplay, but I haven't found anywhere that said it completed. It is still listed as a private company in the UK.

AFAIK they did manage to do it.

They also got the ArrayFire team:

https://www.intel.com/content/www/us/en/developer/articles/news/arrayfire-oneapi-september2022.html

[Doobeedoo]

Neat way with c cores and how they can fit more. The size of PCB though, seems a lot of free space on surface, maybe to some degree but I take it a ton of wiring between dies.

[igormp]

2 hours ago, Doobeedoo said:

Neat way with c cores and how they can fit more. The size of PCB though, seems a lot of free space on surface, maybe to some degree but I take it a ton of wiring between dies.

They still have space for more cores, in theory they could have gone with up to 192 cores per socket, however, there's not enough space for all the wiring with the IOD:

Quote

However, Bergamo’s IO Die only connects to 8 CCDs vs 12 on Genoa, which brings the question: Could AMD have done a 12 CCD, 192-core Bergamo? Other than a much lower power budget and memory bandwidth per core, the silicon could theoretically support it. However, the package cannot.

 

The IO die has 12 Global Memory Interconnect 3 (GMI3) chiplet links, routed through the package substrate. In Genoa, the GMI3 wires for CCDs farther away from the IO Die are routed underneath the L3 cache area of the nearer CCDs. As it turns out, this is more difficult on Bergamo, as the Zen 4c CCD’s higher density means the wires must be routed under the smaller L3 of the nearer CCD using more layers.

https://www.semianalysis.com/p/zen-4c-amds-response-to-hyperscale

[DuckDodgers]

AMD EPYC 9754 CPU V-Ray Benchmarks Show Single Bergamo 42% Faster Than Dual Sapphire Rapids Chips

 

[StDragon]

12 hours ago, DuckDodgers said:

AMD EPYC 9754 CPU V-Ray Benchmarks Show Single Bergamo 42% Faster Than Dual Sapphire Rapids Chips

 

 

V-Ray has nothing to do with the same instruction sets that AI uses.

[igormp]

12 minutes ago, StDragon said:

V-Ray has nothing to do with the same instruction sets that AI uses.

How is that relevant to his post or the thread?

[leadeater]

1 hour ago, igormp said:

How is that relevant to his post or the thread?

Probably forgot the event was for new EPYC CPUs as well. We have mostly been talking about the GPUs.