Samsung's New HBM2 Memory Thinks for Itself: 1.2 TFLOPS of Embedded Processing Power

[Lightwreather]

Summary

Today, Samsung announced that its new HBM2-based memory has an integrated AI processor that can push out (up to) 1.2 TFLOPS of embedded computing power, allowing the memory chip itself to perform operations that are usually reserved for CPUs, GPUs, ASICs, or FPGAs. 

Quotes

Quote

Today, Samsung announced that its new HBM2-based memory has an integrated AI processor that can push out (up to) 1.2 TFLOPS of embedded computing power, allowing the memory chip itself to perform operations that are usually reserved for CPUs, GPUs, ASICs, or FPGAs. 

The new HBM-PIM (processing-in-memory) chips inject an AI engine inside each memory bank, thus offloading processing operations to the HBM itself. The new class of memory is designed to alleviate the burden of moving data between memory and processors, which is often more expensive in terms of power consumption and time than the actual compute operations.

Samsung says that, when applied to its existing HBM2 Aquabolt memory, the tech can deliver twice the system performance while reducing energy consumption by more than 70%. The company also claims that the new memory doesn't require any software or hardware changes (including to the memory controllers), thus enabling a faster time to market for early adopters. 

Samsung says the memory is already under trials in AI accelerators with leading AI solutions providers. The company expects all validations to be completed in the first half of this year, marking a speedy path to market. 

Each memory bank has an embedded Programmable Computing Unit (PCU) that runs at 300 MHz. This unit is controlled via conventional memory commands from the host to enable in-DRAM processing, and it can execute various FP16 computations. The memory can also operate in either standard mode, meaning it operates as normal HBM2, or in FIM mode for in-memory data processing.

Naturally, making room for the PCU units reduces memory capacity — each PCU-equipped memory die has half the capacity (4Gb) per die compared to a standard 8Gb HBM2 die. To help defray that issue, Samsung employs 6GB stacks by combining four 4Gb die with PCUs with four 8Gb dies without PCUs (as opposed to an 8GB stack with normal HBM2). 

Notably, the paper and slides above refer to the tech as Function-In Memory DRAM (FIMDRAM), but that was an internal codename for the technology that now carries the HBM-PIM brand name. Samsung's examples are based on a 20nm prototype chip that achieves 2.4 Gbps of throughput per pin without increasing power consumption.

The paper describes the underlying tech as "Function-In Memory DRAM (FIMDRAM) that integrates a 16-wide single-instruction multiple-data engine within the memory banks and that exploits bank-level parallelism to provide 4× higher processing bandwidth than an off-chip memory solution. Second, we show techniques that do not require any modification to conventional memory controllers and their command protocols, which make FIMDRAM more practical for quick industry adoption."

Unfortunately, we won't see these capabilities in the latest gaming GPUs, at least for now. Samsung notes that the new memory is destined to satisfy large-scale processing requirements in data centers, HPC systems, and AI-enabled mobile applications. 

 

My thoughts

Well, this is interesting. Memory that can increase the performance of a system without increasing power consumption is an amazing thing, This will be a great thing for, especially, Data center applications where power consumption is an important factor and if (when) this eventually will make it's way to other consumer stuff, I think it would be great. But as with any new technology, we'll have to see how it holds up in the real world, its cost-effectivness and its usefulness.

Sources

https://www.tomshardware.com/news/samsung-hbm2-hbm-pim-memory-tflops

[Drama Lama]

that's what I'd call real " smart memory"

[Dabombinable]

50 minutes ago, Drama Lama said:

that's what I'd call real " smart memory"

And exactly what AMD has been needing for their GPU.

[leadeater]

That's really cool, would like to know what operations and data size it actually can process though. Not all things are equal in this area so that is kind of important.

[Doobeedoo]

I wonder when HBM will get back to consumer cards.

[Drama Lama]

1 hour ago, Doobeedoo said:

I wonder when HBM will get back to consumer cards.

only when it gives a lot more performance than gddr or is a lot cheaper

[Orangeator]

1 hour ago, Drama Lama said:

only when it gives a lot more performance than gddr or is a lot cheaper

The former is much more likely to occur than the latter... As I'd imagine the R&D for something like smart processing inside the memory module was not cheap. 

[Fnige]

Dont we already have HBM2 memory? Wouldnt this just be HBM2 2 memory then?

[bcredeur97]

i can see this going mainstream and then being exploited and it being a huge incident. Lol

[Avcn42]

3 hours ago, SlimyPython said:

Dont we already have HBM2 memory? Wouldnt this just be HBM2 2 memory then?

I read it as HBM choo choo and I'm not even ashamed 

[Doobeedoo]

11 hours ago, Drama Lama said:

only when it gives a lot more performance than gddr or is a lot cheaper

It definitely will give a big improvement in performance, but yeah price wise maybe for flagships.

[leadeater]

57 minutes ago, gabrielcarvfer said:

Each PCU seems to be one or a couple of GPU stream processors based on DOI 10.1109/EPEPS47316.2019.193209 .

 

Samsung made a deal to use AMD graphics in the same year the paper got published, so it wouldn't surprise me if this is where it ended up being used.

Probably won't happen but it would be interesting if RT and/or Image Upscaling gets done on HBM2 and the GPU core is left to do everything else.

[Bombastinator]

Sounds like a really awful place to get malware.

[leadeater]

12 minutes ago, Bombastinator said:

Sounds like a really awful place to get malware.

Or the perfect place for firmware level crypto mining malware/hijacking lol

[Shafengrtech]

On 2/17/2021 at 2:43 AM, leadeater said:

That's really cool, would like to know what operations and data size it actually can process though. Not all things are equal in this area so that is kind of important.

I found this presentation interesting. Check it out. It goes a little bit into the research. 
 

 

[Bombastinator]

My understanding is anything that is Turing complete can run literally anything.  It may not do it well or easily but it can do it.  There is talk of a “cpu” attached to the memory.  Can it be used to run arbitrary things? 

[ARikozuM]

Well, damn... Can it turn all of the NPCs into tig biddy onee-sans?? PLEASE LET ME HAVE THIS ONE THING!

[Bombastinator]

2 hours ago, ARikozuM said:

Well, damn... Can it turn all of the NPCs into tig biddy onee-sans?? PLEASE LET ME HAVE THIS ONE THING!

No.  It’s bad for you.  And the ones-sans backs will hurt.

[WaikikiSailor]

They are basically going back to old school hardware design; no fancy software. The Central Navigation Computer in Trident Submarines was designed this way; 1950s technology. It utilized about 900 circuit cards grouped according instruction set; each set controlled by its own processor. Commands were entered in binary, and/or hexadecimal, via a row of push buttons on the front panel and data was accessed via a companion Magnetic reel-to-reel Tape Unit. 
 

Simplistic design but it allowed for accurate and instantaneous processing of instructions (and very fast repair by quickly replacing affected circuit card)  to launch Trident nuclear missiles. It seems Samsung has hired an old school engineer and is bringing back simple direct path hardware communication concepts and waving the “look what we thought of” banner. 
 

Could be wrong. JMTC 

[williamcll]

But can it run doom?

[RejZoR]

On 2/17/2021 at 11:04 AM, Dabombinable said:

And exactly what AMD has been needing for their GPU.

Not really. AMD proved that with smart design you don't need the very latest memory that's running on the edge like GDDR6X. They've achieved this by using larger GPU level cache (Infinity Cache). They've done similar things in the past like the Ring Bus memory on Radeon X1800 series. Imagination did similar in the past with Kyro graphic cards doing tiled based rendering and Z buffer compression which was huge thing back in the day when no one else was doing it and Z buffer compression became a thing years later on Radeon 7000 as well as tile based rendering happening as late as with GeForce GTX 900 series.

[Dabombinable]

1 minute ago, RejZoR said:

Not really. AMD proved that with smart design you don't need the very latest memory that's running on the edge like GDDR6X. They've achieved this by using larger GPU level cache (Infinity Cache). They've done similar things in the past like the Ring Bus memory on Radeon X1800 series. Imagination did similar in the past with Kyro graphic cards doing tiled based rendering and Z buffer compression which was huge thing back in the day when no one else was doing it and Z buffer compression became a thing years later on Radeon 7000 as well as tile based rendering happening as late as with GeForce GTX 900 series.

Considering even now vRAM overclocking gets the best results. Much like my old graphics cards (some on a 64bit bus, some with lower-than-official vRAM clocks)

[RejZoR]

2 minutes ago, Dabombinable said:

Considering even now vRAM overclocking gets the best results. Much like my old graphics cards (some on a 64bit bus, some with lower-than-official vRAM clocks)

Overclocking RAM these days is very hard because there aren't any visible cues on problems. You can literally set your memory to 50GHz and it'll just work. But you'll have worse performance, so you need to actually benchmark memory overclock and decide on how high clock solely by observed gains or loses in memory performance. Which usually means painful observing of minimalistic framerate changes in repeated benchmarks. Unless you know a test that is incredibly memory bound and would display larger changes more vividly. I was using 3DMark Port Royal test and changes were very subtle in framerate, at least at resolutions I used.

[straight_stewie]

On 2/17/2021 at 4:43 AM, leadeater said:

That's really cool, would like to know what operations and data size it actually can process though. Not all things are equal in this area so that is kind of important.

Since it's characterized in TFLOPs and not TOPs, and it's advertised as being an "AI-accelerator" of sorts, I'd venture to guess that it will wind up being 32 bit floating point multiply-accumulate operations. Likely, to achieve that much performance, you will have to be doing fused multiply-adds.

Vector processors are pretty neat, and can increase performance in some applications.

Still, one wonders how an OS might expose an in-memory processor for general use. That will have to be solved pretty neatly before it becomes useful.

[Mling]

insert homer drooling meme