Introducing the DPU, Processing In Ram

[porina]

Quote

The idea behind In-Memory Processing, or ‘Processing In-Memory’, is that a number of those simple integer or floating point operations should be done while the memory is still in DRAM – no need to cart it over to the CPU, do the operation, and then send it back. If the data can stay there and be updated, this saves time and power without affecting the result. Alternatively, perhaps compute on the CPU can be reduced if results are sent back out to main memory and a final XOR is applied to the data in memory. That frees up the main CPU core to do other compute related things, or reduces the effective memory bandwidth should it be a limiting factor.

https://www.anandtech.com/show/14750/hot-chips-31-analysis-inmemory-processing-by-upmem

 

Where can we add more processing potential? Let the ram handle some of the simpler calculations. In this implementation, a Data Processing Unit (DPU) is one "core" per 64MB of ram capable of some functions, but wont replace your CPU any time soon. Potential benefits include saving some ram bandwidth usage, and reducing the energy needed to do those processing tasks it can be used for. Designers claim it does not require much more manufacturing cost or power over standard ram. It does require a recode so it isn't a drop in and run enhancement. So initially it will likely be targeted by applications specifically (re)coded to take advantage of it. Maybe future generations will get more advanced and generic enough it could reach into consumer level gear. 

[sazrocks]

Interesting. Sounds like something that will be picked up in enterprise and super computer stuff for a while before we get our hands on it though.

[Mira Yurizaki]

So who tells the DPU what to do?

 

EDIT: Also what about security concerns? I don't want RAM touching memory just because some other location said to touch it if it isn't supposed to.

[thorhammerz]

9 minutes ago, Mira Yurizaki said:

I don't want RAM touching memory

[Mira Yurizaki]

3 minutes ago, thorhammerz said:

[Dabombinable]

So mining on a HDD isn't that much of a stretch after all...

[rcmaehl]

So basically... a math co-processor, separate from the CPU.

Where have I heard this before?

[porina]

31 minutes ago, rcmaehl said:

So basically... a math co-processor, separate from the CPU.

Where have I heard this before?

Co-processors aren't new, even today you could see a GPU as a co-pro to the CPU. The "new" thing here is sticking it inside the ram. The operations themselves aren't remarkable, but by working in ram you cut out a lot of the transfers between ram and CPU. 

[TechyBen]

What kind of useful calculations though? If the CPU has enough bandwidth, could you compress/encrypt on the fly? Though I'm not sure if it's worth lifting that off 1 CPU core...

[porina]

1 minute ago, TechyBen said:

What kind of useful calculations though? If the CPU has enough bandwidth, could you compress/encrypt on the fly? Though I'm not sure if it's worth lifting that off 1 CPU core...

You'll have to look up the source article for a more detailed list, but there is a variety of operations. A major point of this is there isn't enough bandwidth to go around. There hasn't been for many years and it is getting worse, not better, as cores increase far faster than ram bandwidth. Ram bottlenecks are a major limiting factor in many compute heavy scenarios. Anything you can do to offload some of that can help, with the promise of doing more work, less moving data around, at lower power.

 

For sure, it will be extremely niche in use, so don't expect to see it in consumer level devices any time soon.

[Doobeedoo]

Hm interesting, if it can offload cycles from CPU without any issues good. 

[Franck]

15 hours ago, Mira Yurizaki said:

So who tells the DPU what to do?

Yes, i am very curious. I am guessing it does something like this :

 

- CPU send compute instruction to the RAM

- RAM answer the location of the result

- CPU send 2 INT in 1 cycle

- CPU read memory address.

 

I am unsure how are they syncing so the CPU knows when the result is computed. Also i assume not all computation is sent to the RAM either. Must be some sort of sharing. Very interesting subject. And the document talk of 500 mhz only and that's still pretty good for the performance they are getting. I guess the first iteration of this people will run like 2 stick of standard ram paired with 2 stick of this special RAM.

[porina]

4 minutes ago, Franck said:

Yes, i am very curious. I am guessing it does something like this :

 

- CPU send compute instruction to the RAM

- RAM answer the location of the result

- CPU send 2 INT in 1 cycle

- CPU read memory address.

Look at it the other way around. Data lives in ram. In conventional usage, it has to go to the CPU, get worked on, then returned to the ram. If you can do some of the work locally in ram, you cut the CPU out. The processing in ram still has to be controlled and synchronised by the CPU, but the processed data doesn't need to immediately go to CPU when done.

 

4 minutes ago, Franck said:

Also i assume not all computation is sent to the RAM either.

The computations available to the DPU are relatively limited compared to CPU, so will only be a subset.

 

4 minutes ago, Franck said:

Must be some sort of sharing.

It requires re-coding so I assume it is managed in software to ensure things happen in the right order.

 

4 minutes ago, Franck said:

I guess the first iteration of this people will run like 2 stick of standard ram paired with 2 stick of this special RAM.

These will probably go in dedicated servers/workstations running specific software, and ram will be the same type throughout. 

[Mira Yurizaki]

I'm guessing UPMEM uses the side channels that DIMM slots have to issue commands to the DPU, since obviously there's not going to be an update to x86 to include instructions that could do something like "store, then modify"

 

Still, there's a question of security since memory the CPU is supposed to own can be modified outside of the CPU's control.

[Ezzy-525]

16 hours ago, thorhammerz said:

 

[porina]

7 minutes ago, Mira Yurizaki said:

Still, there's a question of security since memory the CPU is supposed to own can be modified outside of the CPU's control.

Is it possible to write to ram without going through the CPU? If you have to go through CPU, then by definition it is under CPU control.

[Mira Yurizaki]

2 minutes ago, porina said:

Is it possible to write to ram without going through the CPU? If you have to go through CPU, then by definition it is under CPU control.

It may still go through the CPU, but that doesn't mean the thing ultimately working on the data can't be a rogue agent that tells the CPU everything is fine when in fact, everything is on fire.

[thorhammerz]

1 minute ago, porina said:

Is it possible to write to ram without going through the CPU? If you have to go through CPU, then by definition it is under CPU control.

By definition, something has to dictate what gets written where, and what data is operated on when (and where). If not the CPU itself, then a co or embedded processor/controller somewhere along the way.

[porina]

1 minute ago, Mira Yurizaki said:

It may still go through the CPU, but that doesn't mean the thing ultimately working on the data can't be a rogue agent that tells the CPU everything is fine when in fact, everything is on fire.

To my understanding, these DPUs don't have any persistent storage, or even an OS. If you maintain security on your CPU, things should remain in a consistent state. 

[Mira Yurizaki]

Just now, porina said:

To my understanding, these DPUs don't have any persistent storage, or even an OS. If you maintain security on your CPU, things should remain in a consistent state. 

The DPUs still have to have a program to run. So unless it's baked in ROM and mathematically verified to not have issues, you can still have a rogue agent.

[porina]

2 minutes ago, Mira Yurizaki said:

The DPUs still have to have a program to run. So unless it's baked in ROM and mathematically verified to not have issues, you can still have a rogue agent.

Maybe I'm missing something. The CPU has to give those instructions to the DPU. So if the CPU is in good state, the DPU should be in good state. If you get malware on CPU, all bets are off.

[Franck]

4 minutes ago, Mira Yurizaki said:

The DPUs still have to have a program to run. So unless it's baked in ROM and mathematically verified to not have issues, you can still have a rogue agent.

I am assuming that programming wise we will have specific instruction set for these. So i guess that if security is a risk we will be the one's managing what can compute there or not. We will make sure to not do critical stuff there IF there is any issue security wise.

[TechyBen]

Wait. Just realised... AMD were offering complete bit level (IIRC) encryption in ram... so these chips would have to have the access codes and also use them, or be run on systems without it.

Also adds even more problems if running software/services in VMs, as I could see cross talk/access similar to Spectre or RowHammer going full on nuclear on some tech like this. XD

[Mira Yurizaki]

2 minutes ago, porina said:

Maybe I'm missing something. The CPU has to give those instructions to the DPU. So if the CPU is in good state, the DPU should be in good state. If you get malware on CPU, all bets are off.

The DPU has to have something to interpret those instructions. It's very unlikely that whatever it is that interprets those instructions is anything low level like how a CPU does it. Instead it would interpret commands via firmware or something. If there's any access to the portion in how the DPU responds to those instructions in a manner that it can be overwritten, then there's an attack vector in the DPU.

 

It's the same thing with Intel's IME. It lives in the chipset which you would think is commanded by the CPU. But because the chipset (and the IME by extension) is an independent processing unit that handles data outside of the CPU's bounds, it can be compromised to no longer be trustworthy with whatever the CPU sends it.

[porina]

3 minutes ago, TechyBen said:

Wait. Just realised... AMD were offering complete bit level (IIRC) encryption in ram... so these chips would have to have the access codes and also use them, or be run on systems without it.

Also adds even more problems if running software/services in VMs, as I could see cross talk/access similar to Spectre or RowHammer going full on nuclear on some tech like this. XD

In the first instance I think this may be applied to wholly owned servers since the initial use cases are more focused. If it gains sufficient popularity then maybe it gets expanded into cloud servers but they have more time to work that bit out.

 

2 minutes ago, Mira Yurizaki said:

It's the same thing with Intel's IME. It lives in the chipset which you would think is commanded by the CPU. But because the chipset (and the IME by extension) is an independent processing unit that handles data outside of the CPU's bounds, it can be compromised to no longer be trustworthy with whatever the CPU sends it.

I think we're in general agreement but the argument hinges on if there is a way to access the DPU without going through the CPU, which was an earlier question of mine. IME can be accessed externally to CPU (there were network based attacks previously I think) hence it provides another exposed point that can be attacked.