NVIDIA DGX-2 Multi-Rackmount System Announced at GTC: Claimed as "The World's Largest GPU" according to NVIDIA

[Marinatall_Ironside]

https://www.anandtech.com/show/12587/nvidias-dgx2-sixteen-v100-gpus-30-tb-of-nvme-only-400k

https://overclock3d.net/news/gpu_displays/nvidia_reveals_world_s_largest_gpu_-_meet_the_dgx-2/1

https://www.nvidia.com/en-us/data-center/dgx-2/?ncid=pa-tra-d2lh-34943

 

Equipped with 16 NVIDIA Tesla V100 32GB HBM2 HPC Processors (512GB total, 81,920 CUDA Cores combined, and 10,240 Tensor Cores), two Intel Xeon Platinum 8180 28-core CPUs, 1.5TB of 6-channel DDR4 Memory, 30TB of NVM Express Storage, and 8 100-gigabit Ethernet or Infiniband ports for external connectivity, this monstrosity of a server-class computer can execute 2 PetaFLOPS of compute performance. To power this thing at full, 100% load, 10 kilowatts is required. Its power supplies are all redundant. The cost of one of these multi-rackmount systems runs you at a cool USD $399,999, and weighs a beastly 350 pounds.

Quote

 

NVIDIA DGX Series (with Volta)
	DGX-2	DGX-1
CPUs	2 x Intel Xeon Platinum	2 x Intel Xeon E5-2600 v4
GPUs	16 x NVIDIA Tesla V100 32GB HBM2	8 x NVIDIA Tesla V100 16 GB HBM2
System Memory	Up to 1.5 TB DDR4	Up to 0.5 TB DDR4
GPU Memory	512 GB HBM2 (16 x 32 GB)	128 GB HBM (8 x 16 GB)
Storage	30 TB NVMe Up to 60 TB	4 x 1.92 TB NVMe
Networking	8 x Infiniband or 8 x 100 GbE	4 x IB + 2 x 10 GbE
Power	10 kW	3.5 kW
Size	350 lbs	134 lbs
GPU Throughput	Tensor: 1920 TFLOPs FP16: 480 TFLOPs FP32: 240 TFLOPs FP64: 120 TFLOPs	Tensor: 960 TFLOPs FP16: 240 TFLOPs FP32: 120 TFLOPs FP64: 60 TFLOPs
Cost	$399,000	$149,000

From AnandTech

Quote

OC3D

Have you been wondering why Nvidia hasn't released a new consumer GPU in a while? Well, if you can sell a system with 16 Tesla V100s for $399K, it is understandable why Nvidia's focus has shifted to the enterprise market.  

Meet what Nvidia has called the "World's largest GPU", which aside from being a system that interconnects sixteen Tesla V100 32GB GPUs, offering a combined frame buffer or 512GB of HBM2 memory and 2 PFLOPS of total performance. 

 

In total Nvidia's DGX-2 has over 80,000 CUDA cores and 14.4 TB/s of combined/aggregate GPU memory bandwidth, though this level of GPU performance requires 10,000 Watts of power to run at full tilt.

This GPU cluster is powered by two Intel Platinum series 28-core Xeon processors with 1.5TB of DDR4 DRAM and 30TB of NVMe SSD storage, with 8x EDR 100 Gigabit Ethernet connections for external connectivity. When compared to the Nvidia Volta DGX-1, which was introduced in September 2017, the DGX-2 can offer up to a 10x performance uplift.    

 

How all 16 of these GPUs are connected is with a special interconnect called NV Switch (not to be confused with the Nintendo Switch). This interconnect lets all of the GPUs "talk" to each other with little latency, running at 50GB/s. This is also a twin system as well.

Quote

OC3D

One of the most exciting aspects of this system is Nvidia's new "NV Switch" (No, not the Nintendo console), which allows each of these GPUs to connect to each other with ports that run with up to 50GB/s of bandwidth. This interconnect will enable each GPU to speak to each other when required with minimal latency. 

 

AnandTech

NVIDIA’s overall topology relies on a dual stacked system. The high level concept photo provided indicates that there are actually 12 NVSwitches (216 ports) in the system in order to maximize the amount of bandwidth available between the GPUs. With 6 ports per Tesla V100 GPU, each running in the larger 32GB of HBM2 configuration, this means that the Teslas alone would be taking up 96 of those ports if NVIDIA has them fully wired up to maximize individual GPU bandwidth within the topology.

 

The previous NVIDIA DGX-1 computer has also been upgraded with the new 32GB NVIDIA Tesla V100's as well, for those with smaller pockets.

Quote

OC3D

Alongside the release of their DGX-2 system, Nvidia has also confirmed that they will be upgrading their DGX-1 with 32GB Volta GPUs, allowing these systems to handle larger datasets. 

 

How does it leverage all of that performance to attain the 2 PetaFLOPS figure? AnandTech has that answered.

Quote

AnandTech

DGX-2 builds upon DGX-1 in several ways. Firstly, it introduces NVIDIA’s new NVSwitch, enabling 300 GB/s chip-to-chip communication at 12 times the speed of PCIe. This, with NVLink2, enables sixteen GPUs to be grouped together in a single system, for a total bandwidth going beyond 14 TB/s. Add in a pair of Xeon CPUs, 1.5 TB of memory, and 30 TB of NVMe storage, and we get a system that consumes 10 kW, weighs 350 lbs, but offers easily double the performance of the DGX-1. NVIDIA likes to tout that this means it offers a total of ~2 PFLOPs of compute performance in a single system, when using the tensor cores.

 

This thing comes as an announcement from GTC, and considering just how powerful the DGX-2 is, it's rather valuable from a datacenter perspective, as it is the equivalent of two NVIDIA DGX-1 systems compressed and upgraded into a full system. Mind you it does take up multiple rackmount units to install, but it is enormously powerful. In the end, 2 PetaFLOPS of compute performance while taking up less space and requiring less power is a bonus when you compare it to multiple DGX-1's installed to attain similar compute performance. Although more expensive (over 2x the cost vs. one DGX-1), you can see the benefits, as you're getting way more performance, takes up less space, and has better performance per watt. This clearly targets the higher end datacenter market.

 

This kind of performance is going to be important in the future when AI and Deep Learning take off in areas such as Autonomous Driving, as that needs huge amounts of compute performance to allow the vehicles to learn and navigate their environments reliably, and accurately.

 

For other applications outside of autonomous driving and artificial intelligence, it is also excellent for medical research and other HPC applications as well. In medical research, simulating cells is demanding on compute performance to attain 100% accuracy reliably (because in the medical field, fuck-ups will not be tolerated!), and within reasonable times.

 

For a look at the guts of this beast, Gamers Nexus has a video on this thing.

 

 

 

Oh and here's the tweet from NVIDIA making the announcement!

 

Edited March 31, 2018 by JurunceNK
Added video, and removed duplicates.

[mikat]

HOOOOOOOOLY SHIT

[Notional]

Hmm interesting. Don't EPYC have more PCI lanes than XEON? Would that not benefit those 8-16 Tesla V100? 

[mikat]

1 minute ago, Notional said:

Hmm interesting. Don't EPYC have more PCI lanes than XEON? Would that not benefit those 8-16 Tesla V100? 

The tesla's mostly communicate with the other tesla's and they use NVlink for that, I doubt PCIe bandwidth is a bottleneck in deep learning 

(and EYPC has 128 lanes but two Xeon Platinum cpus also deliver 96 lanes so it shouldn't make that much of a difference)

[Notional]

2 minutes ago, mikat said:

The tesla's mostly communicate with the other tesla's and they use NVlink for that, I doubt PCIe bandwidth is a bottleneck in deep learning 

(and EYPC has 128 lanes but two Xeon Platinum cpus also deliver 96 lanes so it shouldn't make that much of a difference)

Good point. But isn't NVlink just interconnects between the GPU's? They still need to be fed data from the CPU's, as they don't have the advanced memory controller that Vega has.

[mikat]

It looks like it's a 6-8U server, it's massive

does anybody know how big it is?

[mikat]

Just now, Notional said:

Good point. But isn't NVlink just interconnects between the GPU's? They still need to be fed data from the CPU's, as they don't have the advanced memory controller that Vega has.

Yeah but I'm guessing the PCIe bandwidth is more than enough to keep 16 V100s fed with data

[Marinatall_Ironside]

2 minutes ago, Notional said:

Hmm interesting. Don't EPYC have more PCI lanes than XEON? Would that not benefit those 8-16 Tesla V100? 

Depends on usage case. If single-core performance along with multi-core performance is required, then it makes sense to go with Xeon Scalable over AMD EPYC.

 

On top of that, you have space considerations too. 16 RAM slots is already challenging to fit perfectly while being able to cram in the other components as dense as possible. Plus you have a power budget to consider too. 24 RAM sticks will consume less power than 32 sticks.

 

Each of those Tesla V100 cards uses 300 watts at full tilt. You're already at 4,800 watts total, and the rest needs to be fed to the other components too, such as the CPUs, chipset and PLX switches, RAM, networking cards, and all of the storage.

 

1 minute ago, mikat said:

It looks like it's a 6-8U server, it's massive

does anybody know how big it is?

If we consider that the DGX-1 is a 3U server, then we can divide the two GPU banks across 2 rackmount units each, so thats 4 rackmount units worth of GPUs (This is because the power supplies of the DGX-1 takes up 1 rackmount unit, and the GPU bank slides out the back), plus we have 1-3U rackmount units of CPUs and networking cards each. In front of it is the storage.

 

So in total, it's more than 8 rackmount units total. My estimate puts it at 10 rackmount units required to install.

[Marinatall_Ironside]

6 minutes ago, mikat said:

Yeah but I'm guessing the PCIe bandwidth is more than enough to keep 16 V100s fed with data

The bandwidth of the interconnect is more than enough. The PCIe bandwidth is used to assign data, and the interconnect takes over with communicating between GPUs.

[Orangeator]

So externally, what do you hook this beast up to? Like this doesn't just get plugged into any regular old wall outlet...

[Marinatall_Ironside]

2 minutes ago, Orangeator said:

So externally, what do you hook this beast up to? Like this doesn't just get plugged into any regular old wall outlet...

230-240V AC outlets depending on what country you're in. Here in the U.S., 240V is the maximum, and for homes, they're for appliances like washing machines.

 

This isn't for consumer use. It's for datacenter use, where they have the power requirements met so they don't trip any circuit breakers when they turn on one or more of these things and put some work on them.

Edited March 27, 2018 by JurunceNK

[RotoCoreOne]

Would like to see Linus purchase the dgx-1 and have some fun with it

[BuckGup]

Pshhh the bitcoin network is at 26 million TH/s which is like 1 Hash is about 13,000 flops and only consumes as much as a small city

[bcredeur97]

81,920 CUDA Cores sheeesh

 

 

This thing will do approx ~16k Sols/s on equihash(based on GTX 1080ti 0.19 Sols/s per cuda core) which amounts to about $370/Day (Zcash)

[ScratchCat]

But can it run Crysis while training multiple neural networks to play Crysis?

 

The improvement over time for training Neural Networks is probably one of the largest in the industry at 31.25X over 5 years or 2X per year.

Given the dedicated hardware (Tensor cores) in Nvidia GPUs it will not be long until they release a TPU ASIC like Google, if this occurs AMD would have completely lost the ML race and a very large source of income.

 

Interesting fact: The NVLink switch chip has more transistors (2 billion) than the GP108.

[Brooksie359]

1 hour ago, bcredeur97 said:

81,920 CUDA Cores sheeesh

 

 

This thing will do approx ~16k Sols/s on equihash(based on GTX 1080ti 0.19 Sols/s per cuda core) which amounts to about $370/Day (Zcash)

Yeah it would only take like 3 years to pay for itself without considering price fluctuation and power bill

[laminutederire]

2 hours ago, JurunceNK said:

This isn't for consumer use. It's for datacenter use, where they have the power requirements met so they don't trip any circuit breakers when they turn on one or more of these things and put some work on them.

It's more about compensating wall power from the grid irregularities. You don't want a micro electricity break on you line to shut down the entire server rack by accident.

[Marinatall_Ironside]

1 minute ago, laminutederire said:

It's more about compensating wall power from the grid irregularities. You don't want a micro electricity break on you line to shut down the entire server rack by accident.

That's why datacenters use battery backups as well, so in the event that happens, they can properly shut down the systems without damaging anything

[laminutederire]

Just now, JurunceNK said:

That's why datacenters use battery backups as well, so in the event that happens, they can properly shut down the systems without damaging anything

And minimizing downtime. Downtime can potentially cost a lot!

[Marinatall_Ironside]

Just now, laminutederire said:

And minimizing downtime. Downtime can potentially cost a lot!

And we're talking millions. IT departments in datacenters don't fuck around!

[Razor01]

4 minutes ago, JurunceNK said:

And we're talking millions. IT departments in datacenters don't fuck around!

 

Easily, the first time NBC streamed the Superbowl in 2012 we had a half hour down time lost tens of millions of dollars.

[laminutederire]

8 minutes ago, JurunceNK said:

And we're talking millions. IT departments in datacenters don't fuck around!

Well sometimes they do but someone is fired quite fast in those cases!

More on topic, I'd wish deep learning wouldn't take the turn it's taking. People want better hardware to train models instead of thinking of making better or more intelligent networks for more practical usage. Less YouTube AI (which doesn't work that well to be fair), and more AI in Computer graphics, for better voice assistants, or better AI for games.

[leadeater]

4 hours ago, Orangeator said:

So externally, what do you hook this beast up to? Like this doesn't just get plugged into any regular old wall outlet...

 

4 hours ago, JurunceNK said:

230-240V AC outlets depending on what country you're in. Here in the U.S., 240V is the maximum, and for homes, they're for appliances like washing machines.

 

This isn't for consumer use. It's for datacenter use, where they have the power requirements met so they don't trip any circuit breakers when they turn on one or more of these things and put some work on them.

It will have multiple 3 phase power supplies, won't be standard 240V.

 

These suckers.

 

Or it's 5 2kw single phase power supplies but I don't think so.

[leadeater]

2 hours ago, JurunceNK said:

That's why datacenters use battery backups as well, so in the event that happens, they can properly shut down the systems without damaging anything

We use generators with external diesel fuel tanks, no down time is required so you gotta do what ya gotta do. Not an uncommon setup though. Even with huge UPSs we only get 15 minutes on battery on normal load levels, getting more run time on battery is actually dangerous because lots of batteries = bad.

 

You have to constantly monitor large battery banks with thermal sensors and battery condition monitors checking things like internal resistance, when they first get installed you have to use a thermal camera while they are charging to watch out for hot spots.

 

Generators are just safer.

 

Edit:

4 hours ago, JurunceNK said:

So in total, it's more than 8 rackmount units total. My estimate puts it at 10 rackmount units required to install.

Looks like 7U to me.

[mr moose]

6 hours ago, Notional said:

Hmm interesting. Don't EPYC have more PCI lanes than XEON? Would that not benefit those 8-16 Tesla V100? 

I think more importantly Nvidia and Intel have some contracts/deals/agreements  going on with AI and deep learning, I can't imagine either of those wanting to support AMD even if Epyc was a better choice.