NVIDIA DGX-2 Multi-Rackmount System Announced at GTC: Claimed as "The World's Largest GPU" according to NVIDIA

[leadeater]

3 minutes ago, mr moose said:

I think more importantly Nvidia and Intel have some contracts/deals/agreements  going on with AI and deep learning, I can't imagine either of those wanting to support AMD even if Epyc was a better choice. 

And I don't think Nvidia really wants to put AMD CPUs in one of their products anyway.

 

Up until now NVLink didn't make much sense to me when not used with IBM Power CPUs that also support NVLink. Originally NVLink was mesh between GPUs but could only link to 4 other end points (CPU or GPU) which means in the DGX-1 the GPUs are in groups of 4 and the other 4 are an extra hop away. Full PCIe x16 to every GPU in the DGX-1 would actually give more effective bandwdith to each GPU. NV Switch solves that, really makes NVLink much more useful.

[yian88]

Interesting.

Look at that number 240TFlops fp32 399k$ rig, remember it in 10 years when you are going to buy single GPU with 100TFlops for 1-2k$.

GTX 280 10 years ago 0.6TF,  today TITAN Xp  12-15 TF, improvement of 25x, in 10 years i would expect a modest 5-10x improvement.

[Salv8 (sam)]

me when seeing those specs:

 

[Marinatall_Ironside]

On 3/27/2018 at 5:02 PM, leadeater said:

 

It will have multiple 3 phase power supplies, won't be standard 240V.

 

-snip-

These suckers.

 

Or it's 5 2kw single phase power supplies but I don't think so.

Actually, from a video from Gamers Nexus, they use C19 plugs, and from the looks of it, 6 power supplies!

 

The C19 plug can handle 240V AC 16A (20A absolute max), and is grounded. It's primarily used in IT applications, but this connector can also be found on power supplies exceeding 1.5 kW total power output.

 

Edited March 31, 2018 by JurunceNK

[leadeater]

1 hour ago, JurunceNK said:

Actually, from a video from Gamers Nexus, they use C19 plugs, and from the looks of it, 6 power supplies!

 

The C19 plug can handle 240V AC 16A (20A absolute max), and is grounded. It's primarily used in IT applications, but this connector can also be found on power supplies exceeding 1.5 kW total power output.

Thanks for the video.

 

Well that's in a way kind of more annoying, most PDUs in racks are C16 not C19 and you tend to go from C16 straight to 3 phase. Bit of a legacy thing though since most server blade chassis used a ton of power and used 3 phase power so you either had C16 in a rack, 3 phase or both. Blades have really fallen from popularity though due to the move to ASIC compute and the PCIe those require.

 

C19 is always used on the input's to small/medium sized UPSs though.

 

Looking at the DGX-2 my guess is there are two power distributors, one for each side, each in 2+1 configuration. The power supplies will be around 3.5kw each which is enough to power the system from a single power distributor should one fail but you will lose redundancy, minimum of 3 online PSUs to power the system.

 

As for the in rack power it will be a split of 3 phase down to 3 single phase feeds each going to a PDU, you then plug each power supply from the same power distributor in to separate PDUs distributing the load across 3 different phases (that's how ours are done).

 

We're a bit behind the times in terms of having lots of ASIC powered servers so I guess the industry made the move to C19 and we never noticed, triple single phase is much easier to work with.

[Marinatall_Ironside]

2 hours ago, leadeater said:

Thanks for the video.

 

Well that's in a way kind of more annoying, most PDUs in racks are C16 not C19 and you tend to go from C16 straight to 3 phase. Bit of a legacy thing though since most server blade chassis used a ton of power and used 3 phase power so you either had C16 in a rack, 3 phase or both. Blades have really fallen from popularity though due to the move to ASIC compute and the PCIe those require.

 

C19 is always used on the input's to small/medium sized UPSs though.

 

Looking at the DGX-2 my guess is there are two power distributors, one for each side, each in 2+1 configuration. The power supplies will be around 3.5kw each which is enough to power the system from a single power distributor should one fail but you will lose redundancy, minimum of 3 online PSUs to power the system.

 

As for the in rack power it will be a split of 3 phase down to 3 single phase feeds each going to a PDU, you then plug each power supply from the same power distributor in to separate PDUs distributing the load across 3 different phases (that's how ours are done).

 

We're a bit behind the times in terms of having lots of ASIC powered servers so I guess the industry made the move to C19 and we never noticed, triple single phase is much easier to work with.

With PDU's, the most common receptacle is the NEMA 5-15 receptacle if the PDU is 120V AC. From what I can see according to Wikipedia, my guess places it at either NEMA 6-15 or NEMA 6-20, which can output up to 15 to 20 amps respectively before tripping breakers. This is factoring in IEC 60320 specifications for the C19 power receptacle (recommended 15 amps, max 20 amps before destroying the connector/equipment).

 

As for the redundant power supplies, up to two (in a 2+1 setup) of those can fail at any one point in the DGX-2 before it can no longer operate (either at 100% load, or can no longer power on). You can run them off of a strong-enough PDU, but if I was to be running my own datacenter, I would plug in three of the six into one PDU to reduce load on the PDU so as to prolong reliability, and it's within my best interest to hold onto those pieces of equipment for as long as I can.

 

And here's the image for what the actual power cord looks like for something like the NVIDIA DGX-2 for their six power supplies.

Edited March 31, 2018 by JurunceNK

[Arika]

still couldnt run PUBG at medium settings with 60fps.

 

goddamn those specs though, i don't know what the most impressive part is the power consumption, the size of the NVME storage, the total RAM, the networking or the price.

[leadeater]

1 hour ago, JurunceNK said:

With PDU's, the most common receptacle is the NEMA 5-15 receptacle if the PDU is 120V AC. From what I can see according to Wikipedia, my guess places it at either NEMA 6-15 or NEMA 6-20, which can output up to 15 to 20 amps respectively before tripping breakers. This is factoring in IEC 60320 specifications for the C19 power receptacle (recommended 15 amps, max 20 amps before destroying the connector/equipment).

 

As for the redundant power supplies, up to two (in a 2+1 setup) of those can fail at any one point in the DGX-2 before it can no longer operate (either at 100% load, or can no longer power on). You can run them off of a strong-enough PDU, but if I was to be running my own datacenter, I would plug in three of the six into one PDU to reduce load on the PDU so as to prolong reliability, and it's within my best interest to hold onto those pieces of equipment for as long as I can.

 

And here's the image for what the actual power cord looks like for something like the NVIDIA DGX-2 for their six power supplies.

Mostly depends on country, I'm a 240V country like a lot of places. Though from my understanding even in the US high power rated stuff is 240V anyway? Also was wrong it's C13/C14 not C15/C16, very similar just different heat rating.

 

As for the rack power 3 single phases is still the most likely. Two PDUs A + B is kind of fine for a single one of these units but not a full rack of them, you'll be exceeding what a single phase is allowed to deliver under electrical regulations. Although again having to base this off my local regulations.

 

PDUs are rated and fused for either 20A or 32A before going specialist 63A (3x C19, http://www.apc.com/shop/nz/en/products/Rack-PDU-Basic-0U-1U-220-240V-63A-3-C19/P-AP6038A).

 

You could fit 6 of these in a rack so would require 12 63A PDUs, in A + B that's 378A you have to account for where if you went with A + B + C the whole dynamic changes completely. Each PSU on each side of the chassis goes to the same phase, if a phase goes out two PSUs go offline leaving 4 online and shifting only 1/3 the load but across two phases so only 1/6 increase per phase.

 

You would still have 4 PDUs per phase though so 252A you have to account for but there's a huge difference in what's required for 250A than there is 380A.

 

Actual load wise it's 250A vs 125A, if that sounds confusing/wrong math you can only cover a single phase failure but more importantly in the event of a failure one configuration is having to redistribute 30kw and the other is only having to shift 20kw but only 10kw on a single phase.

 

Anyway these types of power is WAY higher than I deal with, we cap out at 10kw per rack so 60kw per rack is almost unthinkable lol.

 

[WolfDeville]

Might pick one of these up for the sole purpose of playing Fortnite

[Marinatall_Ironside]

2 hours ago, TehStranger said:

Might pick one of these up for the sole purpose of playing Fortnite

Good luck trying to get one , you'll need to fill in the field saying "company" before you can even blow the USD $400K on one of these things.

 

Plus the company in question needs to be real.

[Marinatall_Ironside]

13 hours ago, leadeater said:

Mostly depends on country, I'm a 240V country like a lot of places. Though from my understanding even in the US high power rated stuff is 240V anyway? Also was wrong it's C13/C14 not C15/C16, very similar just different heat rating.

Yeah in the U.S., high powered appliances like washing machines run off of 240V. Also for 120V or 240V AC, it’s either 3 or 4 phases (minus ground) for either AC voltage.

 

That's why I was using those figures before, because that's where I live. As you said, it's different in your area.