Need some NVLink Answers

phongle123 · June 5, 2019

Hello, I've got some questions about NVLink and we like to know some of these. Links would be great too. I've been searching and not getting much answers.

1) According to this article from PugetSystems [Link here] the Quadro NVLink works with the RTX cards. Since the Quadro NVLink is the one that has a 2 slotter. However, I think I've read somewhere that the Quadro NVLinks have much higher bandwidth than the RTX NVLinks. Will this impact, take away, or else adversely affect performance or functionality?

2) Following up on #1, I read that the NVLinks are separated into: Quadro, Titan RTX, and RTX for 2080 Ti and lower. I think I read somewhere that the Quadros are able to have a 150GB/s transfer and the RTXs are able to have a 100GB/s transfer. Is this limitation due to the GPUs themselves or due to the NVLink where the Quadro has the 150GB/s and the RTX has the 100GB/s? Adversely, are the NVLINK for Titan RTX and the RTX for 2080 Ti and below exactly the same or is the Titan RTX Nvlink better? Because it's like 150$ more.
3) Is it possible to do 3 or 4 way NVLink?

4) I asked about this a while ago and someone stated that Nvidia has not enabled the ability to pool memory yet via NVLink. Are we able to or not yet? (I do not want any discussion about whether or not 2080 Ti can Pool memory or not)

Thank you in advance.

Princess Luna · June 5, 2019

RTX cards uses old fashion SLI, the only difference is that now they standardized the NVLink connector across the board to make production unified and thus cheaper so RTX cards including the RTX TITAN are all limited to SLI functionality.

True NVLink is still only available on Quadro line up and yes on true Quadro NVLink you can stack 3 or 4 cards and even stack memory among other things.

SavageNeo · June 5, 2019

2080tis can do only dual nVlink.

phongle123 · June 5, 2019

I was off about one thing and I did question it. It seems like the Titan RTX and the 2080 Ti and Below share the same NVLink connector. The 150$ difference was something I saw off of eBay for 220$ vs the 79.99$ price I remembered. After looking at the official NVidia page for the Titan RTX it shows a 79.99$ price tag. This leads me to believe that they are the same.

@Princess Luna

Furthermore, when you look on the Titan RTX page it states, "Double the effective GPU memory capacity to 48 GB and scale performance with up to 100 GB/s in total bandwidth of data transfer utilizing the NVIDIA NVLink™ technology." Looking at the Quadro 6000 it states that it can, "Connect two Quadro RTX 8000s together with a Quadro RTX 8000 NVLink HB Bridge for up to 100 GB/s of bandwidth and a combined 96 GB of GDDR6 memory to tackle your largest rendering, AI, virtual reality, or visualization workloads."

From this article [Link here] it is stated that 2080 Ti have 100GB/s and the 2080 have 50GB/s bandwidth. So this probably means that the bandwidth capability is dictated by the GPUs. This is also why 2080 Ti's are able to Pool Memory unofficially. This feature seems to be locked by NVidia on the 2080 Ti cards. While the Titan RTX and the Quadro's can Pool Memory and they are also rated at 100GB/s bandwidth. Thought I can't seem to find the article for the 2080 Ti Memory Pooling anymore.

@SavageNeo @Princess Luna

Could you give me an article that shows more than 2? I have not been able to find one.

Mesterial · June 6, 2019

All this has to do with the number of lanes of the NVlink (in the same way as PCIe). I think they are called "sub link".

For desktop ifaik, the 2080ti has 1 connector (and 2 "lanes", while the 2080 has 1 connector and 1 lane), while recent Quadro has 2 connectors (4 lanes total), and it is not possible to use more than 2 cards together.

Now on the HPC side, there are 4 and even 8 "cards" on a single node ("pc"), BUT they are not PCIe cards form-factor, they use SMX2 weird connectors which wire PCIe lanes and NVlink lanes. Each GPU has 6 NVlink lanes in these cases.