Explanation of PCIe

[straight_stewie]

Preface

 

     This guide hopes to answer the question “what is PCIe” and more specifically to help serve as a guide to system planning. I have tried to make this as accurate and as easy to read as I can. If you spot any inaccuracies please let me know so that I can fix them.

 

What is the PCIe Interface?

 

     Peripheral Component Interface - express (PCIe) is a bus that allows expansion cards inside your computer to communicate with other components. You use the PCIe system by putting cards into slots on the motherboard. These cards can be things like graphics cards, RAID cards, network cards, even co-processors. 

 

How Does the PCIe Interface Work?

Spoiler

     Expansion cards can communicate with other system resources by using lanes Lanes are represented by an “x” followed by the number of lanes being used. Multiple lanes can be used by one card, and we call these configurations “modes”. The PCIe 3.0 specification allows for these modes and slots, in ascending order by bandwidth/length:

1. x1
2. x2
3. x4
4. x8
5. x16
6. x32 (for all intents and purposes this doesn't actually exist).

     When a card is placed into a slot and the computer booted up, the system negotiates with the card to determine which mode to operate in. If there is only one card in the system, and it is in a slot that supports it's maximum mode, then the system will give it that mode, otherwise it will give it the maximum mode supported by the slot. For example: If a x16 card is in a x16 slot, then it will run in x16 mode. If, however, you place a x8 card in a x16 slot, then it will still run in x8 mode.

 

     When multiple cards are used, things get a little more complicated, but it is eased by the fact that all cards can operate in any mode below their listed mode. Processors and motherboards each have a maximum number of lanes that they will support. Three examples of different configurations are listed below.

 

Basic   

Spoiler

     For this example we will use the Sabertooth z87 motherboard and the Intel i7-4770K processor. The processor supports a maximum of 16 PCIe 3.0 lanes and the motherboard supports two PCIe 3.0 x16 slots. The motherboard slots can run either in 1 instance of x16 or two instances of x8. I will show a single GPU and a dual GPU application:

 

Single GPU

    It's simple. Put a single processor in the top slot of the motherboard. It will run in x16 mode and everything will be hunky dory. That was easy.

 

Dual GPU

    This one is a little more complex: We are going to use two GPUs. Specifically, this will be my personal setup, dual GTX 760's. Each GTX 760 operates in a maximum of x16 mode. However, the processor and motherboard only support 16 total lanes. How we use two graphics cards in this example is simple: We can run the cards in x8 mode. To calculate how many lanes you are using, simply add the mode value of all cards in your system. For example: 8 + 8 = 16. Since we have 16 lanes to use and we are only using 16 lanes, everything is hunky dory.

Advanced

Spoiler

     For this one we will use a processor and motherboard that support 40 PCIe lanes. With this setup, we can use 2 GPUs and a nice raid controller. Each GPU will need 16 lanes. 16x2=32, and 40-32=8. We have 8 lanes left to play with. This means that our raid controller will have to live with 8 lanes. Now we are using all 40 lanes, and everything is hunky dory. See, this isn't that hard.

Ludicrous

Spoiler

     Let's imagine for a minute that we are going to build a balls to the wall, absolutely unnecessary, ludicrously overpowered rig just to run Folding@Home. What would that look like?

• 2x Xeon E5-2699 V4

• Asus Z10Pe – D16 WS

• 6x Amfeltec PCIe expansion cluster

• 24x Nvidia Quadro M6000

 

     As you may have noticed there are only 6 PCIe expansion slots on the motherboard yet we have 24 GPUs. So how do we plan to add 24 graphics cards to this system? Enter the PCIe splitter! The Amfeltec PCIe expansion cluster takes one x4, x8, or x16 slot and turns it into 4 x16 slots. Now here's the confusing part: The expansion card only operates in x4 mode, and therefore, each GPU will only see x1 lanes. That may sound slow at first, but remember that the goal of this computer is highly parallel computing, not high speed computing. Even though each individual card will be running rather slowly in x1 mode, combined they produce a ludicrous amount of compute power.

 

 

What Exactly Is A Lane?

Spoiler

     A PCIe lane is simply one channel of serial communication. The supported lane configurations are 1, 2, 4, 8, 16, and 32 (for all intents and purposes this doesn't exist). Each configuration has it's own slot. You can differentiate them by length, a 1 lane slot is the shortest while a 32 lane slot is the longest. There are also PCIe modes. A mode is simply how many lanes a card is actually using. It is independent from how many lanes are available to the card. Any card or slot may operate in any mode below it's maximum mode, but must be in a slot that is the same physical size. Lanes and modes are denoted by an “x” preceding the number of lanes. For example: x1, x2, x4, x8, x16, or x32. 

 

     You can think of each lane like a two lane road (one lane in each direction). If we only have x1 lanes then we have a two lane road. If we have x2 lanes then we have a 4 lane highway. With x4 we have an 8 lane freeway. With x8 we have a 16 lane superhighway. So on and so forth...

 

How Does All This Affect Me?

Spoiler

When you are selecting components for a new build, you should try to make sure that each card you want to have can use it's maximum mode. However, this really isn't all that important. As we can see by looking at the image below:

     There is negligible difference between x16 and x8 modes. There is a much larger difference between x8 and x4 modes, but it is still rather negligible.

 

     When you are building systems, you really only need to pay attention to these things:

• Is my card going in a slot better than or as specified?

• Is the slot the same length as the connector on the card?

• Am I trying to use more lanes than my motherboard/processor support?

 

     If you answered yes to the first two questions and no to the third, you are good to go.

 

[MarcoTexture]

Very informative. I'd love to see more in depth information like how to calculate speed based on generation and number of lanes. I'd also like to see how much power does each generation has to offer. We know gen 3 is rated and 75W and gen 4 will be rated at 300W but what about gen1 and 2.  Keep it updated.

[Mira Yurizaki]

On 9/6/2016 at 10:50 AM, MarcoTexture said:

Very informative. I'd love to see more in depth information like how to calculate speed based on generation and number of lanes. I'd also like to see how much power does each generation has to offer. We know gen 3 is rated and 75W and gen 4 will be rated at 300W but what about gen1 and 2.  Keep it updated.

The speed of one lane is the following:

• 1.0a or 1.1: 250 MB/s
• 2.0: 500 MB/s
• 3.0: 985MB/s
• 4.0: 1969MB/s

You might go "how come 3.0 and 4.0 aren't 1000 and 2000 MB/s?" It's because the encoding of the signal changed. 1.0/1.1/2.0 used 8b/10b encoding which means 8 bits are encoded into 10 bits (it's for DC load balancing and other signal shenanigans). So you lose 20% of your actual bandwidth to encoding. PCIe 3.0 and 4.0 use 128b/130b encoding.

 

The reason for this is because it requires less total bandwidth for usable bandwidth. For instance PCIe 2.0 needs a 625 MB/s channel to get the 500 MB/s usable bandwidth. PCIe 3.0 needs 1000 MB/s for the 985 MB/s usable bandwidth.

 

As for power delivery, I believe 75W was standard since the beginning.

 

EDIT: The spec says 75W for graphics cards only. Everything else is limited to 25W. Though this may change.

[Guest]

I am sorry. But you linked bus to the wrong page. Other than that it looks really nice and informative. Well done

[straight_stewie]

On 9/25/2016 at 8:53 PM, dexxterlab97 said:

I am sorry. But you linked bus to the wrong page. Other than that it looks really nice and informative. Well done

trolololol. I'll fix it. I can't believe that I did that. Thanks for the heads up

[straight_stewie]

Unfortunately I'm not a member of PCI-SIG so I'm not able to take a look at the current Card ElectroMechanical specification. I couldn't find it anywhere online.

 

@Kumaresh