PCIE Card Design

[Wh0_Am_1]

Hey everyone! 

I have recently become interested in how PCIE cards are designed/made, and possibly designing my own card in the future, and I would like to know, does anyone have any good ideas on where I can find data on how to design PCIE cards? Because all I have been able to find so far on Google are some presentations that would be good for a large company and no actual design guides. I know that this a difficult, complex, and specialty subject,  but  I am interested nonetheless...

Thanks in advance!

[Fasauceome]

Do you already know how to make a PCB?

[Wh0_Am_1]

2 hours ago, fasauceome said:

Do you already know how to make a PCB?

Have to admit, I am still working on that one, though I have found it much easier to find source material for that than PCIE, I still have got a lot to learn though.

[Fasauceome]

7 minutes ago, Wh0_Am_1 said:

Have to admit, I am still working on that one, though I have found it much easier to find source material for that than PCIE, I still have got a lot to learn though.

PCIe is a form of PCB, I think if you finish learning about making a PCB in general you'll better understand material you read about PCIe devices

[Wh0_Am_1]

 

1 hour ago, James Evens said:

Read the pcie specs. The rest is normal electronic design.

If making your own crad also means using a prebuild FPGA card and adding your own IO it is simpler.

Maybe, but the card I want to make is going to be rather unique in a multitude of ways, so using a premade board is unfortunately not going to cut it. So I need to figure out what components I would need to use, and how to get them to talk to the CPU. Yeah this is going to be a long project.... Thanks!

[Wh0_Am_1]

1 minute ago, fasauceome said:

PCIe is a form of PCB, I think if you finish learning about making a PCB in general you'll better understand material you read about PCIe devices

Yeah... unfortunately I have yet to find anything in that realm that really discusses how PCIE works as it is a rather niche thing in PCB design, and what I need, to get it working.

[Wh0_Am_1]

31 minutes ago, James Evens said:

The beauty of fpga is that you "programm" hardware with lots of IO. Think of it like a ASIC you build in software. Other option is to place a fast microcontroller or ARM A on the card and let them handle pcie. Both techniques are legitmate and what is better depends on your usecase. Some FPGS like Zynq or Cyclones have both a FPGA and a ARM-A inside.

For the PCIE-spec the source is PCI-sig. The good is there are librarys (code) avaible for PCIE and some scopes have a option for PCIE debugging.

Well I know I am going to need an ARM processor on the card, maybe I can multipurpose it.  

[r4tch3t]

Why do you need pcie? It's a very high bandwidth bus. USB has plenty of bandwidth and is supported by many micros natively. USB 3 to a lesser extent but still a lot easier than pci.

[mariushm]

To make a card that plugs into a pci-e slot, you simply have to make the circuit board be that thickness it's standardized and defined in the pci-e specifications (and various companies that make circuit boards can make that thickness) and the contacts have to be ideally gold plated (which is generally an extra step when making circuit boards and can cost some money (typically cents to dollars per board depending on number of contacts)

 

pci-e is serial communication but the problem is the speeds are super fast, much faster than what a typical microcontroller or cheap arm based micro can sustain, so it would be quite difficult to buy something cheap (let's say <10$ in price) and program it and have your code inside the microcontroller receive from and send data to the computer. Unless it has a pci-e controller or something hardware built in, you can't "bitbang" pci-e like you can usb 1.1 or even usb 2  for example. 

Even on the oldest pci-e standard, you're dealing with speeds of 250 MB/s so whatever you put on the board must be able to send and receive those bits at those speeds.

There are FPGA chips which have pci-e interfaces that would simplify things and allow your fpga to communicate with the pc through thepci-e sot but that's just the beginning.

If you want to actually release a product and call it a pci-e device, you need to sign some agreements with the organization that made the pci-e standard and pay a fee that could be over 10k dollars to receive a unique vendor id and then also have a device id...  the fpga chips come with those values which belong to xilinx or whoever made the fpga chip ... so if you want windows to detect your card as "your company sound card" instead of "xilinx fpga generic device" or something like that... it would cost some more cash.

 

[ZihanYu]

15 hours ago, Wh0_Am_1 said:

Hey everyone! 

I have recently become interested in how PCIE cards are designed/made, and possibly designing my own card in the future, and I would like to know, does anyone have any good ideas on where I can find data on how to design PCIE cards? Because all I have been able to find so far on Google are some presentations that would be good for a large company and no actual design guides. I know that this a difficult, complex, and specialty subject,  but  I am interested nonetheless...

Thanks in advance!

To get the pcb, try 

https://www.pcbway

They make custom pcb for prototype 

[ZihanYu]

Just now, ZihanYu said:

To get the pcb, try 

https://www.pcbway

They make custom pcb for prototype 

And http://www.custompcb.com/ is also a good choice 

[Wh0_Am_1]

9 hours ago, r4tch3t said:

Why do you need pcie? It's a very high bandwidth bus. USB has plenty of bandwidth and is supported by many micros natively. USB 3 to a lesser extent but still a lot easier than pci.

Bandwidth is only part of the issue, I want this card to integrate into the computer itself, and it will require at least 75 Watts to operate, though the reduced lag time to talk to the processor in PCIE, will also greatly increase the utility of the card, basically for the use case of the card, PCIE is the only solution that meets all of my requirements for this card.

[Wh0_Am_1]

6 hours ago, ZihanYu said:

To get the pcb, try 

https://www.pcbway

They make custom pcb for prototype 

 

1 minute ago, Wh0_Am_1 said:

Bandwidth is only part of the issue, I want this card to integrate into the computer itself, and it will require at least 75 Watts to operate, though the reduced lag time to talk to the processor in PCIE, will also greatly increase the utility of the card, basically for the use case of the card, PCIE is the only solution that meets all of my requirements for this card.

Thanks! But I already found a local PCB manufacturer that specializes in custom cards. 

[mariushm]

2 hours ago, Wh0_Am_1 said:

Bandwidth is only part of the issue, I want this card to integrate into the computer itself, and it will require at least 75 Watts to operate, though the reduced lag time to talk to the processor in PCIE, will also greatly increase the utility of the card, basically for the use case of the card, PCIE is the only solution that meets all of my requirements for this card.

You're not always guaranteed the 75w through the slot.

PCI-e has 3.3v and 12v.  The motherboard is suppose to provide up to 10w or something like that on 3.3v and up to 5.5A on 12v (~ 66w) on 12v, but some people read the specs incorrectly and say you get 75w on 12v.

Also, the 5.5A on 12v is only valid if your card is of the "video card" type, anything else should not use more than 25w from the slot, and x1 cards should not use more than 10w

. 

Quoting from wikipedia:

 

Quote

All PCI express cards may consume up to 3 A at +3.3 V (9.9 W). The amount of +12 V and total power they may consume depends on the type of card:^{[15]:35–36[16]}

• ×1 cards are limited to 0.5 A at +12V (6 W) and 10 W combined.
• ×4 and wider cards are limited to 2.1 A at +12V (25 W) and 25 W combined.
• A full-sized ×1 card may draw up to the 25 W limits after initialization and software configuration as a "high power device".
• A full-sized ×16 graphics card^[14] may draw up to 5.5 A at +12V (66 W) and 75 W combined after initialization and software configuration as a "high power device".

Optional connectors add 75 W (6-pin) or 150 W (8-pin) of +12 V power for up to 300 W total (2×75 W + 1×150 W).

 

In general, motherboards for regular users don't have fancy circuitry to monitor how much power a card takes so that everyone would abide by these rules, so you'd be lucky... otherwise you could have a x16 slot in the computer that's only x4 electrically, and the motherboard could have a resettable fuse on it which would trip if your card uses more than 25-30w because that's the rule. 

 

Keep in mind ALL the pci-e slots are powered from the 24pin atx connector, and there's only TWO 12v wires in that connector, which also deliver 12v to fans (and some other crap on the motherboards). Each of the two wires in theory can give a bit more than around 9A of current safely (the metal contacts in the connectors are rated for 9A, wires can carry a bit more), so reserving around 2A for fans and other things on motheboards like RGB strips, you're looking at around 16A at 12v or less than 200w for all the pci-e slots.  If user has a video card plugged in which consumes 70w, all the other pci-e slots may be starved of power.

 

If your card is going to consume that much power, it's much much safer and better overall to just install a 6pin or 8pin pci-e power connector on the card.

[Wh0_Am_1]

 

9 minutes ago, mariushm said:

You're not always guaranteed the 75w through the slot.

PCI-e has 3.3v and 12v.  The motherboard is suppose to provide up to 10w or something like that on 3.3v and up to 5.5A on 12v (~ 66w) on 12v, but some people read the specs incorrectly and say you get 75w on 12v.

Also, the 5.5A on 12v is only valid if your card is of the "video card" type, anything else should not use more than 25w from the slot, and x1 cards should not use more than 10w

. 

Quoting from wikipedia:

 

 

In general, motherboards for regular users don't have fancy circuitry to monitor how much power a card takes so that everyone would abide by these rules, so you'd be lucky... otherwise you could have a x16 slot in the computer that's only x4 electrically, and the motherboard could have a resettable fuse on it which would trip if your card uses more than 25-30w because that's the rule. 

 

Keep in mind ALL the pci-e slots are powered from the 24pin atx connector, and there's only TWO 12v wires in that connector, which also deliver 12v to fans (and some other crap on the motherboards). Each of the two wires in theory can give a bit more than around 9A of current safely (the metal contacts in the connectors are rated for 9A, wires can carry a bit more), so reserving around 2A for fans and other things on motheboards like RGB strips, you're looking at around 16A at 12v or less than 200w for all the pci-e slots.  If user has a video card plugged in which consumes 70w, all the other pci-e slots may be starved of power.

  

If your card is going to consume that much power, it's much much safer and better overall to just install a 6pin or 8pin pci-e power connector on the card.

Looks like I am in for some fun then because the computers that I plan to install the card in don't have PSUs with ATX power cables. And I am going to need every watt from the 16X PCIE slots.

[Unimportant]

On 10/10/2018 at 11:58 PM, Wh0_Am_1 said:

Hey everyone! 

I have recently become interested in how PCIE cards are designed/made, and possibly designing my own card in the future, and I would like to know, does anyone have any good ideas on where I can find data on how to design PCIE cards? Because all I have been able to find so far on Google are some presentations that would be good for a large company and no actual design guides. I know that this a difficult, complex, and specialty subject,  but  I am interested nonetheless...

Thanks in advance!

I'd suggest you try learning to walk before trying to design new warp nacelles for the USS Enterprise.

 

At the speeds PCIe operates at you get transmission line effects such as reflections. This requires proper impedance matching techniques. It's a capacitively coupled differential bus which requires proper layout of the differential pairs with length matching, etc...

 

To drive the point home: At some point in the design process you'll have to start measuring the signals to solve problems, make eye diagrams to check signal integrity, etc.

A oscilloscope with the bandwidth to properly measure these signals (= capture enough harmonics) will set you back by the price of a nice car.

 

If you really want to learn more about this stuff perhaps you should start off much simpler. Find yourself a buck-switcher IC that operates at 1Mhz of more. Design a PCB for it, build it, and use it to power a 1A resistive load. Then scope around and look at the horrendous crap it's generating that most hobbyists simply overlook or don't even realize is there.  You'll see HF switching noise on the input & output, ringing, large negative transients on the switching node, jamming FM radio's on harmonic frequencies of the switching frequency and so on.

Learn proper scope probing techniques so you're not chasing ghosts, understand what's happening and learn proper PCB layout techniques to mitigate the above problems. Learn why adding a 100nF trough hole capacitor to the output does squat to filter out the HF switching noise while adding a 100nF 0805 X7R does wonders, etc, etc...

 

Maybe then start dreaming of designing a PCIe card one day.

[Wh0_Am_1]

8 hours ago, Unimportant said:

I'd suggest you try learning to walk before trying to design new warp nacelles for the USS Enterprise.

 

At the speeds PCIe operates at you get transmission line effects such as reflections. This requires proper impedance matching techniques. It's a capacitively coupled differential bus which requires proper layout of the differential pairs with length matching, etc...

 

To drive the point home: At some point in the design process you'll have to start measuring the signals to solve problems, make eye diagrams to check signal integrity, etc.

A oscilloscope with the bandwidth to properly measure these signals (= capture enough harmonics) will set you back by the price of a nice car.

 

If you really want to learn more about this stuff perhaps you should start off much simpler. Find yourself a buck-switcher IC that operates at 1Mhz of more. Design a PCB for it, build it, and use it to power a 1A resistive load. Then scope around and look at the horrendous crap it's generating that most hobbyists simply overlook or don't even realize is there.  You'll see HF switching noise on the input & output, ringing, large negative transients on the switching node, jamming FM radio's on harmonic frequencies of the switching frequency and so on.

Learn proper scope probing techniques so you're not chasing ghosts, understand what's happening and learn proper PCB layout techniques to mitigate the above problems. Learn why adding a 100nF trough hole capacitor to the output does squat to filter out the HF switching noise while adding a 100nF 0805 X7R does wonders, etc, etc...

 

Maybe then start dreaming of designing a PCIe card one day.

Maybe I can use my ham radio oscilloscope.  Yeah I knew this is going to be a long haul project, I am mostly currently looking for good sources (that are preferably inexpensive) from which I can learn at the moment, I understand that I have got alot to learn. Thanks!

[straight_stewie]

PCIe is just a communication and connector protocol. Perhaps a good primer would be the post in my signature, however, it has not been updated for PCIe 3. You will need engineering documentation to build a board for it, which can be found on the PCI-SIG website, here: https://pcisig.com/specifications

[Cipulot]

First of all.... what are you planing to "create" for PCI_Express?? A comunication card for serial/parallel controll or more complex stuff?

I think that you should first decide the project (a very basic one), look around and search for the easiest one. Second you have to get a Schematic/PCB design software: something like OrCAD Capture and Allegro is a good start, they give also a free (but project size limited) student version. Note that for this you have to get at least a pcb layout prototype for the pci-e card dimensions(for what I recall there aren't in the examples folder). Next learn the basic specifications and requirements for the PCB structure (layest and trace capacity for the power that the project might need from the internal pcie power delivery (I belive is 75W max, without power connectors). Maybe you can find some projects online for free (look on site of electronics, maybe on the EEVBLOG). Once you've started maybe switch to a more "all in one" software solution, like Altium Designer. They have pcie card template with pre compiled specs and are free to download from altium repository website. You can even import OrCAD Capture and Allegro projects into Altium with the incuded tool.

I hope I've helped.

 

[Wh0_Am_1]

14 hours ago, straight_stewie said:

PCIe is just a communication and connector protocol. Perhaps a good primer would be the post in my signature, however, it has not been updated for PCIe 3. You will need engineering documentation to build a board for it, which can be found on the PCI-SIG website, here: https://pcisig.com/specifications

This pcisig website looks helpful, thanks!

 

11 hours ago, Cipulot said:

First of all.... what are you planing to "create" for PCI_Express?? A comunication card for serial/parallel controll or more complex stuff?

I think that you should first decide the project (a very basic one), look around and search for the easiest one. Second you have to get a Schematic/PCB design software: something like OrCAD Capture and Allegro is a good start, they give also a free (but project size limited) student version. Note that for this you have to get at least a pcb layout prototype for the pci-e card dimensions(for what I recall there aren't in the examples folder). Next learn the basic specifications and requirements for the PCB structure (layest and trace capacity for the power that the project might need from the internal pcie power delivery (I belive is 75W max, without power connectors). Maybe you can find some projects online for free (look on site of electronics, maybe on the EEVBLOG). Once you've started maybe switch to a more "all in one" software solution, like Altium Designer. They have pcie card template with pre compiled specs and are free to download from altium repository website. You can even import OrCAD Capture and Allegro projects into Altium with the incuded tool.

I hope I've helped.

 

I will have to take a look at these tools, Thanks! And to answer your question, one of the projects I intend to work on will be a long range telecommunication card, that will operate in the VHF and UHF bands, I already know which frequencies are restricted and which are available to the public, and what kind of authorization I would need to get from my local telecommunication regulatory agency.

[Wh0_Am_1]

2 minutes ago, James Evens said:

 did you already played with pcie SDRs?

 

No I haven't.

 

5 minutes ago, James Evens said:

 

If you build your own wireless card you need to certify it and this is not cheap.

 I never expected it to be cheap, I know that certification is going to take a while and be very expensive, Because this is the government I am dealing with here, things with bureaucrats involved are never fast, nor cheap. 

[kevdagoat]

On 10/11/2018 at 5:22 PM, r4tch3t said:

Why do you need pcie?

Exactly?

 

Use USB or Serial

On 10/12/2018 at 9:34 PM, Unimportant said:

At the speeds PCIe operates at you get transmission line effects such as reflections. 

That is true. You’ll have to length match tracks and the via’s they are connected to.

 

On 10/15/2018 at 3:29 PM, Cipulot said:

Maybe you can find some projects online for free (look on site of electronics, maybe on the EEVBLOG)

I don’t think Dave has done anything with pcie for the reasons above

[ShredBird]

If you're really looking for information, you could go browse Keysight's website.  They used to be Agilent / Hewlett Packard and make oscilloscopes and other instruments you would use to validate PCIe operation.  You will quickly find you are over your head when it's gonna take an ADS license to carefully simulate and design the microstrip waveguides for the correct impedances and then spend tens of thousands of dollars on an oscilloscope with enough bandwidth to validate what you're doing along with the FPGA you'll use a bridge to prototype things.

PCIe is not a hobbyist level endeavor, it's a state-of-the-art bus and someone ought to be paying you to go down that rabbit hole.  Lol

But if you're still interested, here is the rabbit hole: https://www.keysight.com/us/en/solutions/high-speed-digital-system-design/peripheral-component-interconnect-express-pcie.html

[Wh0_Am_1]

15 hours ago, ShredBird said:

If you're really looking for information, you could go browse Keysight's website.  They used to be Agilent / Hewlett Packard and make oscilloscopes and other instruments you would use to validate PCIe operation.  You will quickly find you are over your head when it's gonna take an ADS license to carefully simulate and design the microstrip waveguides for the correct impedances and then spend tens of thousands of dollars on an oscilloscope with enough bandwidth to validate what you're doing along with the FPGA you'll use a bridge to prototype things.

PCIe is not a hobbyist level endeavor, it's a state-of-the-art bus and someone ought to be paying you to go down that rabbit hole.  Lol

But if you're still interested, here is the rabbit hole: https://www.keysight.com/us/en/solutions/high-speed-digital-system-design/peripheral-component-interconnect-express-pcie.html

Thanks!