Building a $100,000 PC Pt. 2 - SO MANY PCIe CARDS

[nicklmg]

Check out One Stop Systems' server gear: http://geni.us/ocuAnN
 

 

[Firewrath9]

first comment! 

my computer is 2000 D:

only 7 SLI Quadro GV100s

that sucks!!!

 

we need 8

 

side note

plz compare evga's hybrid cards

[r2724r16]

WOW, that's only 100 times more expensive than my PC!

[Guest]

The flashing lights were kinda unsettling

[Sprood]

Posted this on youtube but I'm going to post this here as well so I can believe I was heard. Of course I am only speaking for myself and I may be the odd one out for all I know, this is just my 2 cents.

 

This editing style is really off putting for this format in my opinion. I think it was when they got their new editor but the text on screen and memes makes me feel like I'm watching a heavily edited League of Legends video. I love memes just as much as the next guy but I don't understand taking Linus Tech Tips in that direction, it really feels like a weird fit and detracts me from the video a lot. Also, some of the edits that t try to "speak for the viewer" almost feel insulting with how childish it is; I can only speak for myself but I think Linus is engaging enough that I don't need flashing popups and robot voices on the video to keep my attention. 

[Egg-Roll]

Personally I think being in all the videos have started to take a toll on Linus... I've done so much worst to my/work equipment than what he was literally whining about...

 

Perhaps instead of building a 100K computer, take a real vacation?

 

 

Not related to above:

 

38 minutes ago, Sprood said:

Posted this on youtube but I'm going to post this here as well so I can believe I was heard. Of course I am only speaking for myself and I may be the odd one out for all I know, this is just my 2 cents.

 

This editing style is really off putting for this format in my opinion. I think it was when they got their new editor but the text on screen and memes makes me feel like I'm watching a heavily edited League of Legends video. I love memes just as much as the next guy but I don't understand taking Linus Tech Tips in that direction, it really feels like a weird fit and detracts me from the video a lot. Also, some of the edits that t try to "speak for the viewer" almost feel insulting with how childish it is; I can only speak for myself but I think Linus is engaging enough that I don't need flashing popups and robot voices on the video to keep my attention. 

I forget which video (either iJustin tour or the mobile gimbal or some other video), but they literally have a editor that loves memes, my guess is he was the editor of the video.

[Guest zeuschops]

I meant to post this on the last version of this video (it didn't bug me enough to actually write up the post the first time), but Intel Optane is just a shitty storage configuration for the focus of this project. In its current state, m.2 drives are actually a little faster and more reliable with their read/write speeds even after being adapted to PCIe x4 slots from u.2 connectors. If I remember correctly, m.2 has speeds over 2600MBps read/write whereas Optane has speeds near 2100MBps read/write and Optane is nearly 64GB of storage where an NVMe SSD is around 4TB maximum (PCIe and M.2)

 

I haven't ever worked with server-grade equipment, but I have worked with video/display outputs and a PCI Riser is actually going to be the worst in terms of reliability for the editors since those cards and cables can actually just fall out from being connected (they aren't very secure configurations). The better way to do this would've been to wire the video outputs to go from HDMI to HD-SDI back to HDMI then into the display for anything over 50 feet, otherwise you can run a single HDMI cable without having to deal with hanging a video card and potentially causing some issues with breaking the PCB/Connectors for those cards long-term. If you're looking for a DisplayPort solution, the cable's length before signal loss maxes out around 16 feet and given the space for your editors it may be worth while to try to have more GPUs to compensate for the lack of HDMI outputs and then run some cables over HDMI instead of DisplayPort (considering HDMI 2.0 is nearly equivalent to DP 1.4 quality-wise). Maybe running DP to HDMI convertors may also be a solution for the length-related issue.

 

By all means, this isn't the worst design, but there are a lot of functionality improvements that could be done instead of risking hardware failure in some cases. Also, if I remember correctly, doesn't Adobe Premiere CC have a way to push the render jobs to a server to complete? You might not even have to run display cables if you keep the old systems for editing on if that is the case. (Note: I mainly use FCPX if I ever edit video instead of Adobe so I'm not 100% sure about feature availability.)

 

Questioning though, if you're pushing PCIe x4 to handle USB 3.1 extensions, is that connection v3 or v4 PCIe x4? If it's v3 won't USB 3 hit the theoretical cap after USB 3.1 Type-C transfers after about 3 consistent transfers (i.e. 3 editors transferring data over it) and if it's Type-A shouldn't it hit the cap around 7 or 8 simultaneous data transfers? Aside from the USB cap, I don't think there should be any issues with the PCIe expansion board unless the plan is to run PCIe x4 3.0 for Ethernet since you'd be significantly throttled the bandwidth of the lane in the first place.

 

I'd love to see the discussion so I do have notifications for replies turned on, cheers!

[Hellhium]

I don't know if anyone concerned will ever see this tweet so I'm trying here, I managed to ungroup any PCI device using this method and I don't think you tried it yet

[richardsim7]

Is this of any use? https://youtu.be/qQiMMeVNw-o

 

He does something else with IOMMU groups which I hadn't seen before, although unRAID would probably know about this method and told you to try it already

[adelintru]

Linus,

 

you should get one of these and play with. You could put like all your editors on one rig and still have like 5 gaming PCs available...

 

https://www.supermicro.com/products/system/7U/7089/SYS-7089P-TR4T.cfm

 

Cheers

[GDRRiley]

so i've seen some of the PCIe in a different box from an old Video switcher. A broadcast pix granite switcher was set up like this

[LeapFrogMasterRace]

My bathroom PC cost more 

[ahuckphin]

Aprime video editing style is different to normal.

and I don't mind change.

[8 Bit Refugee]

Where is part 1?

[Ikbosh]

I believe you can use native KVM/QEMU USB Host Bus passthrough. This still supports hotpluggin. I'm not sure if UnRAID natively supports it, so you may need to edit the VM xml. 

 

You'll need to find the usb bus associations using lsusb and other steps, and pass them through as a qemu device, usb host bus etc. 

[LinusTech]

4 hours ago, zeuschops said:

I meant to post this on the last version of this video (it didn't bug me enough to actually write up the post the first time), but Intel Optane is just a shitty storage configuration for the focus of this project. In its current state, m.2 drives are actually a little faster and more reliable with their read/write speeds even after being adapted to PCIe x4 slots from u.2 connectors. If I remember correctly, m.2 has speeds over 2600MBps read/write whereas Optane has speeds near 2100MBps read/write and Optane is nearly 64GB of storage where an NVMe SSD is around 4TB maximum (PCIe and M.2)

 

I haven't ever worked with server-grade equipment, but I have worked with video/display outputs and a PCI Riser is actually going to be the worst in terms of reliability for the editors since those cards and cables can actually just fall out from being connected (they aren't very secure configurations). The better way to do this would've been to wire the video outputs to go from HDMI to HD-SDI back to HDMI then into the display for anything over 50 feet, otherwise you can run a single HDMI cable without having to deal with hanging a video card and potentially causing some issues with breaking the PCB/Connectors for those cards long-term. If you're looking for a DisplayPort solution, the cable's length before signal loss maxes out around 16 feet and given the space for your editors it may be worth while to try to have more GPUs to compensate for the lack of HDMI outputs and then run some cables over HDMI instead of DisplayPort (considering HDMI 2.0 is nearly equivalent to DP 1.4 quality-wise). Maybe running DP to HDMI convertors may also be a solution for the length-related issue.

 

By all means, this isn't the worst design, but there are a lot of functionality improvements that could be done instead of risking hardware failure in some cases. Also, if I remember correctly, doesn't Adobe Premiere CC have a way to push the render jobs to a server to complete? You might not even have to run display cables if you keep the old systems for editing on if that is the case. (Note: I mainly use FCPX if I ever edit video instead of Adobe so I'm not 100% sure about feature availability.)

 

Questioning though, if you're pushing PCIe x4 to handle USB 3.1 extensions, is that connection v3 or v4 PCIe x4? If it's v3 won't USB 3 hit the theoretical cap after USB 3.1 Type-C transfers after about 3 consistent transfers (i.e. 3 editors transferring data over it) and if it's Type-A shouldn't it hit the cap around 7 or 8 simultaneous data transfers? Aside from the USB cap, I don't think there should be any issues with the PCIe expansion board unless the plan is to run PCIe x4 3.0 for Ethernet since you'd be significantly throttled the bandwidth of the lane in the first place.

 

I'd love to see the discussion so I do have notifications for replies turned on, cheers!

There is FAR more to storage performance than sequential read/write speed. Optane will be VASTLY superior for this workload.

 

The PCIe riser has nothing to do with the range of an HDMI signal. They are completely independent.

 

Also we don't intend to actually roll this out. It's a theoretical exercise.

 

Again the PCIe riser has nothing to do with the USB cards and any theoretical limits they might have on transfer speeds. It will introduce a bottleneck but it remains to be seen how much it actually affects anything.

 

Hope that helps.

[aw_]

What gives, the PEX 8609 that we see in the PCI info claims to support ACS in their datasheet.  I don't see it listed in the output there, but is the IOMMU group rooted in the motherboard root port?  Note that quirks for 300 series went in upstream during the v4.18 development cycle, but it looks like we may have regressed C246 as the root ports match the PCI IDs for broken devices, but seem like maybe they've actually fixed intel's bug.  See here if this is a relevant chipset for this build https://patchwork.kernel.org/patch/10368103/

[dsun]

So this may have been asked, but I am curious, can ESXi see the expandable PCIE slots?  Can you put a GTX1080 on the expansion PCIE board and VR on it? How much frame rate loss would there be?  
Asking because I am trying to get a VR cafe started and have questions, I have a couple of servers that have only 2 PCIE x16 slots but I need more to have 3-4 VM's per server with 4 separate USB 2.0 or 3.0 controllers and 4 GTX1060's or GTX1070's and passthrough those to a VM.

[johnnydoe9]

Nah I'm okay with the editing, they need to keep it light and fun. Editing is much better than Linus having to act weird jumping around with waving hands or whatever he does. This is much better. 

[Tcrumpen]

11 hours ago, Egg-Roll said:

Personally I think being in all the videos have started to take a toll on Linus... I've done so much worst to my/work equipment than what he was literally whining about...

 

Perhaps instead of building a 100K computer, take a real vacation?

 

 

Not related to above:

 

I forget which video (either iJustin tour or the mobile gimbal or some other video), but they literally have a editor that loves memes, my guess is he was the editor of the video.

Not exactly easy for the CEO of a company to take a vacation, as far as i know there is no Deputy CEO or COO of LTT. Also before someone says Luke is COO he is COO of Floatplane, different company to LTT

[billington.mark]

Long time youtube lurker, and long time UnRaid user here...

 

If you want to have multiple PCIe devices in a single PCIe slot, AND have them in their own IOMMU group to be able to pass through to different VMs without hacking away at config files, you need to use PCIe Bifurcation. Usually you can split a 16x port into 8x/8x, or 4x/4x/4x/4x. Workstation\Server grade mobo's usually have this available, but ive read of some custom bios's being available from manufacturers to enable it where it wasnt. 

 

Im using an AsRock EP2C602-4L/D16 and have 2 16x PCIe slots which i can change from x16 to x8/x8 or x4/x4/x4/x4. Using it to have 4 NVMe drives in a single x16 slot. 2 are passed through as a PCIe device used in their own VMs, and the others are used as cache drives in my array. 

 

Pretty sure supermicro make some PCIe risers which make use of PCIe Bifurcation... 

[joshfrog]

Hi Guys

 

When Watching todays video Linus was playing with the board from One Stop Systems i was thinking he should plug it into a laptop using thunderbolt and see what happens.

Yes i know the thunderbolt has overhead and is only pci-e gen 3 x4 or x2 on some laptops but still.

 

Thanks

[Egg-Roll]

4 hours ago, Tcrumpen said:

Not exactly easy for the CEO of a company to take a vacation, as far as i know there is no Deputy CEO or COO of LTT. Also before someone says Luke is COO he is COO of Floatplane, different company to LTT

Minor technical detail actually... Simply hire him for a week part time/on call (they likely do this for the uploads like the Labo video) and if Colton was as truly incompetent as the videos point out he'd not be there anymore

 

For a company like LMG, you'd be surprised how long a company can run w/o the CEO  If the company crashes and burns in that week, then he has really bad employees and it is time to mass fire and rehire 

 

Afaik Linus does a lot of the back end stuff like manufacture contacts emails phone calls etc and acting (the videos), his wife does the finances and likely a few other things. All that is really required is someone to forward those inquiries to if they can't wait a week (auto away email settings are awesome that way), and to hire a accountant for a few hours once to do the weekly financial stuff if no one can do it.

[jtsoi]

4 hours ago, Tcrumpen said:

Not exactly easy for the CEO of a company to take a vacation, as far as i know there is no Deputy CEO or COO of LTT. Also before someone says Luke is COO he is COO of Floatplane, different company to LTT

I believe Linus stated it in one of his videos that Nick Light is the COO and basically he's the next in line in decision making when he's not available.

[NMS]

This channel is becoming more like Fortnite.

Thank god GN exists!