egpu As-small-as-possible eGPU

[Slidspitfire]

Dear Communauts,

 

I am a physicist and a OpenCL, CUDA and  C++ programmer for a large scientific (high energy physics) collaboration.

I quite often fly through (mostly) Europe and I need a good platform to bring with me in a backsack that allows me to:

1. Run OpenCL code
2. Run CUDA code
3. Run Mac OS X (personal and opportunistic choice)
4. Improve graphics performances for CAD and casual evening gaming sessions

I am actually working with an awesome MacBook Pro Retina 15" mid 2014, equipped with an Iris Pro and a GTX 750m, and I am really happy with that machine.

This machine is provided by my institution and, by the end of my PhD (Oct '18) I'll have to return it.

At that point I would like to get a modern machine, but I cannot find neither a hackintosh-capable laptop that fully satisfies me nor an Apple laptop with CUDA support.

 

At a given point an idea jumped in my head as a crazy monkey: may I be able to create a little thunderbolt box with a small GPU that can handle all my tasks with any TB2 or TB3 equipped computer?

 

To cite Linus, let's find out!

 

The enclosure

I started digging and I found the first piece of the jigsaw: a Sonnet Echo SEL.

 

 

This awesome TBx to PCIe 8x (mechanical, 4x electric) enclosure can be equipped with a low profile and single slot PCIe card. "TBx" means that the enclosure comes in both TB2 and TB3 versions, with the possibility to switch between the two buying the upgrade (downgrade) board from Sonnet itself.
After some email with Sonnet presales I got a precise statement on the max power of the enclosed card:

 

Quote

OK, here’s the deal. the 60W power supply is 12V @ 5A.

 

15W is allocated to a down-stream bus-powered device. If you don’t connect a device, you harvest 15W.

15W is allocated to charging a Mac, but this works only with Thunderbolt 3 Macs that are not plugged into the wall, so you can harvest another 15W.

Up to 10W is allocated to 3.3V, so if you don’t use 3.3V, you can ignore that.

5W is allocated to the box circuits, leaving you with up to 55W for a card.

 

If that is not enough, you can purchase our compatible 120W power supply.

 

Then I got the max specs for my GPU choice: 55W with the enclosed AC Power Supply. The 120W option was not really considered: packing so much power in a so small enclosure is, to me, not ideal.

In addition there are some size limitations, as seen from the following cartoon.

 

 

Low profile, single slot and less than 55W (the less, the best) and strong restrictions on total length: challenge accepted! 

 

The GPU

Diggin' down some more I eventually came out with three options:

1. EVGA GTX 1030 (passive or active) -> not really single slot and not so powerful with respect to a 750m (efficiency is another stuff...) or a Radeon Pro 555 or 560 (the GPUs the new MacBook is equipped with).
2. ASL GTX 1050Ti battle flag -> A nice piece of hardware, a little bit difficult to get, but overall too long for the Sonnet enclosure (190mm vs 167.9mm) and too power hungry (75W).
3. PNY Nvidia Quadro P1000 -> Good video card, 1050ti like, expensive but rated at 47W max. Nice touch: the 4 miniDP will be compatible with my current MacBook display adapters, so no need for other dongles to carry around.

 

As you may already argue from the list I opted for the pricy P1000 (~340EUR on Amazon), with the DVI model, coming with 4 miniDP to DVI adapters instead of the 4 miniDP to DP adapters one can find in the standard version.

 

All the pieces are currently shipped, alongside with a pack of 3 miniDP dummy display dongles which will be useful to accelerate internal MacBook display even without an external display directly attached to the GPU (link).

 

Summary

If everything goes as planned this project will bring me (and possibly you) a neat solution to:

• Get additional display connectivity (and processing power to handle them) with any TB2 (are there any?) or TB3 computer;
• Get a solution to run CUDA in mobility with any TB2 (same question) or TB3 computer;
• Get hardware acceleration for CAD and photo editing software
• Get hardware acceleration for mid-settings casual gaming

 

Isn't it neat? 

 

Stay tuned for further upgrade and a step-by-step guide through al the modifications and all of the tuning that will be necessary to realise this project!

 

Slid

[Slidspitfire]

What you get...

Here I come to the first update, focused on the material I had to acquire (or I simply wished to!) in order to complete the build.

 

The enclosure

The Sonnet Echo SEL Comes in a sturdy cardboard package, with a little bit lacking protection.

I was surprised by the lack of soft material inside the box: the enclosure was simply kept is place by some cardboard structures.
Nothing to worry about. The Sonnet Echo SEL is an awesome piece of engineering, and the steel made frame is sturdy enough to resist to several abuses!

 

  

 

Worth mentioning that the enclosure has a 8x mechanical (4x electric) PCIe slot for the expansion board. This is something we'll address in a while: after all we really want to install a 16x GPU! The enclosure comes with a pre-installed 60mm (mounting holes at 71.5mm) fan blowing air outside from the table-facing side of the chassis.

 

  

 

The enclosure comes along with a TB2 to TB2 cable, the 60W PSU and an instruction booklet in both English and Japanese.

 

The GPU

The Quadro P1000 is produced by PNY, as any other Quadro GPU on the market.
This is a warranty of quality, but the boxes are a little roughing. The GPU comes in a cardboard box, which appears to be the same for the whole PNY lineup. In fact the model specification is pasted right below the "Quadro" writing using a sticker, in my case reporting "P1000".
I have to admit that at first I was a little surprised by this peculiarity and my heart stopped working for some seconds, fearing a fraud, up to the moment I opened the box. And there it is!

 

Inside the box a pair of trays are stack together. Both are suspended with respect to the outer box, hence protected from collisions and damages.

In the first tray I found the instruction booklet, a driver CD (wow!), a plastic bag filled with the 4 miniDP to DVI adapters and, last, but not least, the low profile bracket. The second tray contains the GPU itself, inside an EM-screening plastic bag. An awesome piece of hardware, feeling great in my hands, with the machined heat spreader and a heavy and well built appearance.

 

Accessories

In order to test the GPU accelerating the integrated laptop screen I bought a miniDP dummy display. This adapter emulates a display, with support to almost any resolution. More details on the testing sections of the log. 

 

 

To be consistent with all my other pieces of hardware and computers I wanted to replace the enclosure fan with a Noctua one, adding a dust filter.

I managed to get the Noctua fan and a pair of sheets of dust filtering material on Amazon: 60mm dust filters are quite rare to find!

 

 

... And what you got to do

There we go with the build log and instructions!

 

Getting a low profile P1000

This is a trivial part of the build, but it's indeed something important. Since I am trying to get to a as-small-as-possible setup I've chosen a rather small TB enclosure. For this reason the chosen GPU had to be low profile and single slot.
The wisely selected P1000 offers a low profile rear bracket already in the off-shelf box: great stuff!

To install the bracket one should simply unscrew a pair of nuts which keep the original (full size) bracket in place.

These nuts are orthogonal to the board and are situated in the back of the board, while the corresponding Phillips bolts are facing the top of the board (where the heatsink is).

Once unscrewed, the bracket comes off with ease and the low profile one can be installed even easier, given the performed reverse procedure. Tighten the two bolt and nut pairs and the video card will be ready to go in the brand new enclosure!

 

||

 

Cutting the edge. Really.

As already mentioned the enclosure itself is the piece of hardware that requires the big part of the work needed to accomplish the project.
The work-items I had to work on are the following:

1. The PCIe slot is 8x physical, with a hard edge at the rear of the physical slot, preventing us from installing 16x peripherals;
2. The enclosure is built thinking about reverse airflow (fresh air from PCI brackets, warmed air out from underneath);
3. The fan, in addition to being mounted the wrong way around for a GPU application, is covered by a steel duct for the sucked air.

The solutions to these items are, respectively:

1. Cut apart the plastic edge in order to open the rear-end of the slot to make 16x boards fit (even if the slot will always operate as a 4x electric);
2. Reverse the fan in order to suck in fresh air. Add a dust filter, for obvious reasons;
3. Remove the steel duct in order to let fresh air flow in freely to the GPU fan itself. Eventually get a better fan for lower noise and better performance;

Once everything was planned, after a deep visual study of the enclosure, I started by tearing apart the whole box, and it was easy!

 

First of all I want to point out that the front black panel can be disassembled, but there's no clue on doing that. The front panel is there for aesthetic reasons and that's NOT the way you'll get in the enclosure.

Instead the access to the internals is achieved by removing four Phillips screws whose heads are placed in the center of the four table-facing rubber feet of the Sonnet Echo SEL. Once unscrewed the feet, the black cover of the enclosure comes off in a slid fashion, revealing the appealing and ready for disassembly internals.

Once removed the slid-cover, the first step to follow is to remove the interface logic board. This board can be uninstalled like a normal PCI board, except that an additional Phillips screw is placed right below the power supply jack. Unscrewing the two screws make extremely easy the removal of the interface board.
After removing the interface board I have disassembled the airflow duct (2 internal Phillips screws), the fan (4 external Phillips screws and the power connector to the mainboard) and the protective grid.

After that I have removed the mainboard by unscrewing 5 screws. Pay attention to the removal of the mainboard: a front facing blue LED is connected to the external status light by means of a optic fiber which appears to be fragile and susceptible to twists.

 

   

 

Disclaimer: a full disassembly is strongly encouraged to avoid damages for the following procedure. Even if the PCIe slot cutting can be performed with the logic board assembled it is much more comfortable to work on the detached one.

 

Once done with the disassembly, I have patiently worked on the PCIe expansion slot (the one marked with "x8 (x4)") to cut away the plastic edge to open up the slot for beefier boards.

As expected the plastic is rather rigid and strong, a choice driven by structural reasons. For this reason one should not be discouraged if at first the cutting tools are not really performing as expected. After having sawed the edge for a while, it began to dissolve in little plastic threads. With a precision work of cutter, scalpel and some miniature wood saw and rasps and a lot of time (~30 minutes!) I come with a quite professional result.

Of course I would have been able to use a Dremel-like tool, but I wanted to avoid electrical tools to limit the possibility of damaged to the PCIe slot pins. Approaching the cutting work by hand is, to me, the right way of proceeding.

Here some nasty pictures of the result.

 

    

 

The following steps concern the reassembly of the enclosure. I had to follow the disassembly steps in reverse order, except for reversing the flow direction of the fan, avoiding putting back on the air duct and, of course, installing the GPU in the modified slot.

 

 

The reversed fan seems to be pointing exactly in the right spot to always feed the GPU with fresh air. Cool (as a P1000)! 

 

Time to put back the sliding cover and the four rubber feet, and to enjoy the clean and professional look of the modded Sonnet Echo SEL with a Quadro P1000 installed in it. Definitely the highest performance density external GPU enclosure with TB2 (or optional TB3) connectivity around here.
My preccciousss...

 

[Slidspitfire]

Reserved (for tests and outcome)

[Jurrunio]

12 minutes ago, Slidspitfire said:

The 120W option was not really considered: packing so much power in a so small enclosure is, to me, not ideal.

Yeah, because 115w cards wont be small enough to get inside the case anyway.

[Slidspitfire]

2 minutes ago, Jurrunio said:

Yeah, because 115w cards wont be small enough to get inside the case anyway.

Yep, exactly what I meant! 

I have to admit: the ASL solution with that beefy 75W was a little bit scary in such a small form factor...

 

Slid

[Slidspitfire]

Up!

Added details on the unboxing (pictures to come, currently being uploaded)

[Slidspitfire]

Up!

Finally uploaded photos of the unboxing phase. 

And the step by step (and picture documented) guide!

[beirren]

i was wondering how this can work

sonnet offers spezial solutions for graficcards and even in this enclosures p1000 is not mentioned as compatible

does your solution work now with tb3 and 10.13.4?

[Slidspitfire]

Hello everyone,

It's been a long time since this project was completed and, well, got obsolete.
Thanks to Nvidia and Apple disputes and inability to get an agreement, the Nvidia Web Drivers which allowed me to run this GPU as an eGPU are part of the past.
Nobody knows wether in the future new Web Drivers for anything more recent than High Sierra will come out.
This is a sad story, both for the Hackintosh community as well as for anyone using a Mac as a developing machine and willing to test some (recent) CUDA code on the go on its laptop.

After some waiting I decided myself to return the Nvidia P1000 that served me and my girlfriend very well until when we upgraded to Mojave.
Amazon EU was kind enough to allow me to return the GPU with complete refund after 1.5 years (in EU the limit is 2 years, but you've got to have a broken part... Good for warehouse dealmakers, I suppose ).
I bought on eBay a used WX4100, the only low profile and short AMD GPU fitting the Sonnet Echo SEL, for about 220€ and, surprise surprise, it worked!
I just had to perform the automatic install provided in this GitHub repo link (just follow instructions), and I got back my ASAP-eGPU.
In addition a really interesting feature like GPU ejection is now fully supported even on my TB2 MacBook 2015, avoiding kernel panics in case of accidental unplug.

For anybody looking in this thread, please get in touch if you happen to need more information!

Yours,

Slid

[dizmo]

Wouldn't it have made more sense to buy a laptop with a more powerful GPU?

[TheRandomness]

3 hours ago, dizmo said:

Wouldn't it have made more sense to buy a laptop with a more powerful GPU?

This is more fun.