Huge Server project; strange parts $$$$; losing my mind

Phas3L0ck · May 3, 2021

Budget (in USD): Estimated cost was $15,000 -- project on track to remain below $10,000 (for now). *I don't have nearly enough funding to finish it*

Programs or workloads: 3D design with AutoCad/Revit, flight simulation, VMware, and general office stuff.

Other details: Some parts may need to be upgraded or modified; chassis is long overdue for a complete redesign, but critical mods can be done cheaply.

I started an insane server project back in 2017 to end the need for another desktop or machine rebuild, and while it's mostly done, there are still a few things missing and modifications that need to be done which, without new funding or decent sources, is really starting to suck. Here's what the problem currently looks like:

- planning to order 16x DDR4 32GB ECC 2400MHz RAM modules; prices are in flux and there are many distributors, so I can probably strike a deal to get all modules for around $1,600 (after sales tax)

- Rear I/O panel needs a redesign very badly (try to imagine Dell screwing Supermicro) I can design a new I/O plate myself by reverse engineering the existing one, but getting it manufactured is a problem of its own.

- Power supply upgrade; required for GPU upgrade, but can't be mounted until rear I/O modification is done. A full replacement is not possible without a complete chassis redesign.

- GPU upgrade... **slams head** Don't... even... ask... (but the cost should be under $600 - we'll see)

- Serious re-arrangement of storage hardware (You would not believe how this server has been living!)

You can find what drives I use on my profile page, but what you don't know is the story involved.

First off, my chassis is super weird, so I had to pull a Hewlett-Packard and mount my operating SSD next to the PDB behind the power modules.

Second, while I could have had the best RAID array for this system, I totally panicked at a critical point in time and bought hard drives instead of the proprietary SAS adapter I need because I found a deal on hard drives and I thought they were being bought up, so I had to get them before they were gone, but it turns out the company had even more in stock that weren't listed until after I bought mine! Then I didn't have enough money to buy the special OCP SAS adapter from Tyan, and by the time I did again they completely stopped making and selling them, which is the worst thing that can happen as it was a BTO part for some odd reason, and NO ONE has it at all! (It was a M7076-3108-8I in case anyone is wondering)

I had to get something to use my SAS drives, so I got their $H1tty little SAS3008 mezz card (which is still in production... yuck...) and now my drives are running the most awful excuse of an array in RAID-10 (my goal was RAID-6).

For the past year, in desperate searches to find a suitable alternative, the closest I ever got was some weird OCP mezz from Gigabyte, but even if I could find a place to buy their parts, (which is very hard) their mezzanine is upside down like the traditional OCP card, but my socket has components below it and will only support Tyan's proprietary version with the chips and connectors on top. And then there's the QCT-type SAS3108 adapter from Quanta, but obviously you can't use that on an OCP socket.

There is one other way to get past the issue, but the cost has me wondering if there's any benefit... Tyan has a custom NVMe mezzanine, model M5539-2E. But here's the catch; no one has it in stock, they no longer make it, and they only way to get it is to buy their early-generation Xeon-D NVMe server GT62-B5539-D41, outlined here: https://www.storagereview.com/review/tyan-gt62b-b5539-1u-server-system-review

Attached below is a close-up (as good as I can get) of the part itself. The only reason I even know about it is because I found it at random while digging through their old system catalogs.

While this NVMe mezzanine has its benefits, in order to get it, the system it sits inside costs nearly $2,000 before shipping and sales tax!

If anyone out there can figure out some way to solve this problem of strange parts, please let me know!

Windows7ge · May 3, 2021

What physical server are you working with? I assume something proprietary given you're in need of a OCP card?

My solution here would be to opt for a standard PCI_e adapter. You can buy NVMe capable SAS HBA cards.

Phas3L0ck · May 3, 2021

7 hours ago, Windows7ge said:

What physical server are you working with? I assume something proprietary given you're in need of a OCP card?

My solution here would be to opt for a standard PCI_e adapter. You can buy NVMe capable SAS HBA cards.

All the specs are on my profile page. Tyan GN70 7086.

For hardware, and OCP device is necessary to make effective use of the bandwidth as it's routed.

But for software, newer PCIe cards won't work (no drivers are being made for windows 7 for anything newer than the SAS3108) and besides that wouldn't solve my problem. Sure I can get a PCIe card for a RAID array, but that doesn't answer my question of what do I do with the remaining OCP socket? (and no, it can't be ethernet.)

Windows7ge · May 3, 2021

17 minutes ago, Phas3L0ck said:

All the specs are on my profile page. Tyan GN70 7086.

For hardware, and OCP device is necessary to make effective use of the bandwidth as it's routed.

But for software, newer PCIe cards won't work (no drivers are being made for windows 7 for anything newer than the SAS3108) and besides that wouldn't solve my problem. Sure I can get a PCIe card for a RAID array, but that doesn't answer my question of what do I do with the remaining OCP socket? (and no, it can't be ethernet.)

LGA2011-v3? That new enough to support Windows Server 2016 and likely Windows Server 2019. Personally I'd have gone the hypervisor route and used PROXMOX or vSphere/ESXi instead of sticking to Win7. With a hypervisor you could virtualise and use what's known as paravirtuaization. This gives near native speeds for VM guests meaning the hypervisor could support a newer HBA/RAID card and the VM doesn't have to. It just needs to support the paravirtuaization driver (which Win7 should). This works for both storage and networking and works fantastically.

As for other suggestions...hmn, do OCP to PCI_e adapters exist?

Is there a standard PCI_e variant of the card still available?

Riser cable the adapter over to the RAID card.

Phas3L0ck · May 3, 2021

On 5/3/2021 at 2:22 PM, Windows7ge said:

LGA2011-v3? That new enough to support Windows Server 2016 and likely Windows Server 2019. Personally I'd have gone the hypervisor route and used PROXMOX or vSphere/ESXi instead of sticking to Win7. With a hypervisor you could virtualise and use what's known as paravirtuaization. This gives near native speeds for VM guests meaning the hypervisor could support a newer HBA/RAID card and the VM doesn't have to. It just needs to support the paravirtuaization driver (which Win7 should). This works for both storage and networking and works fantastically.

As for other suggestions...hmn, do OCP to PCI_e adapters exist?

Is there a standard PCI_e variant of the card still available?

Riser cable the adapter over to the RAID card.

Never gonna happen! The only reason I even have Windows Server 2008-R2 (Win-7 server) is because of the new ability to integrate GPU acceleration and native x64 hardware. If it wasn't for that, I would still be using Windows XP. Too many things have not only changed, but gone horribly wrong in the past few years, and having experienced both Windows 8, 8.1, and 10 (by a great deal of misfortune) everything that ever was or ever will be good with Windows is DEAD!!! This is the last one. Last system I build, last OS I use as my machine platform. There can't be any others. Too many great things have died after a pitifully short life. I can't take any more losses. And NO, I can't do virtualization because I NEED bare metal!!! In order to run the most basic operations, I need ACCESS to EVERYTHING. NONE of this rambling about software does anything to solve my hardware dilemma.

Conversion from an OCP socket to a traditional PCIe slot is possible, but even if it did exist, (which, I've looked for and it doesn't) I still have limited space inside the system.

PCIe variant? Well duh! The LSI 9631-8i is common in several AIB shapes, but that's only a half-solution and doesn't answer my need for an OCP device.

Riser cable? First off, limited space. Second, moot point.

Maybe I should have spelled this out in the beginning, but the only thing I came here for is advice on finding a suitable alternative OCP device, at least some feedback on the one I described that is available.

Do you know of anyone here who maybe knows more about OCP devices?