The Pure Solid State Server Build Log

[Windows7ge]

Over the next couple of days I'm going to be building my next server and I thought anybody who has a similar interest might like to come along for the ride.

 

(The pictures are 4208x3120 so you can view them in a new tab for a larger image)

There is so much equipment I found it difficult to get it all in one shot. Here is a run down of the core components:

 

CPU: 2x Intel Xeon E5 2698 V3

Motherboard: Supermicro X10DRi-T

RAM: 2x NEMIX 64GB 2400MHz ECC RDIMM

PSU: Athena Power AP-RRP4ATX6808

HBA: 3x LSI 9201-16i (live product page N/A)

5.25" drives: Athena Power BP-SAC1425AVL12

Chassis: iStarUSA D-410

CPU Cooler: 2x Noctua NH-D9L

 

The remaining small pieces I will go over as I build but this is an immediate run down of the bulk of the system. Everything except the processors were purchased new. I have actually not verified that that CPUs work properly. I'm going off the sellers word so we will get to find out together if it's going to POST at all.

 

For right now I'm going to start by installing the 5.25" drive caddies.

[8uhbbhu8]

Oooooo

[JacobFW]

 

Very cool project.  For work?

 

 

[Windows7ge]

21 minutes ago, JacobFW said:

Very cool project.  For work?

No, just an extreme hobby. I don't doubt I could find use for it in a future occupation though so kind of nice to have.

[Windows7ge]

3 minutes ago, CUDAcores89 said:

Will you use it to take over the world?

If it POSTs...we'll see.

[Windows7ge]

So I have half the bays installed and I wanted to stop to mention why I bought these ones specifically. They don't use the standard SATA interface.

That port is known as SFF-8643 and it's usually used for SAS interfaces which these caddies actually support. I can plug in SAS or SATA devices. The reason I went with these specifically is because 10 cables to manage is a heck of a lot easier than 40.

 

I also have to say the quality of the chassis could be better. (What did I reallyt expect though)

It's not super noticeable in the picture but it's more noticeable in person. There are uneven gaps in between the caddies. The gaps themselves don't bother me. In fact it just makes for better airflow to the components behind the caddies. I hate the fact they're crooked & uneven.

 

Also it appears this one caddie arrived damaged. Can you see how this sled is slightly recessed compared to the others?

 

It's because on that one this little nub of plastic had broken off

If that bit comes off nothing else keeps it from moving backwards. Although it sucks I'm not too concerned because once I put a SSD in it it should come back out where it is suppose to line up.

 

Alright back to work.

[Windows7ge]

If I can allow myself to ignore the gaps I have to say it looks quite nice. Intense even.

 

Opening up the back I have to say I much prefer the view from behind. The colored fan wires mixed with black & chrome is beautiful.

 

The next things I'm going to take care of are changing out the two fans (left & right) with some Noctua NF-F12s. They are just a couple that I have on hand. I haven't decided yet if I'm going to grab some of the Industrial 3000RPM fans to put in here in the future. We'll see how the temps are and go from there.

 

I'm also going to go ahead and install the PSU.

[Windows7ge]

So I had never worked with a mini-redundant PSU before and I have to say using it in this chassis scared me about four times because each time I thought "OK, I need to remove this bracket..." & "OK, I need to screw it in there..." It ended up as "The bracket is permanently attached..." & "The screw holes don't line up..." or "This extra bit of metal is in the way".

 

As it turned out I was just doing everything wrong. I did have to remove a bracket which was an inconvenience because it has this little button attached to it. I'm not sure if it's like a reset button for the PSU or what.

 

This is the current state of system.

It's finally starting to look like something just a little bit. I went ahead and ran the SSF-8643 to SFF-8087 cables because if I waited until after I got the fans in running them would have been impossible.

 

That power supply took up a lot of time. I'm going to stop for the night. If I don't get too many interruptions I might get to power it on for the first time tomorrow.

 

We'll see.

[Windows7ge]

The next thing on the job list is to prepare some of the components for the motherboard. Starting with the RAM I ordered the EVERCOOL EC-MC-CO.

They're very basic heatsinks. They may not be necessary but I know some RDIMM memory gets quite hot. Some does not. Being on the safe side I got them.

 

Unfortunately the tape that comes with them is extraordinarily thin and testing it the heatsink makes no contact with the memory chips.

 

So I ordered some of this.

This is 5589H-10 which could be better known as a thermal pad material. One side appears to be sticky the other is not.

 

I wanted some that was about 0.75mm thick but I couldn't find any. The best I could do was either 0.62mm or 1mm and I wanted to make sure contact is properly made. So we will find out very soon if 1mm is too thick. I hope the material is compressible.

[Windows7ge]

I think they look quite nice.

 

They did turnout a little V shaped and that's after some modification. It was worse.

You can also see the bottom most chips aren't really covered but I'm not too worried about them.

 

To get this fit I had to remove the small tabs at the top and bottom of the heatsinks

(left - modified, right not-modified)

 

Due to the thickness of the TIM those tabs prevented the heatsinks from interlocking and sitting on top of the stick properly so I removed them.

 

Next up is the TIM for the CPUs I'll get that started and post back.

[WereCatf]

20 hours ago, Windows7ge said:

CPU Cooler: 2x Noctua NH-D9L

Nitpick: those have fans on them, meaning that they are not solid-state and thus your title isn't quite accurate!

[Windows7ge]

9 minutes ago, WereCatf said:

Nitpick: those have fans on them, meaning that they are not solid-state and thus your title isn't quite accurate!

Solid State as in storage. Also if you look around a bit more there are many more fans than just those.

[WereCatf]

11 minutes ago, Windows7ge said:

Solid State as in storage. Also if you look around a bit more there are many more fans than just those.

I noticed. Woulda been a lot cooler, if you'd managed to make a truly solid-state server!

[Windows7ge]

1 minute ago, WereCatf said:

I noticed. Woulda been a lot cooler, if you'd managed to make a truly solid-state server!

That would be quite the challenge because a lot of things in this server are going to get very hot.

 

Also there's always the PSU. Although there are some server PSU's that don't have fans they're specifically designed to go into chassis that provide the cooling from chassis fans so even if I went out of my way to create a true solid state server one component or another would suffer greatly.

[Windows7ge]

Moving onto the CPUs the TIM I'm using is Panasonic Graphite EYG-S1431ZLAA

 

It's basically a bulk material version of the IC Graphite material LTT featured except it costs a lot less, performs just as well and you can cut it to whatever dimensions you want.

 

IT is fragile though so there are limitations to how intricate the cuts are you can make. This was attempted with an exacto knife.

It rips easily, so the edges don't look great.

 

To explain the small tabs around the perimeter it is to keep it from moving around when I put the cooler on.

 

Next we can start assembling the motherboard.

[Windows7ge]

So this is the Supermicro X10DRi-T it is a dual socket LGA2011-3 motherboard using the C612 chipset. One of it's greatest features is a built-in dual port Intel X540 10Gbit NIC.

Something to note is the socket CPU cooler mounting holes. This is known as narrow-ILM and you need the mounting gear to use it.

 

The NH-D9L does not come with this equipment.

 

However a simple e-mail to Noctua will get you the narrow-ILM mounting gear. They didn't charge me anything.

 

An additional component I'm adding because I only have two sticks of RAM are the modCover DDR Port Silicone Rubber Dust Cover

They're soft, stretchy, & flexible. I only bought them because the RAM is understandably expensive so I only bought 2 sticks. The system is going to have unused slots for a long time and this will just keep all the dust out.

 

and with that the motherboard is finished.

 

Next up is installing the I/O shield, standoffs, motherboard, and expansion boards.

[Windows7ge]

Interesting. The I/O shield seems to have this metal mesh padding on it. I can only assume it's to keep out interference.

 

We have a problem. Three of the standoff holes don't have pegs underneath them. Damn it iStarUSA.

 

I noticed rummaging though my odds and ends of computer gear that 2 standoffs screwed into each other is basically the same height of the server chassis standoffs.

 

There we go. This should solve it.

If I ever go tugging on the board for any reason I'll just have to be careful in those areas.

 

Unrelated to that issue I ordered a StarTech SATA Removable Hard Drive Bay for PC Expansion Slot

As if forty 2.5" bays isn't enough I want the boot drive away for the storage drives so this lets me mount a 2.5" drive in a rear PCI slot.

 

And with that everything is installed.

All that's left is to plug everything in, cable management, and see if it POST's.

[lieder1987]

This is fascinating, thanks for doing this.

[Windows7ge]

Alright, everything's plugged in. I just have a couple of cables I need to tuck in somewhere but before I do that it's time to see if it will POST because if it doesn't we'll be ripping it apart again to troubleshoot.

Here we go...

[Windows7ge]

We have a POST! @CUDAcores89 Maybe I will take over the world. Starting with the forum. 

 

Both 64GB sticks of RAM & CPU's are detected.

This didn't come without something going wrong. The cheap Microsoft keyboard I was going to use here mysteriously died causing me to believe every USB port on the server was dead. Great omen.

 

Now the RAM is rated to operate at 2400MHz but it got underclocked to 2133MHz so I'm going to need to fix that. Servers don't have X.M.P. profiles so I have to do it and the timing manually. Shouldn't be too much work.

[Windows7ge]

16 minutes ago, CUDAcores89 said:

I hope you have your RAM installed correctly. Because you only have two DIMMs install one for each cpu. Do not install the ram all on one cpu because then more information will have go to through the QPI (looks like you did the proper thing).

 

Also is this RAM RDIMMs or LRDIMMs? These two memory standards are not iterchangeable so if you have RDIMMs, you can only add more RDIMMs and if you have LRDIMMs, you can only add more LRDIMMs in the same system. Looks like they are RDIMMs but im guessing you never plan to install 2TB of RAM in this thing so that doesnt matyer as much.

 

Good job on the build, looks quite expensive.

This isn't my first dual socket system so I'm aware of which slots should be occupied per socket. It's also marked on the sink screen and mentioned in the manual.

 

They are RDIMM. I was looking at the LRDIMM (128GB sticks) but what research I did on the memory concluded that when using LRDIMM there's a trade off. To double the memory you lose some performance. That wasn't worth it. Current thoughts are that 512GB is a milestone goal and from there I may or may not go for 1TB.

 

I have no idea what it cost. I just bought the parts over a period of 5 months until finally I had everything I needed.

[Windows7ge]

It is now past midnight so I'm done for today.

 

I currently don't have the planned boot drive for the server but I'm going to throw a temporary install of Windows Server 2016 on an spare SSD just so we can do some initial testing. If you guys have any tests you'd like me to run on this thing (like Cinebench) besides a quick burn-in let me know. I'll go over them later.

[Windows7ge]

That's a lot of threads.

With the server apparently stable I tried to run a stress-test with AIDA64 but ADIA64 couldn't handle it and kept (not responding). I'll have to find a different tool.

 

Now that the server is assembled and booting properly next comes the networking.

So these are both STP Cat6a cables. According to the labeling that runs the length of the wire it's SFTP cable specifically.

As unnecessary as shielded cable is in my application I've decided to use it for no more reason than I wanted to try it out. I've never played with STP cable before.

 

The plan here with these is to hopefully utilize Windows native SMB3.0 Multi-Channel to create a single virtual link capable of 20Gbit.

If it works and if I manage to saturate it I plan to bump it up to 40Gbit by buying a dedicated Intel X540 NIC.

 

Right now I need to run these two cables and make two cables using Cat5e patch cable for the 1Gbit NIC (local network/internet access) & then dedicated IPMI so I can manage the server without having to sit in-front of it.

 

Unfortunately this will require me to climb up into the ceiling to run the cable and I don't feel the strength to do that today. I also have to work tonight so we'll see how things go tomorrow. I'll get everything plugged in, setup & running (network-wise) and come back with performance results.

[Ben17]

2 hours ago, Windows7ge said:

That's a lot of threads.

With the server apparently stable I tried to run a stress-test with AIDA64 but ADIA64 couldn't handle it and kept (not responding). I'll have to find a different tool.

 

Now that the server is assembled and booting properly next comes the networking.

So these are both STP Cat6a cables. According to the labeling that runs the length of the wire it's SFTP cable specifically.

As unnecessary as shielded cable is in my application I've decided to use it for no more reason than I wanted to try it out. I've never played with STP cable before.

 

The plan here with these is to hopefully utilize Windows native SMB3.0 Multi-Channel to create a single virtual link capable of 20Gbit.

If it works and if I manage to saturate it I plan to bump it up to 40Gbit by buying a dedicated Intel X540 NIC.

 

Right now I need to run these two cables and make two cables using Cat5e patch cable for the 1Gbit NIC (local network/internet access) & then dedicated IPMI so I can manage the server without having to sit in-front of it.

 

Unfortunately this will require me to climb up into the ceiling to run the cable and I don't feel the strength to do that today. I also have to work tonight so we'll see how things go tomorrow. I'll get everything plugged in, setup & running (network-wise) and come back with performance results.

good luck not criticizing but just next time use solid core cable in a loft/patch panel. as it tends to be more reliable than stranded core from my research  (that one you have) nice work so far though!  

[Windows7ge]

13 minutes ago, Ben17 said:

good luck not criticizing but just next time use solid core cable in a loft/patch panel. as it tends to be more reliable than stranded core from my research  (that one you have) nice work so far though!  

It would be stranded based on its flexibility. At my house here though this install won't be in the wall or include a patch panel. It just drapes down the wall leading up to a drop ceiling which next to it is the effective network closet where there is no patch panel and it just plugs directly into the 10Gbit switch. Not professional at all. The Cat5e cable will be solid core though if that means anything to you. So reliability here isn't a concern as the wires aren't going to be constantly unplugged and re-plugged or moved around very much. I'm pretty sure if I wanted solid core I would have had to buy it in bulk and terminate it myself. UTP Cat5e is easy enough but STP Cat6a would be a bit challenging to do right so I just bought a pre-made cable.