Jump to content

[Build] HCI development/test stack

Dark

Lately I've been working on an HCI hardware stack for development testing and thought it would be good to share what I'm doing while hearing others thoughts/suggestions.

 

My goal for this was to create a relatively inexpensive and compact 3-node cluster that still offers some flexibility along with creature comforts (IPMI). This cluster will allow me to test new deployments, configurations, and upgrades before considering them in a production environment.  

 

For now the cluster will get VMWare 6.5, VSAN 6.6, and NSX (probably NSX-T)

 

Here's what I've got so far--

 

Motherboard: Supermicro X9SRW-F

Processor: E5-2640

Cooler: Dynatron R24

Memory: ECC DDR3 1600 16GB (64GB per node)

PSU: Supermicro 560w

Storage: Intel 520 180GB SSD for cache, Samsung 850 evo 500GB SSD for capacity (per node)

Storage rack: Kingwin 4x 2.5 hot swap chassis

Boot drive: Sandisk 16GB USB stick

 

___________________________________________________________________________________________

 

*9/14/17--

Shipped one of the three motherboards back due to a damaged PCI connector.

 

*9/17/17--

Started creating the 'open' style frame for the boards, drive chassis, and power supplies.

 

I6eGCLw.jpg

 

S3pKNf3.jpg

 

6yDJu4u.jpg

 

w0StWTv.jpg

 

U28EUEC.jpg

 

ai3kevT.jpg

 

Vf5m5Fg.jpg

 

AoFj0VH.jpg

Link to comment
Share on other sites

Link to post
Share on other sites

While I'm still waiting for the 3rd motherboard and sata cables, I'll likely create some sort of base for this to sit on that will be wider than the 3 power supplies (it's surprisingly stable as it is though).

 

I'd to hear how otherwise would build this differently, if it appeals to anyone else. lol

Link to comment
Share on other sites

Link to post
Share on other sites

18 minutes ago, Dark said:

While I'm still waiting for the 3rd motherboard and sata cables, I'll likely create some sort of base for this to sit on that will be wider than the 3 power supplies (it's surprisingly stable as it is though).

 

I'd to hear how otherwise would build this differently, if it appeals to anyone else. lol

This is really cool. you thound make a way to add some more to the sides. It's hard to tell, can you fit a full sized graphics card in the pcie slot without it interfering with the other board? If not, maybe you could make them further apart so you can fit one.

Build:                                                                          

Intel Core I7 6700k (clocked to 4.6ghz)

Patriot Viper 3000mhz ddr4

ASUS maximus 8 hero (I hate this mobo)

Evga gtx 1080 superclocked

250GB Samsung 850 evo

2tb WD Black 7200rpm

Ek watercooling kit a240g with 360 expansion pack

Primochill Vue Red fluid

Corsair Graphite 600t (white)

 

You thought "Gee, Donald Trump sure has a great build!"

Link to comment
Share on other sites

Link to post
Share on other sites

28 minutes ago, DarthSmartt said:

This is really cool. you thound make a way to add some more to the sides. It's hard to tell, can you fit a full sized graphics card in the pcie slot without it interfering with the other board? If not, maybe you could make them further apart so you can fit one.

Thanks!


Supermicro has a couple of different style risers (2x 16x, 4x 8x, etc) that would mount on the two 1x and proprietary slots sandwiched next to that 16x slot.  Those risers would allow for a video card but it would protrude out the side which would require bracing.  But definitely an option!

Link to comment
Share on other sites

Link to post
Share on other sites

No substantial update yet, still waiting on the replacement board (shipped out today and should be here by the weekend).

 

Did get the power and sata cabling knocked out.

 

ybuVuDy.jpg

Link to comment
Share on other sites

Link to post
Share on other sites

Went ahead and installed the drives for the two nodes, loaded up ESXi, and setup vcenter.


I was a bit shocked that it let me start the vsan configuration with only two nodes (there's not 'witness' or fault node with just 2).  I'll wait until I'm at 3 nodes before benching the array.

 

iT4vLfT.png

Link to comment
Share on other sites

Link to post
Share on other sites

Not a whole lot of exciting news yet. 

The replacement board came in and I finished building the cluster.  I setup a Win10 vm and ran quick ATTO disk benchmark (vaguely recall ~200MB write, 1600MB read).  When I bring the cluster back online I'll pull off the screen caps and share them here.


What I'm now waiting on is a 10Gb network setup.  Received a Ubiquiti US-16-XG switch and I've got some 10Gb SFP+ cards on order.

 

Uxb41GK.jpg

Link to comment
Share on other sites

Link to post
Share on other sites

Create an account or sign in to comment

You need to be a member in order to leave a comment

Create an account

Sign up for a new account in our community. It's easy!

Register a new account

Sign in

Already have an account? Sign in here.

Sign In Now

×