192 Core Dell PowerEdge C6145 Server restoration project

[Fast_N_Curious]

96 Cores? Not bad for under $150... Now multiply that by two! Yes, just today, I ordered a second 96 core server to compliment the first one. That means we have 192 cores at our disposal. 

 

First of all this project is just for fun and for benchmarking... It may also provide features on the home network but that is yet to be determined. I have a wide verity of options with this machine. I wanted to go with AMD, and since I loved the Phenom II as a consumer chip, I had to take a to look at AMD based servers that came stock with the Opteron 6100 series chips. This was the last family of the K10 - very similar to the Phenom II architecture, before the switch in the 6200 series to the horrifically under-performing FX line of processors. My intent is to fit it with as much 1600MHz memory as possible (it has 32 slots) and fit some solid state drives as well, plus some mechanical drives for storage.

 

 

 

Dell write up on the server:

 

Designed with your needs in mind With complex problems to solve in mapping, visualization, simulations and rendering, the number of cores, amount of memory and bandwidth can influence whether you see results first or last. The Dell™ PowerEdge™ C6145 server is the only server in its class with the performance of two 4-socket AMD Magny-Cours processor-based servers in a 2U. It has up to 128 cores, 1TB memory, and 40Gb/second I/O per card slot. It has six PCIe Gen 2 x16 slots, two x8 mezzanine and two dedicated x16 host interface card slots (HICs), so you have 10 total slots to flex with your needs. Best of all, it’s all in a hyper-efficient shared infrastructure 2U, so it saves space, weight and energy. Satisfy the need for speed The pure volume of calculations calls for performance, all the way from the processor to the I/O bandwidth. The PowerEdge C6145 is one of the highest performing 2U rack servers ever, with two 4-socket AMD Opteron® 6200 series processors. These processors have up to 84 percent higher performance with up to 73 percent more memory bandwidth.1 It’s not just the FLOPS in the PowerEdge C6145 server that make the difference. It can also accommodate up to 1T of memory. Satisfy the need for efficiency

 

The PowerEdge C6145 server packs two highperformance servers in the same 2U space, while its shared infrastructure design can reduce the number individual fans by 1/4 compared to a traditional 2U, and the associated power to cool. Those two 4-socket servers also use less floor space, cabling, and racks. At the processor level, the AMD Opteron 6200 series uses half the power per core, requiring 2/3 less floor space and up to 2/3 lower platform price.1 Select Samsung’s® latest low-voltage high-density DDR3 memory and you can save more than 70% in energy over previous generation memory solutions. Mix, match and maximize connections Cores and memory just aren’t enough without the I/O bandwidth to quickly compile data and results, and connect to other resources.

The PowerEdge C6145 server ramps up the I/O with 40Gb/second throughput per x16 slot. Each node has five slots: three PCIe Gen 2 x16 slots, one dedicated x16 HIC slot and one x8 slot for a mezzanine card. With two nodes per server, 10 total PCIe slots, you can mix and match RAID controllers, Infiniband cards, 10GE cards, and HIC cards all in the same server. You can even drive your workloads to the cutting edge. For example, combine the PowerEdge C6145 server with the PowerEdge C410x PCIe expansion chassis and you can double the server to graphics processing unit (GPU) ratio to 1:8 to create a number-crunching powerhouse. Global services and support Dell is dedicated to simplifying IT, and Dell Services can help you manage the complexities of growing and maintaining your scale-out  environments. Dell’s broad portfolio of planning, implementation and maintenance services can help accelerate your IT initiatives and help grow your business. Options include, but are not limited to,  consulting services to help you optimize your data center, 
custom rack integration specific to your IT environment, and expert-level solution support with Dell Online Self Dispatch.
PowerEdge C6145 • 2x the performance and density of a traditional 2U • Up to 281% more  performance per U compared the HP Proliant DL980 G7, in a quarter of the rack space2 • Shared infrastructure uses less floor space, power and cooling • 5 PCIe slots/server node, 10 total

 

Using this as my project log. I will update you once it arrives with new information.

 

Day Two - 4/15/20

- Ordered a second Dell PowerEdge C6145 server with an additional 96 cores of Opteron 6180 SE number crunching power, effectively doubling our performance and bringing the total core count to 192.

-Researching PCIe GP-GPU enclosures to boost performance across the board and engage a high performance GPU component solution to this build.   Focused on the Dell CloudEdge C410X or similar 3U chassis for up to pcie16 GPUs.

-Spec'd out the 1400W PSU... which is 220v only. So I've ordered four 110v 1200w PSU's that will work with the houses existing electrical system.

-The main focus of this project is to reveal the full potential of two PowerEdge 6145's and then apply that computational power to something meaningful in the real world. 

[Pickles von Brine]

Dope. 

[Firewrath9]

the power draw on that must be huge o_O

hopefully you live somewhere with low energy costs.

[Fast_N_Curious]

41 minutes ago, Firewrath9 said:

the power draw on that must be huge o_O

hopefully you live somewhere with low energy costs.

 

No doubt the twin 1400 w power supplies will pull some serious power. 

 

But it just so happens that we recently upgraded the electric in our house to 200 amp service. 

 

New server can practically have it's own dedicated circuit breaker. lol 

[Fast_N_Curious]

I actually need to get the lower rated 1200 watt PSUs because they work with the existing electrical system in my house. The 1400W PSUs that come with the new server are 220 v only. 

 

Welcome to my World.

 

[Fast_N_Curious]

I am seriously considering a second Dell PowerEdge C6145 (with 8 more Opteron 6180SEs CPUS) and pairing both servers (4 nodes) with a Dell c410x 3U GP-GPU chassis. This will allow me to harness some serious computational power in taking advantage of all 16 PCIe GPU slots, 4 per node. No doubt I already have the most horsepower in the area but the pursuit of performance is an evolution, ever onward and ever higher. 

 

Here is a quick overview of the 3U PowerEdge C410x.

 

PowerEdge C410x • Up to 16.5 TFLOPs in 3U
• Flex from 1-8 servers connected to 1-16 PCIe devices
• 25% greater density, 31% lighter, 11% fewer racks and switches than its closest competitor • Individually serviceable PCIe modules, power supplies, fans

 

Designed with your needs in mind The results of your research, mapping, rendering or model simulation have the potential to impact the lives of millions. Yet, the required processing of massive amounts of data and computations takes time. Meanwhile, application and computing infrastructure demands increase while the budget remains the same. You can see results faster, while increasing efficiency with the Dell PowerEdge C410x PCIe expansion chassis. Loaded with NVIDIA® Tesla™ M2050 computing modules, it brings 16.5 TFLOPs of computing throughput in single precision performance to your fingertips in a 3U. The PowerEdge C410x PCIe expansion chassis does not have CPU or memory. It provides “room and board” for 16 PCIe devices to connect to 1-8 server host nodes. So, as your needs grow and change, the PowerEdge C410x PCIe expansion chassis flexes with you. See your results faster Increase graphics processing unit (GPU) to host ratios so you can see your results faster without over-investing in CPU cores. Off-load more parallel processing to up to 16 GPUs without having to buy another server. With a wide range of flexible server-to-PCIe device configurations, you can set the right ratio to maximize application performance.

 

 

Now the question is finding the rare c410x and populating all 16 PCIe slots with high performance GPUs. 

 

I have a budget of about $4,000 left for this build. I think I can pull it off. Make no mistake when finally set up and configured it will not just look "pretty", we will be maximizing the theoretical performance and pushing all 192 Cores and it's infrastructure to the limit. Yes, you read that right. That's 192 cores in total, between both servers!

 

EDIT: 2nd Server ordered! Effectively doubling our overall performance!

 

How about 2TB of RAM? Do you think that's enough to handle the 20 tabs I have open right now in Microsoft Edge? 

 

 

[jSON]

I know this is REALLY late, but I thought I'd add that you weren't the only one to get one of these from that eBay seller

I have ASSET#2013366

 

I opted to swap for a set of 4 Opteron 6366 HE CPU's for the power and heat savings, sold the original set for $40. Honestly was the best $ spent lol. US $165.00 after shipping.

Odd though the seller isn't on eBay anymore. Sucks, they had good deals.

 

Currently dabble in CPU mining (GPU mining is heresy), getting north of 4000H/s @ 300W give or take.

Otherwise I run Unraid Pro and use it for VM's.

[Fast_N_Curious]

4 hours ago, jSON said:

I know this is REALLY late, but I thought I'd add that you weren't the only one to get one of these from that eBay seller

I have ASSET#2013366

 

I opted to swap for a set of 4 Opteron 6366 HE CPU's for the power and heat savings, sold the original set for $40. Honestly was the best $ spent lol. US $165.00 after shipping.

Odd though the seller isn't on eBay anymore. Sucks, they had good deals.

 

Currently dabble in CPU mining (GPU mining is heresy), getting north of 4000H/s @ 300W give or take.

Otherwise I run Unraid Pro and use it for VM's.

lol @ heresy comment

 

I ended up getting a C4130 and 4 Tesla K80 compute GPUs (which is actually two GPU dies on the same PCB), its a really cool machine. Could be used for deep learning and alike, but right now it sits idle until I can find a good use for it. Much better than what I mentioned above.