Build Log Data Server 'Baldur'

[Ahnzh]

I can't wait for the first Aluminum parts to arrive. I will start with building the HDD case though, since i'm not 100% sure which dimensions i want to have for the rest of the case  mATX, ATX, EEB, dunno have to see. I mean i am doing a lot of work now and want to expand it later possibly without spending an enormous amount of additional time

[MoBo]

ohh my gawd this is amazing the storage is sexy :wub:

btw can you talk german 

wenn ja dann hi

[Ahnzh]

ohh my gawd this is amazing the storage is sexy :wub:

btw can you talk german 

wenn ja dann hi

ich kann durchaus

 

but i refrain from it out of respect for all the people here that cannot.

 

I'm setting up my backup server at the moment. saving data on it, then I need to get it into a datacenter.

 

It will synchronize with my server later on.

[Anthony10]

ich kann durchaus

 

but i refrain from it out of respect for all the people here that cannot.

we appreciate it thanks!

[MoBo]

same gotta practice my english some more

[Ahnzh]

So as a small update: I pretty much changed everything...

 

 

Now it's going to be :

3x LSI SAS 9300-8i Host Bus Controller Cards

2x Xeon E5 v2 2630

128GB total RAM (16x8GB  ECC RAM - Hynix)

X9DAX-iTF Dual Processor Mainboard by Supermicro with 2x 10GBase-T on it to finally get 10Gbit into play

 

The Hardware that i already received will be built into a calculation node for my private cloud. 

 

And i got a pic to share with you ;D

 

 

And I've got a question: should I connect all HBAs to a single CPU or is it better to connect a single one to CPU1 and 2 to CPU2? The Intel X540 10Gbase-T chip is connected to CPU1. Still, when I think about it, the data from the HBAs will be sent from CPU2 via the QPI Link to CPU1 and then gets sent via LAN. Sounds fishy from my point of view. Does anybody know more about that? How does QPI compare to sending it through the FSB?

[OverachievingSnowfall]

I don't really understand what you just said, but that's a lot of storage sitting there and a lot of processing power too.  I can't wait to see the benchmarks.  Subbed.

 

 

Definitely subbed...

[Ahnzh]

I don't really understand what you just said, but that's a lot of storage sitting there and a lot of processing power too.  I can't wait to see the benchmarks.  Subbed.

 

 

Definitely subbed...

I'm a bit distracted so it's understandable if you didn't understand what I wrote, my GF is pregnant and the birth is just a question of 48 hours. She's making my live a living hell. Today we were in a movie. We said 'ok it will be the last movie for a long time'!

 

It ended with she first destroying my balls during an uterine contraction and then after I shouted out loud she started insulting me as an impotent fucker in some kind of screaming voice! In the middle of a movie! DEAD SILENCE! GOD please let this end!

 

 

I suffered so much already!

[Ahnzh]

I don't really understand what you just said, but that's a lot of storage sitting there and a lot of processing power too.  I can't wait to see the benchmarks.  Subbed.

 

 

Definitely subbed...

Oh and i didn't sleep for 48 hours already PRE birth! fckk

[Ahnzh]

this is the circuit diagram of the motherboard. maybe this helps understanding what my problem is:

 

 

It shows how the connections are routed.

 

CPU1 has got the following connections:

• The chipset
• 2x PCI-E x16
• LAN x540 (2x 10Gigabit LAN adapters)

CPU2 is able to connect to:

• 4x PCI-E x8

Both CPUs are connected via QPI (that's a data port to exchange data directly between CPUs instead using the Front Side Bus) with 2 8G QPI ports

 

In general all data that is transferred to the CPU is sent via the Front Side Bus. That's the interface where you transfer data to the different PCI Express devices, the RAM, the Chipset (which handles Hard Drives, USB and so on), or pretty much everything.

 

As you can see, the RAM isn't shared between CPUs. That again means that that when shared tasks are executed, the same data needs to be stored through the different RAM blocks. Not identically, like in the same blocks, but the same data needs to be accessible.

 

In the beginning data was shared via the Front Site Bus. That means, that the same Interface in which data was exchanged with all other devices was used to exchange data through CPUs. That again means that the data that you could transfer to additional devices was more limited. You accidentally weren't able to send data to your GPU if the data exchange between the CPUs was too high. To change this, Intel invented 'QPI'. It's a direct data connection between the CPUs. Then again it has got a higher latency than sending it through the Front Side Bus. The question now is, what's the performance of this connection. How does it handle the data (latency/throughput wise) if BOTH CPUs are connected to the same storage. Is it advantageous to connect everything to CPU2 or does the dual CPU layout profit from connecting extension cards to different CPUs. 

 

I hope that made it clear where the problem is

[alpenwasser]

I'm a bit distracted so it's understandable if you didn't understand what I wrote, my GF is pregnant and the birth is just a question of 48 hours.

Well then, congratulations?

As for your QPI issue, I have no idea, you'd probably need to try it out.

[simson0606]

isn't that backplate meant for ssd caching? i thought sharkoon has one

[Ahnzh]

isn't that backplate meant for ssd caching? i thought sharkoon has one

 

hmmn? i don't understand. The Pactech backplane? It does nothing actually. It's grabbing the usage signal from the Sata power cable, amplifies it and sends it to 2 pins. except that it's pretty much as if you plug in a cable. 

 

Oh and my son was born today at 8:25 GMT+2

[darkspeedy26]

-snip-

Oh and my son was born today at 8:25 GMT+2

congratulations

[d33g33]

Jelly. Subbed.

[wpirobotbuilder]

Are those SSDs going to be used for an SSD volume? Exciting

 

You have my attention, especially with all that memory.

[OverachievingSnowfall]

this is the circuit diagram of the motherboard. maybe this helps understanding what my problem is:

 

 

It shows how the connections are routed.

 

CPU1 has got the following connections:

• The chipset
• 2x PCI-E x16
• LAN x540 (2x 10Gigabit LAN adapters)

CPU2 is able to connect to:

• 4x PCI-E x8

Both CPUs are connected via QPI (that's a data port to exchange data directly between CPUs instead using the Front Side Bus) with 2 8G QPI ports

 

In general all data that is transferred to the CPU is sent via the Front Side Bus. That's the interface where you transfer data to the different PCI Express devices, the RAM, the Chipset (which handles Hard Drives, USB and so on), or pretty much everything.

 

As you can see, the RAM isn't shared between CPUs. That again means that that when shared tasks are executed, the same data needs to be stored through the different RAM blocks. Not identically, like in the same blocks, but the same data needs to be accessible.

 

In the beginning data was shared via the Front Site Bus. That means, that the same Interface in which data was exchanged with all other devices was used to exchange data through CPUs. That again means that the data that you could transfer to additional devices was more limited. You accidentally weren't able to send data to your GPU if the data exchange between the CPUs was too high. To change this, Intel invented 'QPI'. It's a direct data connection between the CPUs. Then again it has got a higher latency than sending it through the Front Side Bus. The question now is, what's the performance of this connection. How does it handle the data (latency/throughput wise) if BOTH CPUs are connected to the same storage. Is it advantageous to connect everything to CPU2 or does the dual CPU layout profit from connecting extension cards to different CPUs. 

 

I hope that made it clear where the problem is

 

Yes actually that does clear it up for me, thanks.  The little reading on QPI I've done seems to indicate it has much higher throughput for the interface itself vs. FSB.  However, since that connection is only between the CPUs in this case, I would imagine there would be less latency if all drives are connected to just one CPU.  If the drives are distributed between the 2 CPUs, QPI must be used as well as the FSB when accessing data, theoretically increasing latency. So then, if all drives are connected to CPU1, then this would limit the use of QPI, therefore reducing latency to your LAN connection.  I have no idea how this would work in practice, so, like alpenwasser said, you'll have to try it out. I guess the question is, how much does QPI increase latency in this config?

 

btw, congrats on the birth of your son!

[Ahnzh]

I got new pics for you

 

 

The RAM came today, so did the fans and the PSU.

 

So what am I doing right now? switching up the built in fan (PSU) to a new one. 

[Ahnzh]

And here some more pics, the RAM, and other stuff.

 

And the PSU with the new fan inside!

[Ahnzh]

as a 'smaller' update i wanted to show the pcb boards that I'm using for the drive cages. I received them some time ago but never published the pictures.

 

[darkspeedy26]

woah, all those bags of connector thingies. this is to make it hot swapable right?

[Ahnzh]

woah, all those bags of connector thingies. this is to make it hot swapable right?

 it's hot swappable 'in some sort of way' from the start. You can plug out the cable of drives from the beginning. I want to build drive cages that you can pull out and put in without having to screw in some screws n stuff. I would have to move my whole server for that and considering something around 40 kg that's something you don't want to do.

[Ahnzh]

oh and I figured that it's hard to understand what it will look like in the end. here's some further detail:

 

 

It's just a rough design focussing on dimensions. it's 1:1 for the parts that you see, i have to add a PSU and fans to the design sometime but it should show you the dimensions of everything

[sirtoby]

This looks great. Do plan on using SAS or SATA harddrives?

[Ahnzh]

This looks great. Do plan on using SAS or SATA harddrives?

it's going to be both. SATA for storage and SAS (I got 12G SAS Host Bus Adapters) for SSDs