Jump to content
Search In
  • More options...
Find results that contain...
Find results in...
alpha754293

"Server room" build

Recommended Posts

Posted · Original PosterOP

It's not really a "build" per se, but I just moved my stuff from the basement into my utility room.

 

Equipment list:

Supermicro 6027TR-HTRF (quad dual socket half-width blade nodes in a 2RU rackmount, each node having dual Intel Xeon E5-2690 (v1) (8-cores, 2.9 GHz stock, 3.3 GHz max all core turbo, 3.6 GHz max turbo), 8x Samsung 16 GB DDR3-1866 ECC Reg. 2Rx4 RAM (128 GB total per node, 512 GB total for the whole system) running at DDR3-1600 speed (because it's 2R), SATA SSD for OS, HGST 7200 SATA for data, Mellanox ConnectX-4 dual port 100 Gbps (4x EDR Infiniband) NIC) (Storage configuration varies a little depending on which OS I am booting into. I have it physically separately by different OS SSDs and data HDDs.)

 

Mellanox 36-port 100 Gbps 4x EDR Infiniband externally managed switch (MSB-7890)

 

Qnap TS-832X 8-bay NAS (8x 10 TB HGST SATA 7.2krpm drives, in RAID5)

Qnap TS-832X 8-bay NAS (7x 6 TB HGST SATA 7.2krpm drives, in RAID5)

(those two are tied together via dual SFP+ to SFP+ 10GbE connections)

 

Buffalo Linkstation 441DE (4x 6 TB HGST SATA 7.2 krpm drives, in RAID5)

 

Netgear GS116 16-port 1 GbE switch

Netgear GS208 8-port 1 GbE switch

 

There's a bunch of other stuff that's not pictured here (4 workstations, another NAS, and some old, decommissioned servers). I'll have to get longer cables before I can bring those systems back up online.

IMG_1880.JPG

Link to post
Share on other sites
Posted · Original PosterOP
6 hours ago, dogwitch said:

posting for the switch alone. dam its a costly thing... like freaken uber top end pc lvl

It actually wasn't too bad.

 

I originally bought a used, managed switch and there was a problem with it where it would reboot itself at least once, every hour. Logged into the console and there was nothing indicative that was forcing or causing it to reboot itself (temps were nominal, and nothign in the logs), so I sent that one back and ended up picking up the unmanaged switch instead.

And to be honest, the ONLY downside that I have found with using the unmanaged switch over the managed switch is the fact that you "lose" one of the ports to a system that has to run the Infiniband subnet manager externally (hence externally managed) rather than running it on the unit itself. Other than that, the switch is great. Course, that only really matters if you're approaching 34-35 4x EDR Infiniband devices plugged into the switch, so... (and if you're doing that, let's say because you want blazing fast storage speeds, you have other problems to contend with like being able to find a server (or a whole rack/cabinet of servers) that will be able to spit out 3.6 TERAbits per second of data, sustained.) In other words, there are other issues that arise moreso than "losing" a port on the switch if you have a headnode or something else that's running the subnet manager.

 

However, making the switch (no pun intended) from a managed switch to an unmanaged switch, saved me about $1000 US on the used market, so it was worth it to me (which only worked out to be about $2200 US).

 

I was looking at a 28-port Netgear 10GbE switch (24 RJ45 ports + 4 SFP+ ports) last night and that was running at around $1900 US. So for only $300 US more, I'm running more ports, AND each port is rated for a data/link transfer rate that's 10 TIMES faster, at which point, why would I even bother with 10 GbE?

 

I use (the 100 Gbps IB switch) all the time because it's my system interconnect for my micro compute cluster.

Link to post
Share on other sites

ah. yeah i score .kid you not. a 48 port full 1gb switch enterprise class at a good will. for less then 30 bucks. works to. only issue you need a Ethernet to vga cable... which i need to get. which i never used before. so .... but yeah atm am using 15 ports off the thing.

Link to post
Share on other sites
Posted · Original PosterOP
On 7/27/2019 at 2:12 AM, dogwitch said:

ah. yeah i score .kid you not. a 48 port full 1gb switch enterprise class at a good will. for less then 30 bucks. works to. only issue you need a Ethernet to vga cable... which i need to get. which i never used before. so .... but yeah atm am using 15 ports off the thing.

Yeah -- being able to plug in to the console is super important, especially on first start up for used/old gear so that you can reset the unit to factory defaults so that it'll wipe the passwords, routing tables, configurations, etc.

 

I just picked up a 48 port GbE switch (with four of the ports doubling as RJ45 and SFP+ ports, so I'll FINALLY be able to plug my two NASes in with the rest of the network AT 10G speeds) off eBay for $70.

 

I was running out of ports on my old 16-port GbE switch.

 

My idea is that the GbE will just be for the management traffic (predominantly) and then everything else will be running off Infiniband (mostly because since I already bought everything/have it -- it's now kind of like "ehhh....why not?".

 

Of course, with that, my "new" #firstworldproblems is that a) not all of my systems has a PCIe 3.0 x16 slot available (e.g. on the NAS units and also on my HTPCs), again, not that the HTPC NEEDS 100 Gbps link. (NAS won't really be able to use it either as I just have spinning rust drives in there -- no NVMe yet.)

 

And the other thing is that Mellanox dropped their NFSoRDMA support in their drivers a while ago, so I'll have to run iSER for it to work, but again, with my lack of a fast enough storage subsystem on both ends, the IB is really useful only at solution runtime and less so for storage purposes because it's going to be a while before I'll be able to upgrade the storage.

Link to post
Share on other sites
Posted · Original PosterOP

The small update over this past weekend was I now have a 52-port L2 managed GbE switch (Netgear GSM7248) instead of the 16-port GbE switch that I had previously (Netgear GS116).

 

This is in preparation for my office moving to the basement from where it is now, currently occupying a room.

 

There is a proposal that I might actually end up consolidating almost all of my centralised network/computing equipment (with very few exceptions) to the rack now that I have it up and running.

 

It's going to be quite the PITA to take down though, when we eventually move to a bigger house.

Link to post
Share on other sites
Posted · Original PosterOP
15 hours ago, dogwitch said:

is their any way to log into a enterprise switch with out using a Ethernet to display cable?

If you have a managed switch and it has a RJ45 management port, you can try logging into it via ssh.

 

The problem, of course, is that you would need to know the IP address that the management port has and that comes with one of two possibilities: Either it STILL has DHCP enabled in which point, you might be able to find out what IP address it picked up from your DHCP server OR worse, if it has a static IP address assigned to it, if you can't directly plug your console into it, now you would have to sweep through the entire IPv4 address space to find out what's the IP address that's been set as the static IP address.

 

And if you manage to find/figure out what the static IP address is, now you gotta figure out what the password is.

 

SOME switches offer a way to do a factory reset, but that doesn't always work. (This was the case with my previous Mellanox MSB-7800 managed 4x EDR IB switch because there was a problem with the management board itself such that it would reboot at least once, every hour, on the hour -- so my connection would keep dropping and it wouldn't reset properly.)

 

So in order for me to try and find/figure out why -- I had to log in directly to the console so that I can figure out what the heck was going on with that thing.

 

However, once you've configured the management port yourself, then after that, no, you don't need to plug in directly because you can log in via ssh.

 

But that IS dependent on you either being able to successfully reset the managed switch to factory defaults and/or being able to figure out what the management port's IP address (assuming that it HAS a RJ45 management port, which, not all managed switches have that), and also the management console's password.

Link to post
Share on other sites
Posted · Original PosterOP
12 minutes ago, dogwitch said:

ah ok. thank you for the info on that. i will try some of it.

No problem.

 

You're welcome.

 

Worse case scenarios, you might be able to find those cables on eBay relatively inexpensively.

Link to post
Share on other sites
On 8/8/2019 at 3:13 PM, alpha754293 said:

No problem.

 

You're welcome.

 

Worse case scenarios, you might be able to find those cables on eBay relatively inexpensively.

yeah but it sucks thru. none the less. but still  for less then 30  bucks  is one hell of a deal

Link to post
Share on other sites
Posted · Original PosterOP
7 hours ago, dogwitch said:

yeah but it sucks thru. none the less. but still  for less then 30  bucks  is one hell of a deal

It might be able to save you quite a bit of headache.

 

You might end up spending more than $30 worth of your time trying to get into the switch using other means/methods vs. just spending the $30 and be done with it.

 

It's too bad that you can't just "RENT" a cable like that, because if your switch is working (i.e. unlike my original managed Mellanox switch where it DID have an issue with it), then you only really need it once to do the initial setup and everything else after that, you can probably manage with just ssh.

Link to post
Share on other sites
Posted · Original PosterOP
On 8/16/2019 at 6:13 PM, dogwitch said:

am going to  to try it tonight . i got some free time for networking stuff

Cool.

 

Let me know if that works for you or not and if there's any other questions that you might have that might be able to help you.

 

Thanks.

Link to post
Share on other sites

Create an account or sign in to comment

You need to be a member in order to leave a comment

Create an account

Sign up for a new account in our community. It's easy!

Register a new account

Sign in

Already have an account? Sign in here.

Sign In Now

Buy VPN

×