"Server room" build

[alpha754293]

It's not really a "build" per se, but I just moved my stuff from the basement into my utility room.

 

Equipment list:

Supermicro 6027TR-HTRF (quad dual socket half-width blade nodes in a 2RU rackmount, each node having dual Intel Xeon E5-2690 (v1) (8-cores, 2.9 GHz stock, 3.3 GHz max all core turbo, 3.6 GHz max turbo), 8x Samsung 16 GB DDR3-1866 ECC Reg. 2Rx4 RAM (128 GB total per node, 512 GB total for the whole system) running at DDR3-1600 speed (because it's 2R), SATA SSD for OS, HGST 7200 SATA for data, Mellanox ConnectX-4 dual port 100 Gbps (4x EDR Infiniband) NIC) (Storage configuration varies a little depending on which OS I am booting into. I have it physically separately by different OS SSDs and data HDDs.)

 

Mellanox 36-port 100 Gbps 4x EDR Infiniband externally managed switch (MSB-7890)

 

Qnap TS-832X 8-bay NAS (8x 10 TB HGST SATA 7.2krpm drives, in RAID5)

Qnap TS-832X 8-bay NAS (7x 6 TB HGST SATA 7.2krpm drives, in RAID5)

(those two are tied together via dual SFP+ to SFP+ 10GbE connections)

 

Buffalo Linkstation 441DE (4x 6 TB HGST SATA 7.2 krpm drives, in RAID5)

 

Netgear GS116 16-port 1 GbE switch

Netgear GS208 8-port 1 GbE switch

 

There's a bunch of other stuff that's not pictured here (4 workstations, another NAS, and some old, decommissioned servers). I'll have to get longer cables before I can bring those systems back up online.

[dogwitch]

posting for the switch alone. dam its a costly thing... like freaken uber top end pc lvl

[alpha754293]

6 hours ago, dogwitch said:

posting for the switch alone. dam its a costly thing... like freaken uber top end pc lvl

It actually wasn't too bad.

 

I originally bought a used, managed switch and there was a problem with it where it would reboot itself at least once, every hour. Logged into the console and there was nothing indicative that was forcing or causing it to reboot itself (temps were nominal, and nothign in the logs), so I sent that one back and ended up picking up the unmanaged switch instead.

And to be honest, the ONLY downside that I have found with using the unmanaged switch over the managed switch is the fact that you "lose" one of the ports to a system that has to run the Infiniband subnet manager externally (hence externally managed) rather than running it on the unit itself. Other than that, the switch is great. Course, that only really matters if you're approaching 34-35 4x EDR Infiniband devices plugged into the switch, so... (and if you're doing that, let's say because you want blazing fast storage speeds, you have other problems to contend with like being able to find a server (or a whole rack/cabinet of servers) that will be able to spit out 3.6 TERAbits per second of data, sustained.) In other words, there are other issues that arise moreso than "losing" a port on the switch if you have a headnode or something else that's running the subnet manager.

 

However, making the switch (no pun intended) from a managed switch to an unmanaged switch, saved me about $1000 US on the used market, so it was worth it to me (which only worked out to be about $2200 US).

 

I was looking at a 28-port Netgear 10GbE switch (24 RJ45 ports + 4 SFP+ ports) last night and that was running at around $1900 US. So for only $300 US more, I'm running more ports, AND each port is rated for a data/link transfer rate that's 10 TIMES faster, at which point, why would I even bother with 10 GbE?

 

I use (the 100 Gbps IB switch) all the time because it's my system interconnect for my micro compute cluster.

[dogwitch]

ah. yeah i score .kid you not. a 48 port full 1gb switch enterprise class at a good will. for less then 30 bucks. works to. only issue you need a Ethernet to vga cable... which i need to get. which i never used before. so .... but yeah atm am using 15 ports off the thing.

[alpha754293]

On 7/27/2019 at 2:12 AM, dogwitch said:

ah. yeah i score .kid you not. a 48 port full 1gb switch enterprise class at a good will. for less then 30 bucks. works to. only issue you need a Ethernet to vga cable... which i need to get. which i never used before. so .... but yeah atm am using 15 ports off the thing.

Yeah -- being able to plug in to the console is super important, especially on first start up for used/old gear so that you can reset the unit to factory defaults so that it'll wipe the passwords, routing tables, configurations, etc.

 

I just picked up a 48 port GbE switch (with four of the ports doubling as RJ45 and SFP+ ports, so I'll FINALLY be able to plug my two NASes in with the rest of the network AT 10G speeds) off eBay for $70.

 

I was running out of ports on my old 16-port GbE switch.

 

My idea is that the GbE will just be for the management traffic (predominantly) and then everything else will be running off Infiniband (mostly because since I already bought everything/have it -- it's now kind of like "ehhh....why not?".

 

Of course, with that, my "new" #firstworldproblems is that a) not all of my systems has a PCIe 3.0 x16 slot available (e.g. on the NAS units and also on my HTPCs), again, not that the HTPC NEEDS 100 Gbps link. (NAS won't really be able to use it either as I just have spinning rust drives in there -- no NVMe yet.)

 

And the other thing is that Mellanox dropped their NFSoRDMA support in their drivers a while ago, so I'll have to run iSER for it to work, but again, with my lack of a fast enough storage subsystem on both ends, the IB is really useful only at solution runtime and less so for storage purposes because it's going to be a while before I'll be able to upgrade the storage.

[alpha754293]

The small update over this past weekend was I now have a 52-port L2 managed GbE switch (Netgear GSM7248) instead of the 16-port GbE switch that I had previously (Netgear GS116).

 

This is in preparation for my office moving to the basement from where it is now, currently occupying a room.

 

There is a proposal that I might actually end up consolidating almost all of my centralised network/computing equipment (with very few exceptions) to the rack now that I have it up and running.

 

It's going to be quite the PITA to take down though, when we eventually move to a bigger house.

[dogwitch]

is their any way to log into a enterprise switch with out using a Ethernet to display cable?

[alpha754293]

15 hours ago, dogwitch said:

is their any way to log into a enterprise switch with out using a Ethernet to display cable?

If you have a managed switch and it has a RJ45 management port, you can try logging into it via ssh.

 

The problem, of course, is that you would need to know the IP address that the management port has and that comes with one of two possibilities: Either it STILL has DHCP enabled in which point, you might be able to find out what IP address it picked up from your DHCP server OR worse, if it has a static IP address assigned to it, if you can't directly plug your console into it, now you would have to sweep through the entire IPv4 address space to find out what's the IP address that's been set as the static IP address.

 

And if you manage to find/figure out what the static IP address is, now you gotta figure out what the password is.

 

SOME switches offer a way to do a factory reset, but that doesn't always work. (This was the case with my previous Mellanox MSB-7800 managed 4x EDR IB switch because there was a problem with the management board itself such that it would reboot at least once, every hour, on the hour -- so my connection would keep dropping and it wouldn't reset properly.)

 

So in order for me to try and find/figure out why -- I had to log in directly to the console so that I can figure out what the heck was going on with that thing.

 

However, once you've configured the management port yourself, then after that, no, you don't need to plug in directly because you can log in via ssh.

 

But that IS dependent on you either being able to successfully reset the managed switch to factory defaults and/or being able to figure out what the management port's IP address (assuming that it HAS a RJ45 management port, which, not all managed switches have that), and also the management console's password.

[dogwitch]

ah ok. thank you for the info on that. i will try some of it.

[alpha754293]

12 minutes ago, dogwitch said:

ah ok. thank you for the info on that. i will try some of it.

No problem.

 

You're welcome.

 

Worse case scenarios, you might be able to find those cables on eBay relatively inexpensively.

[dogwitch]

On 8/8/2019 at 3:13 PM, alpha754293 said:

No problem.

 

You're welcome.

 

Worse case scenarios, you might be able to find those cables on eBay relatively inexpensively.

yeah but it sucks thru. none the less. but still  for less then 30  bucks  is one hell of a deal

[alpha754293]

7 hours ago, dogwitch said:

yeah but it sucks thru. none the less. but still  for less then 30  bucks  is one hell of a deal

It might be able to save you quite a bit of headache.

 

You might end up spending more than $30 worth of your time trying to get into the switch using other means/methods vs. just spending the $30 and be done with it.

 

It's too bad that you can't just "RENT" a cable like that, because if your switch is working (i.e. unlike my original managed Mellanox switch where it DID have an issue with it), then you only really need it once to do the initial setup and everything else after that, you can probably manage with just ssh.

[dogwitch]

am going to  to try it tonight . i got some free time for networking stuff

[alpha754293]

On 8/16/2019 at 6:13 PM, dogwitch said:

am going to  to try it tonight . i got some free time for networking stuff

Cool.

 

Let me know if that works for you or not and if there's any other questions that you might have that might be able to help you.

 

Thanks.

[alpha754293]

*edit*

 

Here are some changes/additions, etc. since this was originally posted.

 

I've now added a headnode to the cluster.

 

Previously, I was using the first node of my micro compute cluster also as the head node, but now I've offloaded that role onto another system entirely.

 

Headnode:

Asus P9X79-E WS

Intel Core i7-4930K (6 core, 3.4 GHz stock, 3.6 GHz max all core turbo, 3.9 GHz max turbo, HTT disabled)
8x Crucial Ballistix Sport 8 GB DDR3-1600 Unbuffered, non-ECC 9-9-9-24 RAM (64 GB total)

Corsair H80i

Corsair CM750

1x Intel 535 Series (I think?) 240 GB SATA 6 Gbps SSD

4x Samsung 860 EVO 1 TB SATA 6 Gbps SSD (in RAID0, off of the Marvell 9230 controller)

(coming soon) 4x HGST 6 TB 7200 rpm SATA 6 Gbps HDDs (in RAID0, off of the X79 chipset, but plugged into SATA 3 Gbps ports only because I don't have any more SATA 6 Gbps ports)

Mellanox ConnectX-4 dual port 100 Gbps 4x EDR Infiniband NIC

EVGA nVidia GeForce GTX Titan (6 GB, I think?)

CentOS 7.6.1810

 

Head node also now has the scratch directory as well for the micro cluster slave nodes and it is presented back to the IB network/system interconnect as a XFS formatted, NFSoRDMA mount.

 

The micro cluster slave nodes connect to the IB switch using copper direct attached cables (I think they're like 2 m long). The cable connecting the headnode to the switch (because it's going to be stationed outside of my utility room in my basement) is using a 30 m active optic cable. (And I also purchased an extra one as well, for future growth.)

 

The 4 TB SSD RAID array is up. Next is going to be when my 6 TB hard drives come in, to bring that up RAID0 array up as well for the cluster to use as a large capacity scratch directory.

 

I switched the headnode so that the headnode would have something better than the Matrox G200e that was in the micro cluster slave nodes, and also because the Asus P9X79-E WS had PCIe 3.0 x16 slots whereas my old Asus X79 Sabertooth did not (at least not necessarily) and also my old Intel Core i7-3930Ks definitely did not support PCIe 3.0 (also hence the new processor).

 

This is so that the Mellanox ConnectX-4 card can have full PCIe 3.0 x16/x16 bandwidth.

[dogwitch]

k tried and it will not let me log into the default ip address it suppose to have.

here is model of device

 3com 3CBLSG48 Baseline Switch 2948-SFP Plus

 

[alpha754293]

15 hours ago, dogwitch said:

k tried and it will not let me log into the default ip address it suppose to have.

here is model of device

 3com 3CBLSG48 Baseline Switch 2948-SFP Plus

 

What is the default IP address you think that the switch is supposed to have?

[dogwitch]

the one listed on it. but when i enter that in . it not going thru at all.  error out

[alpha754293]

13 hours ago, dogwitch said:

the one listed on it. but when i enter that in . it not going thru at all.  error out

???

 

I'm confused.

 

What do you mean "the one listed on it"???

 

(I don't have your switch so I can't see that.)

 

What's the last two bytes of the management MAC address? (in decimal)

 

Is that the same as "the one listed on it"?

 

Is the IPv4 address for the management port set up for auto DHCP or is the IPv4 address on the management port manually set/assigned?

 

Can you run a port/IP scan on it?

 

You might need to do that with a system that has multiple NICs so that you can physically connect to it via a cross-over cable so that you can run a port/IP scan on it with trying different classes to see if you can try and figure out what the IPv4 address is of the management port.

 

Refer to the user's manual for the switch online.

[dogwitch]

i will try .

i will see if i can find the info.

 

the manual odd. seeing at the start it say to log in do this.... issue is that 18 pages later that you have to set up.

[alpha754293]

5 hours ago, dogwitch said:

i will try .

i will see if i can find the info.

 

the manual odd. seeing at the start it say to log in do this.... issue is that 18 pages later that you have to set up.

Again, I don't have a switch like yours, so it's hard for me to guide you explicitly, but in the manual that I found online, it says this:

 

Quote

IP Configuration The switch’s IP configuration is determined automatically using DHCP,
or manually using values you assign.
Automatic IP Configuration using DHCP
By default the switch tries to configure its IP Information without requesting user intervention. It tries to obtain an IP address from a
DHCP server on the network.
Default IP Address If no DHCP server is detected, the switch will use
its default IP information. The default IP address is 169.254.x.y, where
x and y are the last two bytes of its MAC address

 (p. 12)

 

Again, if that doesn't work, or if someone else had previously configured the manual IP address, you would need to probably connect to it via a serial console cable and log into the switch directly so that you can clear out whatever configuration was on the switch already and start all over again from scratch.

 

The instructions on that starts on page 14, and using the console cable, you don't need to know the IP address of the switch since you are going to be plugged into it directly.

[dogwitch]

tbh what would be a good console cable. from amazon? it would be the first time i ever buy one.

[alpha754293]

5 minutes ago, dogwitch said:

tbh what would be a good console cable. from amazon? it would be the first time i ever buy one.

I'm not sure that there really is a brand preference.

 

As long as it does what it needs to do according to the manual.

 

I think that when I bought my now-returned Mellanox MSB-7800 managed switch, it came with the cable, so that's what I used.

 

As long as it works, that's all that it really matters.

 

Again, you only need to the rare times that you have to be directly plugged into the switch to set the IP address. Once you do that, then you can pretty much do everything else either through their web interface and/or you might be able to ssh into the switch.

 

Conversely though, again, per the documentation, it says that if you do a factory reset on it and it defaults to auto DHCP and it doesn't obtain an address from a DHCP server, then it will be 169.254.x.y where x.y are the last two bytes of the MAC address. So, that SHOULD work. I don't see why it wouldn't.

 

You might have to bring a laptop with a RJ45 port and set the IP address on it so that it's also 169.254.a.b/16 so that might be able to help your laptop find your switch so that you can remotely log into it.

 

Many different ways to approach and solve/address this one.