Jump to content

Server Cluster Video for YouTube

I've been watching LMG on youtube for a few years now and I see a lot of videos of his server rack and trying things such as 8 (or 10) gamers 1 CPU but what I've been waiting to watch and what doesn't really have very many (not-so-advanced proof) tutorials online is a cluster server or HPC. From the research I've done, HPC requires some kind of permit to build (at least a permanent build) and the most stable way to accomplish HPC is to use a Linux Host and VM Windows. It would be pretty sick to see Windows server peak at it's limitations as mentioned on Wiki. I know it's an outdated version of windows but it was the latest I could find with some type of limitations. Makes me wonder if the newer versions could support more than the previous versions... (possible video) It could also be a way to make use of your unnecessary 10Gbps Internet speeds. Maybe after you're finished, you could get that permit and sell usage time for people to do some crazy amount of number crunching for extra cheddar.

 

Anyway, I've tried a few times but maybe I'm just thinking a cluster acts as 1 computer... I mean surely an IT guy wouldn't want to remote into each node if there is hundreds or thousands of nodes. Is it possible to get thousands of nodes to share memory and have basically a HDD or SSD for folders (which would be kind of an obvious solution for windows if there are more drives than the amount of letters in the alphabet)

 

A video of windows limitations and what happens if you exceed them... What could go wrong... Everything will be fine... you have plenty of... well I'm pretty sure you'll have to order a bunch of the same model and sizes of things since there are a lot of things that have to be identical in clusters... Well there's my idea for a video. Hope the team accepts the challenge even if it isn't on a large scale to exceed windows limitations.

Link to comment
Share on other sites

Link to post
Share on other sites

Operating systems are not distributed, and they don't act as a distributed computing node - neither Windows or Linux. 

 

Most compute (CPU & Memory) cluster software is for failover purposes - if a node goes down, it moves services to another node. 

Typically compute distributed clustering is software specific, and is typically for something that isn't real-time, like crunching through algorithms used in physics modelling. In these cases you typically have a Master Node, followed by Slave Nodes. The master dishes out the work to the slave workers. 

 

If you're talking about storage, it's quite commonly distributed...for example a SAN. They're addressed at the block chunk level, with those being assigned as a single LUN within the same Datacenter. If you're talking about on the same hosts that do the computing, VMware has this solution, their product is called vSAN. Where it builds a virtual SAN network using the management vnic, across vCenter hosts. 

 

If you're using Twitch as an example, it is distributed networking, but not distributed compute. The broadcasters stream is encoded on one of their encode servers, then is sent out as an RTMP broadcast across the network, so at that point they're only constrained by their aggregated bandwidth. 

 

You can manage a cluster together quite easily using your scripts, for example this person creates an RPi cluster and uses a script to be able to execute the same command to all at the same time and to assign jobs to the slaves/workers for distributed (parallel) compute: https://epcced.github.io/wee_archlet/

Spoiler

Desktop: Ryzen9 5950X | ASUS ROG Crosshair VIII Hero (Wifi) | EVGA RTX 3080Ti FTW3 | 32GB (2x16GB) Corsair Dominator Platinum RGB Pro 3600Mhz | EKWB EK-AIO 360D-RGB | EKWB EK-Vardar RGB Fans | 1TB Samsung 980 Pro, 4TB Samsung 980 Pro | Corsair 5000D Airflow | Corsair HX850 Platinum PSU | Asus ROG 42" OLED PG42UQ + LG 32" 32GK850G Monitor | Roccat Vulcan TKL Pro Keyboard | Logitech G Pro X Superlight  | MicroLab Solo 7C Speakers | Audio-Technica ATH-M50xBT2 LE Headphones | TC-Helicon GoXLR | Audio-Technica AT2035 | LTT Desk Mat | XBOX-X Controller | Windows 11 Pro

 

Spoiler

Server: Fractal Design Define R6 | Ryzen 3950x | ASRock X570 Taichi | EVGA GTX1070 FTW | 64GB (4x16GB) Corsair Vengeance LPX 3000Mhz | Corsair RM850v2 PSU | Fractal S36 Triple AIO | 12 x 8TB HGST Ultrastar He10 (WD Whitelabel) | 500GB Aorus Gen4 NVMe | 2 x 2TB Samsung 970 Evo Plus NVMe | LSI 9211-8i HBA

 

Link to comment
Share on other sites

Link to post
Share on other sites

This would only be best covered by a TechQuickie as well, to do an actual LTT video covering any of this in practical detail is beyond the resource capabilities of LMG in both required hardware and skill set. There's too much to cover, too much hardware required to properly demonstrate it, and huge differences between application clustering, virtualization clustering, storage clustering and/or distributed storage as well as HPC (distributed vs parallel etc) being another different category again with differences within that too.

 

I would love more server videos, but LMG needs a proper server guy with industry experience otherwise they will be limited to 'watch me try out something I have never done before' videos like is normal, highly entertaining though just not so much educational (in the how it's supposed to be done way). LTT server videos inspire passion and drive people in to the field and towards education where they can become more expert in the field. 

Link to comment
Share on other sites

Link to post
Share on other sites

16 hours ago, leadeater said:

This would only be best covered by a TechQuickie as well, to do an actual LTT video covering any of this in practical detail is beyond the resource capabilities of LMG in both required hardware and skill set. There's too much to cover, too much hardware required to properly demonstrate it, and huge differences between application clustering, virtualization clustering, storage clustering and/or distributed storage as well as HPC (distributed vs parallel etc) being another different category again with differences within that too.

 

I would love more server videos, but LMG needs a proper server guy with industry experience otherwise they will be limited to 'watch me try out something I have never done before' videos like is normal, highly entertaining though just not so much educational (in the how it's supposed to be done way). LTT server videos inspire passion and drive people in to the field and towards education where they can become more expert in the field. 

Soooo true - the problem is, to get the guy they need (decent industry experience), they'd need to pay quite a bit to lure them away from the corporate sector.


Then, on top of that, Vancouver is (by a decent margin) the most expensive city to live in, in all of Canada. Townhouses are more than 3 times as expensive in Vancouver as they are in the city I live in (~$300K-$400K compared to $1m+).

 

So, not only would they need to pay more to lure them away, but they'd also have to pay even more on top of that to compensate for the increased cost of living.

 

Granted, I assume LMG makes enough profit that they could still afford to pay whatever is required.

 

I love the server content LMG puts out - mostly for entertainment value - but sometimes it does give people the wrong impression on how to do things in the enterprise world (We see those results here on the forums now and then).

For Sale: Meraki Bundle

 

iPhone Xr 128 GB Product Red - HP Spectre x360 13" (i5 - 8 GB RAM - 256 GB SSD) - HP ZBook 15v G5 15" (i7-8850H - 16 GB RAM - 512 GB SSD - NVIDIA Quadro P600)

 

Link to comment
Share on other sites

Link to post
Share on other sites

Create an account or sign in to comment

You need to be a member in order to leave a comment

Create an account

Sign up for a new account in our community. It's easy!

Register a new account

Sign in

Already have an account? Sign in here.

Sign In Now

×