Jump to content
Search In
  • More options...
Find results that contain...
Find results in...
cjsomclo14989

Server Cluster Video for YouTube

Recommended Posts

Posted · Original PosterOP

I've been watching LMG on youtube for a few years now and I see a lot of videos of his server rack and trying things such as 8 (or 10) gamers 1 CPU but what I've been waiting to watch and what doesn't really have very many (not-so-advanced proof) tutorials online is a cluster server or HPC. From the research I've done, HPC requires some kind of permit to build (at least a permanent build) and the most stable way to accomplish HPC is to use a Linux Host and VM Windows. It would be pretty sick to see Windows server peak at it's limitations as mentioned on Wiki. I know it's an outdated version of windows but it was the latest I could find with some type of limitations. Makes me wonder if the newer versions could support more than the previous versions... (possible video) It could also be a way to make use of your unnecessary 10Gbps Internet speeds. Maybe after you're finished, you could get that permit and sell usage time for people to do some crazy amount of number crunching for extra cheddar.

 

Anyway, I've tried a few times but maybe I'm just thinking a cluster acts as 1 computer... I mean surely an IT guy wouldn't want to remote into each node if there is hundreds or thousands of nodes. Is it possible to get thousands of nodes to share memory and have basically a HDD or SSD for folders (which would be kind of an obvious solution for windows if there are more drives than the amount of letters in the alphabet)

 

A video of windows limitations and what happens if you exceed them... What could go wrong... Everything will be fine... you have plenty of... well I'm pretty sure you'll have to order a bunch of the same model and sizes of things since there are a lot of things that have to be identical in clusters... Well there's my idea for a video. Hope the team accepts the challenge even if it isn't on a large scale to exceed windows limitations.

Link to post
Share on other sites

Operating systems are not distributed, and they don't act as a distributed computing node - neither Windows or Linux. 

 

Most compute (CPU & Memory) cluster software is for failover purposes - if a node goes down, it moves services to another node. 

Typically compute distributed clustering is software specific, and is typically for something that isn't real-time, like crunching through algorithms used in physics modelling. In these cases you typically have a Master Node, followed by Slave Nodes. The master dishes out the work to the slave workers. 

 

If you're talking about storage, it's quite commonly distributed...for example a SAN. They're addressed at the block chunk level, with those being assigned as a single LUN within the same Datacenter. If you're talking about on the same hosts that do the computing, VMware has this solution, their product is called vSAN. Where it builds a virtual SAN network using the management vnic, across vCenter hosts. 

 

If you're using Twitch as an example, it is distributed networking, but not distributed compute. The broadcasters stream is encoded on one of their encode servers, then is sent out as an RTMP broadcast across the network, so at that point they're only constrained by their aggregated bandwidth. 

 

You can manage a cluster together quite easily using your scripts, for example this person creates an RPi cluster and uses a script to be able to execute the same command to all at the same time and to assign jobs to the slaves/workers for distributed (parallel) compute: https://epcced.github.io/wee_archlet/


Spoiler

Intel i7 3770K @ 4.6ghz | EVGA Z77 FTW | 2 x EVGA GTX1070 FTW | 32GB (4x8GB) Corsair Vengeance DDR3-1600 | Corsair H105 AIO, NZXT Sentry 3, Corsair SP120's | 2 x 256GB Samsung 850EVO, 4TB WD Black | Phanteks Enthoo Pro | OCZ ZX 1250w | Samsung 28" 4K Display | Ducky Shine 3 Keyboard, Logitech G502, MicroLab Solo 7C Speakers, Razer Goliathus Extended, X360 Controller | Windows 10 Pro | SteelSeries Siberia 350 Headphones

 

Spoiler

Corsair 400R, IcyDock MB998SP & MB455SPF, Seasonic X-Series 650w PSU, 2 x Xeon E5540's, 24GB DDR3-ECC, Asus Z8NA-D6C Motherboard, AOC-SAS2LP-MV8, LSI MegaRAID 9271-8i, RES2SV240 SAS Expander, Samsung 840Evo 120GB, 2 x 8TB Seagate Archives, 12 x 3TB WD Red

 

Link to post
Share on other sites

This would only be best covered by a TechQuickie as well, to do an actual LTT video covering any of this in practical detail is beyond the resource capabilities of LMG in both required hardware and skill set. There's too much to cover, too much hardware required to properly demonstrate it, and huge differences between application clustering, virtualization clustering, storage clustering and/or distributed storage as well as HPC (distributed vs parallel etc) being another different category again with differences within that too.

 

I would love more server videos, but LMG needs a proper server guy with industry experience otherwise they will be limited to 'watch me try out something I have never done before' videos like is normal, highly entertaining though just not so much educational (in the how it's supposed to be done way). LTT server videos inspire passion and drive people in to the field and towards education where they can become more expert in the field. 

Link to post
Share on other sites
16 hours ago, leadeater said:

This would only be best covered by a TechQuickie as well, to do an actual LTT video covering any of this in practical detail is beyond the resource capabilities of LMG in both required hardware and skill set. There's too much to cover, too much hardware required to properly demonstrate it, and huge differences between application clustering, virtualization clustering, storage clustering and/or distributed storage as well as HPC (distributed vs parallel etc) being another different category again with differences within that too.

 

I would love more server videos, but LMG needs a proper server guy with industry experience otherwise they will be limited to 'watch me try out something I have never done before' videos like is normal, highly entertaining though just not so much educational (in the how it's supposed to be done way). LTT server videos inspire passion and drive people in to the field and towards education where they can become more expert in the field. 

Soooo true - the problem is, to get the guy they need (decent industry experience), they'd need to pay quite a bit to lure them away from the corporate sector.


Then, on top of that, Vancouver is (by a decent margin) the most expensive city to live in, in all of Canada. Townhouses are more than 3 times as expensive in Vancouver as they are in the city I live in (~$300K-$400K compared to $1m+).

 

So, not only would they need to pay more to lure them away, but they'd also have to pay even more on top of that to compensate for the increased cost of living.

 

Granted, I assume LMG makes enough profit that they could still afford to pay whatever is required.

 

I love the server content LMG puts out - mostly for entertainment value - but sometimes it does give people the wrong impression on how to do things in the enterprise world (We see those results here on the forums now and then).


For Sale (lots of stuff):

Spoiler

[FS] [CAD] Various things

 

 

* Intel i7-4770K * ASRock Z97 Anniversary * 16GB RAM * 750w Seasonic Modular PSU *

* Crucial M4 128GB SSD (Primary) * Hitachi 500GB HDD (Secondary) *

* Gigabyte HD 7950 WF3 * SATA Blu-Ray Writer * Logitech g710+ * Windows 10 Pro x64 *

 

Link to post
Share on other sites

Create an account or sign in to comment

You need to be a member in order to leave a comment

Create an account

Sign up for a new account in our community. It's easy!

Register a new account

Sign in

Already have an account? Sign in here.

Sign In Now

Buy VPN

×