1 CPU multiple student build - suggestions/help wanted

[MakeItZone]

Hi-

 

hoping the community can help me figure out a reasonable 1 CPU, multiple user system.

 

Target use is as for a computer learning lab. Rough outline of hardware I'm looking for, and uses:

- 6, but preferably 8 student seats

- PCIe Passthrough to mid level AMD, or possibly NVidia GPUs

- to be used for programming, lego & vex robotics, WebGL (online CAD/CAM software, eg TinkerCAD), CAD (fusion 360)

- VMs will typically be Windows 10 home, but also Linux, FreeBSD

- will probably use ESXi, but possibly Unraid (but I've had bad luck with USB sticks failing as boot drives, so that aspect makes me nervous.)

 

Goals are:

- reduced IT management (snapshots are easier to manage than multiple desktops...)

- flexible CPU resources (eg 1 VM with all resources, or 8 with 1/8th...)

- hardware cost savings (I know, they won't be huge to start with, but over several years I think there will be. E.g. even one extra stick of RAM benefits everyone.)

 

I'm reasonably experienced with computer architecture, OS installation, etc.  Right now I use a lower end i7, 3 GPUs, Linux, XNest, and lots of USB 2 ports to power an up-to 10 seat system that can handle Minecraft & TinkerCAD, Python, etc. Unfortunately I can't get some things to work with Wine.

 

I just don't have the bandwidth to keep up/figure out all the (poorly) documented nuances of PCI passthrough, eg ACS support.

 

Right now I'm looking at this MoBo:

 

https://www.asus.com/ca-en/Motherboards/X99E_WSUSB_31/ 

 

And pairing it with the best Xeon E5 I can afford, to get the most PCIe lanes.

 

I believe this combo will provide true ACS isolation for all 7 PCIe slots, allowing a 1 CPU, 7 student setup.

(I'm not interested in ACS override patches- tried that in an older system, got random weird behaviour after some hours, will be multiplying that potential several fold.)

 

But there's been a few CPU and hardware announcements lately ;-)

 

So question time-

 

Is this going to have the ACS isolation I'm looking for?

 

Is there a better MoBo/CPU combo I should be looking at?

 

I'm not against dual CPU boards, but would probably have to start with Xeon E3s. My understanding is they don't have the same ACS isolation- so that might not work. Or populate one CPU socket with an E5. But my understanding is that can mess up the PCIe map/access as well- maybe not all PCIe sockets would be active?

 

Suggestions appreciated! Also for Canadian suppliers; the ones I've checked all have 4-8+wk lead times on that motherboard.

 

Thanks!

 

[Electronics Wizardy]

Look at vdi solutions with thin clients. Doing it this messy way is a bad idea. With vdi it's only cheaper with lots of systems. You paying thousands for software for this to work right. 

 

Are you on ad?

 

I'd just get multiple low end systems. One system is a pain to setup and use not officially supported and likes to have random issues. 

[YedZed]

Just now, CUDA_Cores said:

 Just because Linus does something does not in any way mean you should copy him.

This should be the motto of the forum.  

[MakeItZone]

My understanding of the more common VDI solutions is that:

- requires Windows Server

- multiple (higher than win 10 home) licenses 

- high end grid GPU cards tailored for this use

- "low end" desktops/thin clients to display the remote systems... which aren't cheap when I've looked

- poorer "feel" for the end users, both latency and compression artifacts

- much higher IT management, server, client images/config, "thin clients"; my plan is to snapshot the Windows VMs and regularly restore them to a known state. Student data will be on their USB drive or a network share.

- much higher overall cost for ~10 users (win server + per user licenses + server + grid card/non consumer GPUs, thin clients)

 

This is for a small lab, starting out; budget is pretty tight. The "backup plan" would be that if this fails, convert it to an onsite server and buy some low end desktop units; at least the GPUs would be re-usable in that scenario.

 

Let me know if I'm way off on the software costs, I had some conversations with CDW a few months ago but I told them to stop when the software cost started getting beyond multiples of 10x individual Win 10 home licenses ;-)

[MakeItZone]

4 minutes ago, CUDA_Cores said:

Agreed. Just because linus does something does not in any way mean you should copy him.

This was my own idea... only came across Linus's work in researching specific hardware setups.

 

Unfortunately low end systems can also have their own sets of problems- admittedly easier to diagnose and fix, but potentially just as time consuming and costly :-)

 

I just don't have big $$$ to pay for "enterprise" solutions, and have had surprising success with my 8-10 student Linux box.

 

 

[BubblyCharizard]

you could look into unraid...they might give you a good price for each license if they know its for education use

 

just drop them a line and they might help

[Electronics Wizardy]

Just now, MakeItZone said:

My understanding of the more common VDI solutions is that:

- requires Windows Server

- multiple (higher than win 10 home) licenses 

- high end grid GPU cards tailored for this use

- "low end" desktops/thin clients to display the remote systems... which aren't cheap when I've looked

- poorer "feel" for the end users, both latency and compression artifacts

- much higher IT management, server, client images/config, "thin clients"; my plan is to snapshot the Windows VMs and regularly restore them to a known state. Student data will be on their USB drive or a network share.

- much higher overall cost for ~10 users (win server + per user licenses + server + grid card/non consumer GPUs, thin clients)

 

This is for a small lab, starting out; budget is pretty tight. The "backup plan" would be that if this fails, convert it to an onsite server and buy some low end desktop units; at least the GPUs would be re-usable in that scenario.

 

Let me know if I'm way off on the software costs, I had some conversations with CDW a few months ago but I told them to stop when the software cost started getting beyond multiples of 10x individual Win 10 home licenses ;-)

You can be cheap and use multipoint. It's built into Windows server and let's you have multiple users on one pc and is made for this use. You don't need one gpu per user abd just need one copy of windows server and a few cals. I'd your a school these are cheap. 

 

For vdi there are two main ways with 

[MakeItZone]

Thanks @Electronics Wizardy! 

 

I did not know about Multipoint. It looks very promising. Especially the split screen and "blank all the student's screens" features. (Why did none of the CDW or MS reps I contacted point this out?!)

 

I've spent the last couple of hours googling and reading and see lots of goodness re RemoteFX and GPU sharing for *remote* terminals, but can't find anything that says DirectX/OpenGL applications directly connected get GPU acceleration. (eg https://blogs.technet.microsoft.com/hybridcloudbp/2016/11/15/new-rds-capabilities-in-windows-server-2016-for-service-providers/)

 

Do you know if apps running on "stations" directly connected to a video card will have GPU acceleration? Or is that limited to only Remote RDP/RemoteFX clients?

Even the deployment guide (https://docs.microsoft.com/en-us/windows-server/remote/multipoint-services/hardware-and-performance-recommendations) is vague- talking about video (eg H264) performance and not 3D rendering. Update: Just found this (http://windowsitpro.com/windows-server-2016/when-gpu-can-be-utilized-session-virtualization-2016) - looks promising.

 

There are also some notes re applications needing to be "session aware". I sort of remember what windows sessions are, but have no idea if most modern windows apps "will do the right thing." It looks like the Multipoint service can also present a VM instance to a station- ie to run apps that aren't session aware/don't run under virtualized sessions.

 

Windows Server 2016 essentials (~$800, I'm not a school) supports up to 25 users- I'm hoping that includes Multipoint users. This would be very price effective!

The only downside is the Hyper-V role appears to be limited to max of 2VMs.  So i wouldn't be host a VM/student if an app isn't multipoint compatible. That would mean moving up to the datacenter edition + CALs- way to expensive!

 

So it looks to me that getting the Asus Xeon Workstation board + the GPUs I already have is a reasonable bet- if everything works with Multipoint I'm set to go.

If it doesn't (eg software problems with Multipoint) I have a chance to "fall back" to a virtualized box.

And the nice thing is it looks like Server 2016 Essentials has a 6 month trial... so I can just try it with minimal risk.

 

I could even try something crazy: run server essentials under virtualization and pass through some/all of GPUs :-)

 

Now to find out if that board really has good ACS support...

 

 

 

[Verrm]

I don't know as much on the computer architecture side of things, but I read a bit here and there and this is what I think:

 

- 1 server should be used as multiple will not cover your requirements (it mgmt increase cost increase etc)

- You need many pcie ports and many pcie lanes. So you're looking at xeons and Threadripper/Epyc.

- Only HEDT mobos (X299 or X399) or server mobos will fulfill your needs

- You're on a budget so I would look at AMD side of things (especially incoming new cpus) or at the after-leasing used servers

- You can think about either pinning all monitors and peripherals to this 1 computer or try to setup VDI. Having this in mind:

-- First option you can gett 6-8 gpus and pin a separate monitor to each gpu, then dedicate this gpu and monitor to the given VM

-- Second option means easier resource mgmt and no need for 1 gpu for each VM. A cheap VDI-like experience can be achieved with Raspberry Pi 3.

-- You can investigate possibility of using moonlight/teamviewer/ other vdi alternatives.

- You probably can have a good use of Docker. Easy image and resource management. If not this then maybe KVM/Xen/ESX. I wouldn't use windows-based OS on a server. It doesn't give you anything beside costs.

- VDI or VDI-like alternatives transfers only signal, so if you assign a gpu to a given VM using VT-d or AMD-Vi then almost 100% of performance will be available for it.

 

If I were you I'd try to find a used-but-powerfull after-leasing Xeon server + stations for students with Rpi 3 based VDI-like alternatives. You can investigate Threadripper and/or Epyc, but likely the initial support would be tough.

[Electronics Wizardy]

7 hours ago, MakeItZone said:

Do you know if apps running on "stations" directly connected to a video card will have GPU acceleration? Or is that limited to only Remote RDP/RemoteFX clients?

Then will, its about the same as the remote desktop connections so you can run programs like maya will full gpu power.

 

7 hours ago, MakeItZone said:

Windows Server 2016 essentials (~$800, I'm not a school) supports up to 25 users- I'm hoping that includes Multipoint users. This would be very price effective!

I think you need to move to standard and buy cals.

 

Are you on AD?

 

7 hours ago, MakeItZone said:

The only downside is the Hyper-V role appears to be limited to max of 2VMs. 

This is on standard, i don't think its like this on essentials.

 

 

 

Why not just get a lot of cheap pcs? Like these

http://www.ebay.com/itm/Dell-Optiplex-3010-DT-i5-3-20GHz-4GB-RAM-500GB-HDD-Win-7-Pro-Great-Deal-/232301355927?hash=item36163d6397:g:eqwAAOSwvzRX0cqt

[bdubbs]

This is a fun solution but the cheaper solution will be multiple machines. You can lower management by creating windows images and re-imaging machines when students are done with them. Or use a solution to destroy the profile and any changes when a user logs out. Depending on your use case. 

 

It just doesn't make sense with your use case. This would make more sense if the students had their own laptops or a way to connect to these VMs. 

[MakeItZone]

I've got a cost for the workstation components, new for ~$3000. Buying refurbed PC's (in Canada) is ~$300 each, but then need an extra ~$100 for a GPU to make it closer to an "apples-to-apples" comparison. Add shipping, and it's very close to the same cost. And that's refurb vs new; so to be a fair comparison I'd need to look at refurb'd servers, which could be quite a bit cheaper.

 

So for what I can get locally, I'm not seeing much, if any, savings in older individual machines.

 

@Electronics Wizardy - by AD I assume you mean active directory? Then no. This would be a standalone machine. It's not clear how many VMs server '16 essentials supports; I maybe getting confused with some documentation about using the same license on the hardware and then also installed in a VM. If I have to move to standard or higher the per core pricing pushes the software cost too high.

 

I'm on the fence, am installing server 2016 on a machine to take a look at multipoint.

 

 

 

 

[Electronics Wizardy]

58 minutes ago, MakeItZone said:

Buying refurbed PC's (in Canada) is ~$300 each

Here is one for 150 http://www.ebay.ca/itm/Dell-Optiplex-3010-SFF-Core-i3-3220-3-3GHz-4GB-250GB-Win-7-Pro-1-Yr-Wty-/201757759917?hash=item2ef9b319ad:g:nR4AAOSw44BYW9Bo

 

 

 

 

What are you doign on these systems?

 

59 minutes ago, MakeItZone said:

ny VMs server '16 essentials supports;

seems to be limited to 1 or no vms included in the lincence, you can run as many as you want, but you need to pay for the linence. I don't think it supports multipoint either. Id just run standard.

[MakeItZone]

@Electronics Wizardy - ebay is telling me ~$300 CAD including shipping. It'd be a true pain for me to deal with them if some aren't up to snuff. Not to mention how everytime I order something cross-border I seem to get a nasty bill from the Tax Man.

 

This is for a classroom that needs to support CAD/CAM (Fusion 360, TinkerCad), other WebGL webapps, Minecraft, Lego, Vex and general programming environments, InkScape and some other graphical apps. Not huge CPU loads, but does require some hardware GPU acceleration.

 

Keeping an eye on the Server Essentials 2016 install. Just had to force install the Intel NIC... then video card drivers, then later tonight I'll figure out if multipoint is available in essentials and try to enable it.

[Electronics Wizardy]

50 minutes ago, MakeItZone said:

This is for a classroom

Then you should be able to get a big discount on microsoft lincensing, go talk to them and its very cheap for server.

 

 

[MakeItZone]

Nope- because I'm not an accredited school, part of a school system, etc. 

 

I'm just a plain old small business. At some point this may change if we move to a non-profit model. But right now I'm going to be paying list price :-(

[MakeItZone]

@Electronics Wizardy- you were correct re essentials not having remote/multipoint support. The role is listed both as a role and as part of the feature/role admin interface, but you just get an "unknown feature name" error during installation. More googling and using powershell to list the known features -> feature not listed.

 

More googling and I found a non-MS website that listed a summary of roles and which editions they're supported in. Multipoint is not listed for Essentials.

 

Also there seems to be no way to in-place upgrade from essentials -> standard. So I think it's a particularly stripped down version; so much so that some areas are broken.

 

And then I discovered there is no (obvious) way to make local login accounts beyond the first admin account.

 

Downloading and trying the Server Standard edition. Looks like CDW has a 10 cal box for about the cost of 10x Win 10 Home licenses... so it's worth trying just to see if it works. But I suspect (for new hardware) a virtualized workstation level board might end up being the most cost effective (for new parts.) If it works...

 

[MakeItZone]

And for anyone curious: the evaluation installer for Server Standard offers options to install standard or datacenter. It doesn't offer installation of essentials. So essentials appears to be a hacked down, almost broken in places, SKU, with no upgrade path to "full". Caveat emptor!