Tasked to build a super weird setup

[KirbyTech]

So before I start, this is not my idea and the issues I am getting are some just weird.

 

I have been tasked to build a window 2012 r2 VM server with AD, DNS, DHCP and 2 Ubuntu linux VM's all of which need to run on ESXi. Sounds fine right? Well not so fast, I need to build this on Windows 10 VMWare WorkStation Pro 14.x inside with ESXi 5.5.

I also need to duplicate this and then change the subnet to a different one. Oh and need to put vCenter on there too.

This is a project to test the upgrade process to ESXi 6.5.u1

 

I have built it mostly and it is working well other than DHCP anyway (was on corp network didn't want to fuck up their DHCP).

 

My issue is I have no connectivity on it once I disconnect from the corp network. I also can't install a windows 10 vm in any possible way since it keeps failing.

 

My brain is fried at this point I have been at it since Monday....

 

So what I have hardware wise is a Dell r710, Dell 3020 desktop and a Cisco 3560-C. How in the hell do I set this up so I don't smash my head in? 

 

I am thinking 

 

SOFTWARE

 

                                                                                                          Windows 10

                                                                                                                  |

                                                                                                                 \/

                                                                                                         WorkStation

                                                                                                                  |

                                                                                 -----------------------------------------------------------

                                                                                 |                                                                     |

                                                                                \/                                                                    \/

                                                                           ESXi 5.5                                                         ESXi 5.5

                                                                                |                                                                      |

                                                                               \/                                                                     \/

                                                          --------------------------------------                        -------------------------------------------

                                                        |                                             |                        |                                                   |

                                                       \/                                            \/                       \/                                                  \/

                                               Server 2012                           Ubuntu x2            Ubuntu x2                               Server 2012 

                                                       |                                                                                                                          |

                                 ----------------------------------------                                                                             ----------------------------------------

                                 |                     |                        |                                                                             |                      |                        |

                                \/                    \/                       \/                                                                            \/                     \/                       \/

                              AD               DNS               DHCP                <- Services not VMs_ ->                  AD               DNS               DHCP

 

 

 

HARDWARE

 

                                         Dell R710 -- Nic 1 ----------- Corp network

                                                |

                                              Nic 0

                                                |

                                                |

                                                |

                                             Port 1

                                                |

                               Cisco Catalyst 3560-C  -- Port 8   ---------- Dell Optiplex 3020

                                                |

                                 Console Connection

                                                |

                                       Surface Pro 2

                   

 

Question then becomes what do I do for ESXi and Workstation settings?

Edited February 1, 2018 by KirbyTech
Clarification

[MrMcMuffinJr]

ESXi expert co-worker says you should be running server2016 instead of windows 10

 

edit: dont feel like typing this out

 

 

[KirbyTech]

Just now, MrMcMuffinJr said:

ESXi expert co-worker says you should be running server2016 instead of windows 10

 

I am only allowed to do this with the licenses I have been given. Why would I want Server 2016 anyway? I just need a platform to run workstation on.

[MrMcMuffinJr]

1 minute ago, KirbyTech said:

I am only allowed to do this with the licenses I have been given. Why would I want Server 2016 anyway? I just need a platform to run workstation on.

thats just what I interpenetrated from  this

[MrMcMuffinJr]

Just now, MrMcMuffinJr said:

thats just what I interpenetrated from  this

running it in a server instead of windows 10 makes more sense to me

 

[KirbyTech]

1 minute ago, MrMcMuffinJr said:

running it in a server instead of windows 10 makes more sense to me

 

As I said only given certain licenses and told to not use eval licenses either so I'm very limited. The software portion of what I have above is not going to change without a long fight and maybe pulling a few teeth too.

 

I posted hoping for some insight on how I can best utilize the hardware and what settings I will need to do so that I can have access on everything I need to have access. Ie the ESXi will need to be on 192.168.X.X while the corp network (and my internet connection) is on 10.X.X.X

[brwainer]

Just making sure I understand, you are trying to have 3 levels of virtualization?

ESXi on bare metal

Windows 10 running inside the top ESXi

2x ESXi running inside Windows 10 with vmware workstation

 

Do all of these support nested virtualization? I’m not a VMWare expert but I know nested virtualization isn’t meant to be more than 2 levels

[KirbyTech]

8 minutes ago, brwainer said:

Just making sure I understand, you are trying to have 3 levels of virtualization?

ESXi on bare metal

Windows 10 running inside the top ESXi

2x ESXi running inside Windows 10 with vmware workstation

 

Do all of these support nested virtualization? I’m not a VMWare expert but I know nested virtualization isn’t meant to be more than 2 levels

It is only two levels, you are adding in the esxi on bare metal. Maybe I phrased something wrong above.

[dalekphalm]

1 hour ago, KirbyTech said:

It is only two levels, you are adding in the esxi on bare metal. Maybe I phrased something wrong above.

I'm still confused.

Quote

 I need to build this on Windows 10 VMWare WorkStation Pro 14.x inside ESXi 5.5.

That sounds like a third level of nesting.

 

ESXi 5.5 -> Windows 10 VM running Workstation Pro -> Windows Server VM running Hyper V (At least I assume Hyper V - you didn't specify) -> DC, DNS, etc VM's

 

That's three levels of nesting.

 

EDIT: Unless you mean the Windows Server VM is simply running AD, DNS, etc, as services?

[KirbyTech]

44 minutes ago, dalekphalm said:

I'm still confused.

That sounds like a third level of nesting.

 

ESXi 5.5 -> Windows 10 VM running Workstation Pro -> Windows Server VM running Hyper V (At least I assume Hyper V - you didn't specify) -> DC, DNS, etc VM's

 

That's three levels of nesting.

 

EDIT: Unless you mean the Windows Server VM is simply running AD, DNS, etc, as services?

 

Windows 10 -> ESXi -> my linux/windows server VMs

 

That easier to understand? Maybe my diagram was too complex or something.

 

All the windows services (AD, DNS, DHCP) are running in a single VM to save my some time and license limits mostly.

 

Edit: I will update the diagram to better reflex this.

 

Edit 2: I changed the wording in the OP since I now see where the confusion was coming from, should not have used the word 'inside'. Made sense in my head at the time. 

Edited February 1, 2018 by KirbyTech
Edit 2

[brwainer]

2 hours ago, KirbyTech said:

 

Windows 10 -> ESXi -> my linux/windows server VMs

 

That easier to understand? Maybe my diagram was too complex or something.

 

All the windows services (AD, DNS, DHCP) are running in a single VM to save my some time and license limits mostly.

 

Edit: I will update the diagram to better reflex this.

 

Edit 2: I changed the wording in the OP since I now see where the confusion was coming from, should not have used the word 'inside'. Made sense in my head at the time. 

OK, so Windows 10 with VMWare Workstation, and inside VMWare Workstation you want to install ESXi, inside which will be some VMs. Have you read this article? https://kb.vmware.com/s/article/2034803

and also this: https://kb.vmware.com/s/article/2009916?r=2&Quarterback.validateRoute=1&KM_Utility.getArticleData=1&KM_Utility.getGUser=1&KM_Utility.getArticleLanguage=1&KM_Utility.getArticle=1

 

The baseline Windows 10 is being install on the R710? what CPUs are in it, and is there any BIOS options relating to EPT or SLAT? (look in both the CPU section as well as the Chipset or Platform section).

 

4 hours ago, KirbyTech said:

I also can't install a windows 10 vm in any possible way since it keeps failing.

are you having this issue at the Workstation level, or at the ESXi level?

[KirbyTech]

15 hours ago, brwainer said:

OK, so Windows 10 with VMWare Workstation, and inside VMWare Workstation you want to install ESXi, inside which will be some VMs. Have you read this article? https://kb.vmware.com/s/article/2034803

and also this: https://kb.vmware.com/s/article/2009916?r=2&Quarterback.validateRoute=1&KM_Utility.getArticleData=1&KM_Utility.getGUser=1&KM_Utility.getArticleLanguage=1&KM_Utility.getArticle=1

 

The baseline Windows 10 is being install on the R710? what CPUs are in it, and is there any BIOS options relating to EPT or SLAT? (look in both the CPU section as well as the Chipset or Platform section).

 

Correct and I will read those links yo said and see if they help Monday when I get to the office again. Racking Exagrids today. 

CPU's are 2 x 4 core not sure the model I think its E5 5620 or something like that. Again will check Monday. 

 

15 hours ago, brwainer said:

are you having this issue at the Workstation level, or at the ESXi level?

At the workstation level. Everything installs fine inside ESXi. I am going to try and see difference of config from work to a home setup fresh with a clear head. 

[Erkel]

I would be inclined to say RUN AWAY.  Whoever is telling you to do this is an idiot.

[KirbyTech]

2 hours ago, Erkel said:

I would be inclined to say RUN AWAY.  Whoever is telling you to do this is an idiot.

Not an option. It has to be done weather I like it or not. It is dumb yes. But not my choice to run from, it is a project for the city here and they determined that is how they want it done so be it I will figure it out.

[dalekphalm]

2 hours ago, KirbyTech said:

Not an option. It has to be done weather I like it or not. It is dumb yes. But not my choice to run from, it is a project for the city here and they determined that is how they want it done so be it I will figure it out.

Who actually decided how it would work?

 

Was it a senior IT person, or a manager/bureaucrat?

 

It just seems like a waste of time to do it the way you’re doing it.

 

Why not simply install ESXi bare metal onto the Windows 10 workstation and wipe Windows 10 from the HDD? Or just pull the HDD, set it aside, and throw in a spare to install ESXi?

 

After all, the base license for ESXi is free sooooo... makes sense to me. 

[KirbyTech]

29 minutes ago, dalekphalm said:

Who actually decided how it would work?

 

Was it a senior IT person, or a manager/bureaucrat?

 

It just seems like a waste of time to do it the way you’re doing it.

 

Why not simply install ESXi bare metal onto the Windows 10 workstation and wipe Windows 10 from the HDD? Or just pull the HDD, set it aside, and throw in a spare to install ESXi?

 

After all, the base license for ESXi is free sooooo... makes sense to me. 

It's not up for discussion, it is either being done that way or not at all. Seems some people are hard of hearing... Everyone seems to be missing the point of the post.. Not sure why you(all) can't understand that in a work environment you don't always get your way. I can't do ESXi on bare metal because look above at my diagram you should be able to figure that part out... Need TWO instances of ESXi on the same host running on different subnets. This is not going to change, the software layout is not changing no matter what some person here says. Unless you want to fly in, speak to the person who told me to do it, and then get me a second identical server, it just isn't going to happen.

 

I remember why I left this forum a while back... 

[brwainer]

I appreciate that you have to deal with decisions from people who fon’t understand the difficulties in execution, and I think everyone else here does also. The reason you’re getting all of these suggestions for other ways to conduct the test is because the way you were instructed is very difficult to pull off - I would even say that its not worth the time of your salary for the gain of the test, and you could make a solid argument on the basis of saving money for conducting this upgrade test a different way.

 

Let us know how things go on Monday with following the official VMWare articles on this type of setup that I linked, and we’ll continue to try to help out.

[Erkel]

I would go back to the person who told you to do it  this way and raise your concerns with them.

 

 

[dalekphalm]

8 hours ago, Erkel said:

I would go back to the person who told you to do it  this way and raise your concerns with them.

 

 

At this point, that's beating a dead horse. We've raised our concerns with the OP, and he is adamant that he cannot change the instructions he was given.

 

@brwainer gave him some good links to get started with. We'll see how the OP gets along on Monday or Tuesday.

[Jay Deah]

the reson this wont work (rather than why its a bad idea) is youre trying to run 2 nested instnces of ESXi on a single machine.

 

you need to PHYSICAL machines to do what yorue after.

 

but... what are you trying to achieve? proof of concept of a vSphere 5/6 -> 6.5 upgrade?

 

as if so, none of wht you are about to do will prove one way or the other if your upgrade will succeed or not! you are literally wasting your time.

 

the only way to know if your vcenter upgrade will work (and i assume youre going for VCSA as part of this) is to actually do it for real on either live machines or clones of live machines.

 

the only way to know if your ESXI hosts will work or not is to actually do it on those peices of hardware (and if theyre on the HCL, it WILL work)

 

ive done this stuff dozens of times, feel free to ask questions about how to actually test your upgrade path

[leadeater]

On 2/2/2018 at 8:58 AM, KirbyTech said:

I am only allowed to do this with the licenses I have been given. Why would I want Server 2016 anyway? I just need a platform to run workstation on.

Use ESXi Free Edition on the base hardware then create two ESXi VMs and license them with what ever way you have be told to do. The underlying hypervisor makes no difference to your lab setup and you'll have a far better time using ESXi at all layers than what you are trying to do and is commonly done in many lab environments and you'll find a few blog posts showing you the step by step process to do it.

 

Make your life easier, use ESXi Free direct on the hardware.

[KirbyTech]

16 hours ago, leadeater said:

Use ESXi Free Edition on the base hardware then create two ESXi VMs and license them with what ever way you have be told to do. The underlying hypervisor makes no difference to your lab setup and you'll have a far better time using ESXi at all layers than what you are trying to do and is commonly done in many lab environments and you'll find a few blog posts showing you the step by step process to do it.

 

Make your life easier, use ESXi Free direct on the hardware.

You come argue for that then, really don't get why you all are still pushing it when I said it's not going to change. If I had my way I would get a second server for $500 and do it that way. Or yes run ESXi bare metal. This is not my choice nor will it ever be. So the setup above sticks.

 

On 2/4/2018 at 4:00 AM, Erkel said:

I would go back to the person who told you to do it  this way and raise your concerns with them.

 

22 hours ago, dalekphalm said:

At this point, that's beating a dead horse. We've raised our concerns with the OP, and he is adamant that he cannot change the instructions he was given.

The reason I can't convince them to change it is because they just don't understand what they are talking about. The person who told me to do this is 60-65 ish and thinks that you can do ANYTHING with virtualization not matter the challenges with it. I have not done anything up until last week on it since I have been fighting to get a second server to test on. Heck it took 3 weeks or emails back and for me just to get more than one failing drive to work with since he also wants vSan installed. So while I agree with everyone here this is not the right way to do it, it is simply how I was told to and not given much options in changing that. I have sent an email this morning above his head asking for a $1k budget to do it properly but until then I need to work on this how it is.

 

16 hours ago, leadeater said:

Use ESXi Free Edition on the base hardware then create two ESXi VMs and license them with what ever way you have be told to do. The underlying hypervisor makes no difference to your lab setup and you'll have a far better time using ESXi at all layers than what you are trying to do and is commonly done in many lab environments and you'll find a few blog posts showing you the step by step process to do it.

 

Make your life easier, use ESXi Free direct on the hardware.

Read the above and you will see changing from what I am told is just not going to happen. I agree there are better ways but simple problem is I can't change it so no point in hearing it from this forum 50 times after I stated it was not changing.

 

19 hours ago, Jay Deah said:

the reson this wont work (rather than why its a bad idea) is youre trying to run 2 nested instnces of ESXi on a single machine.

 

you need to PHYSICAL machines to do what yorue after.

 

but... what are you trying to achieve? proof of concept of a vSphere 5/6 -> 6.5 upgrade?

 

as if so, none of wht you are about to do will prove one way or the other if your upgrade will succeed or not! you are literally wasting your time.

 

the only way to know if your vcenter upgrade will work (and i assume youre going for VCSA as part of this) is to actually do it for real on either live machines or clones of live machines.

 

the only way to know if your ESXI hosts will work or not is to actually do it on those peices of hardware (and if theyre on the HCL, it WILL work)

 

ive done this stuff dozens of times, feel free to ask questions about how to actually test your upgrade path

Yes I am trying for a 5.5 to 6.5 u1 upgrade. Can you point me to some docs that would show I can't do this so I can have actual proof that it will not work instead of just having it so I am fighting the boss.

What exact parts of this won't work, more than just general. I know in order to either get what I need or just to not do it I will need at least a few pages on VMWare docs and a written page or two by me breaking it down into simple steps. 

I have sent off an email to someone else with buying power to see if they will get me 2 servers with one having multiple drives in it instead of the one 146GB drive that is failing I have now. 

[leadeater]

17 hours ago, KirbyTech said:

You come argue for that then, really don't get why you all are still pushing it when I said it's not going to change. If I had my way I would get a second server for $500 and do it that way. Or yes run ESXi bare metal. This is not my choice nor will it ever be. So the setup above sticks.

My suggestion is not to run ESXi bare metal though, it's to replace the Windows 10 and Workstation with ESXi and then create 2 ESXi VMs on that, your diagram you posted does not change in layout at all. Replacing the bare metal Windows 10 install with ESXi is all that I am suggesting, it's only a tool change not a design change.

 

Have you even asked though? I get what you're saying however the functional requirements are what matters and that doesn't change if using ESXi or VMware Workstation and the VM compatibility is the same, both will run up the ESXi nested VMs created by each other. The difference is ESXi is catered to actually do what you need where VMware Workstation is not.

 

I understand you're annoyed at getting suggestions to change the setup, some were a big change to the design but this one isn't so I see no reason to not propose it if it doesn't work using Workstation. As the documentation @brwainer pointed to shows you can do it so try that first, I would never use Workstation for this though like I wouldn't use my shoe to try and hammer a nail in (you can do it sure but use a hammer).

 

17 hours ago, KirbyTech said:

Yes I am trying for a 5.5 to 6.5 u1 upgrade. Can you point me to some docs that would show I can't do this so I can have actual proof that it will not work instead of just having it so I am fighting the boss.

What exact parts of this won't work, more than just general. I know in order to either get what I need or just to not do it I will need at least a few pages on VMWare docs and a written page or two by me breaking it down into simple steps. 

It can work with both, if you need the two nested ESXi hosts to properly talk to each other so the VMs on each can and for it to be functionally the same as two physical servers you need to enable network promiscuous mode. From what I can see that is very hit and miss for VMware Workstation.

https://superuser.com/questions/1139028/configuring-promiscuous-mode-in-vmware-workstation-running-on-windows-10-serving

https://communities.vmware.com/thread/474611

 

And the why you need it:

Quote

Promiscuous Mode:
Both VMware VSS (Virtual Standard Switch) and VDS (vSphere Distributed Switch) do not implement MAC Learning like a traditional network switch, since the vSphere platform already knows which MAC addresses are assigned to a particular Virtual Machine. This means that the virtual switch will only forward network packets to a Virtual Machine if the destination MAC Address matches the ESXi vmnic's (pNIC) MAC Address.

 

In a Nested ESXi environment where you can have Nested Virtual Machines, the destination MAC Address for network packets destined to those Virtual Machines will differ from the Nested ESXi vmnic's MAC Address. Due to this, the physical ESXi host's virtual switch will drop the packet if Promiscuous Mode is not enabled. Promiscuous Mode allows the underlying Nested ESXi VM vmnic to monitor all traffic of the virtual switch it is connected to and thus providing connectivity to the underlying Nested Virtual Machines.

https://www.virtuallyghetto.com/2013/11/why-is-promiscuous-mode-forged.html

 

A good full guide on how to do it using ESXi with all the extra required steps:

https://www.altaro.com/vmware/how-to-set-up-a-nested-vsphere-6-environment-part-1/