Cannot allocate more then 32GB of RAM to Hyper V VMs after Upgrade

[Black3nding]

Hey!

 

We upgraded the Server of a Customer from 32GB to 64GB of RAM.

Windows shows all 64GB as useable.

But if we want to give the VMs in the HypeV Mananger more RAM then 32 GB combined they wont start anymore.

Old Setup was Host 4GB - Domain 4GB - TerminalServer 24GB.

New Setup shuold be Host 8GB - Domain 8GB - TerminalServer 48GB.

If the Domian gets 8 and the TS stays at 24 everything is fine - but as soon as we cross 32GB combined from TS and Domain the Machine that would cause this ( so the one we stard 2nd ) wont start anymore...

 

Anyone got an Idea?

[leadeater]

What is the Windows OS version of the Hyper-V host?

[brwainer]

Sounds to me like you have a NUMA issue - i.e. 32GB of RAM is connected to one CPU, and 32GB of RAM is connected to another CPU. This page is from Windows 2012 but I think it still applies.

https://docs.microsoft.com/en-us/previous-versions/windows/it-pro/windows-server-2012-r2-and-2012/dn282282(v=ws.11)

[Black3nding]

3 minutes ago, leadeater said:

What is the Windows OS version of the Hyper-V host?

Its an Server 2012R2 Standard.

[Black3nding]

4 minutes ago, brwainer said:

Sounds to me like you have a NUMA issue - i.e. 32GB of RAM is connected to one CPU, and 32GB of RAM is connected to another CPU. This page is from Windows 2012 but I think it still applies.

https://docs.microsoft.com/en-us/previous-versions/windows/it-pro/windows-server-2012-r2-and-2012/dn282282(v=ws.11)

Its just a single CPU with 4 out of 8 Ports populated. Does that article still matter for us? 

[NelizMastr]

4 minutes ago, brwainer said:

Sounds to me like you have a NUMA issue - i.e. 32GB of RAM is connected to one CPU, and 32GB of RAM is connected to another CPU. This page is from Windows 2012 but I think it still applies.

https://docs.microsoft.com/en-us/previous-versions/windows/it-pro/windows-server-2012-r2-and-2012/dn282282(v=ws.11)

Good catch. If the host has 2 physical CPUs, this might very well play a role. Afaik, you can force the VM to ignore the NUMA domains and just steal the memory from the other CPU, but performance will suffer. It'd be a shame though, as you'd need 56GB per CPU now to get what OP wants.

[leadeater]

16 minutes ago, Black3nding said:

Its an Server 2012R2 Standard.

There should be an error when you try and start the last VM, can you post that? If you can't find it it'll also be in Event Viewer.

[NelizMastr]

1 minute ago, Black3nding said:

Its just a single CPU with 4 out of 8 Ports populated. Does that article still matter for us? 

NUMA is of no concern with a single CPU. We'll need eventlog data now

[leadeater]

7 minutes ago, brwainer said:

Sounds to me like you have a NUMA issue - i.e. 32GB of RAM is connected to one CPU, and 32GB of RAM is connected to another CPU. This page is from Windows 2012 but I think it still applies.

https://docs.microsoft.com/en-us/previous-versions/windows/it-pro/windows-server-2012-r2-and-2012/dn282282(v=ws.11)

 

Just now, NelizMastr said:

Good catch. If the host has 2 physical CPUs, this might very well play a role. Afaik, you can force the VM to ignore the NUMA domains and just steal the memory from the other CPU, but performance will suffer. It'd be a shame though, as you'd need 56GB per CPU now to get what OP wants.

Don't think this is the problem as that's more for a single VM that would span across multiple NUMA nodes, this is multiple VMs which can just be placed on a NUMA node that fits. Even then spanning NUMA nodes works as it is and all OS versions Windows 2012 and up are NUMA aware in a VM hosted on Server 2012 and up.

[Black3nding]

3 minutes ago, leadeater said:

There should be an error when you try and standard the last VM, can you post that? If you can't find it it'll also be in Event Viewer.

Since these VMs are currently running I cant reproduce the ErrorMessage right now. Where do I find the corresponding entry int the Event Viewer?

[leadeater]

Just now, Black3nding said:

Since these VMs are currently running I cant reproduce the ErrorMessage right now. Where do I find the corresponding entry int the Event Viewer?

Normally I just Right click View All on Errors and then Warnings until I find what I'm looking for, on the home/top of tree of Event Viewer

 

 

Don't have Hyper-V installed currently but here is a pic of where it should be if you can't find them in the above

[Black3nding]

Thanks for the tip with the EventViewer - seems like we ran out of Space on the HardDrive . Gonna make some Space later today (as soon as we have the possibility to shut down the VMs and then test again.

Im writinig if it worked as soon as we know it ( may be tomorrow tho ).

 

Thanks for now ! 

[NelizMastr]

15 hours ago, Black3nding said:

Thanks for the tip with the EventViewer - seems like we ran out of Space on the HardDrive . Gonna make some Space later today (as soon as we have the possibility to shut down the VMs and then test again.

Im writinig if it worked as soon as we know it ( may be tomorrow tho ).

 

Thanks for now ! 

Must be related to swap. When you add more memory, you also need more space for swap

[Black3nding]

Update: After cleaning up some Space on the Harddrive everthing works as intended ! Thank you all for the help