VMWare guests and Windows Server 2012

Finally VMware have released a KB for the dreaded Windows Server 2012 reboot and hang problem.
I’m sure this is familar to many 🙂

OMFG will you ever start???

Here’s a link to the KB from VMware

We started this off in our Test environment, just to be sure.

Firstly I used RVTools to generate a list of all guests in the TEST environment. I then filtered it down to just those with Windows Server 2012 as the OS and saved that as my txt file to target the script against.

To be sure my guests were all reporting the correct GuestID I uncommented lines 15/16 snd commented out 4-13. This gave me a file with the guest name and GuestID, which were all “windows8Server64Guest” so no dramas there.

I then reversed the commenting above and ran the script against my txt file.

$vms = Get-Content C:\scripts\2012servers-test.txt
foreach ($vm in $vms) {
$vmv = Get-VM $vm | Get-View
$name = $vmv.Name
$guestid = $vmv.Summary.Config.GuestId
$vmx = New-Object VMware.Vim.VirtualMachineConfigSpec
$vmx.extraConfig += New-Object VMware.Vim.OptionValue
$vmx.extraConfig[0].key = "monitor_control.enable_softResetClearTSC"
$vmx.extraConfig[0].value = "TRUE"
if ($guestid -like "windows8*Guest") {
($vmv).ReconfigVM_Task($vmx)
}
# $text = "VM: " + $name + " GuestId " + $guestid
# $text | out-file c:\scripts\output.txt -Append
}

Now the fix has been applied BUT it won’t actually stop the issue until the guests are either cold booted or vmotioned.

As it isn’t feasible to down these guests, even in test (and certainly isn’t an option in Prod) I stretched the process out over a week. After 4 days of normal service, I used the script below to report on which guests had been vmotioned by the system and then did the remaining moves manually.

$hours = 96 # Number of hours back
$start = (Get-Date).AddHours(-$hours)
$tasknumber = 999 # Windowsize for task collector
$eventnumber = 100 # Windowsize for event collector
$tgtTaskDescriptions = "VirtualMachine.migrate","Drm.ExecuteVMotionLRO"
$migrations = @()
$report = @()
# Get the guest for which we want the report
$vmHash = @{}
$vms = Get-Content C:\scripts\2012servers-test.txt
foreach ($vm in $vms) {
Get-VM $vm | %{$vmHash[$_.Name] = $_.VMHost}
}
# Retrieve the vMotion tasks and the corresponding events
$taskMgr = Get-View TaskManager
$eventMgr = Get-View eventManager
$tFilter = New-Object VMware.Vim.TaskFilterSpec
$tFilter.Time = New-Object VMware.Vim.TaskFilterSpecByTime
$tFilter.Time.beginTime = $start
$tFilter.Time.timeType = "startedTime"
$tCollector = Get-View ($taskMgr.CreateCollectorForTasks($tFilter))
$dummy = $tCollector.RewindCollector
$tasks = $tCollector.ReadNextTasks($tasknumber)
while($tasks){
$tasks | where {$tgtTaskDescriptions -contains $_.DescriptionId} | % {
$task = $_
$eFilter = New-Object VMware.Vim.EventFilterSpec
$eFilter.eventChainId = $task.EventChainId
$eCollector = Get-View ($eventMgr.CreateCollectorForEvents($eFilter))
$events = $eCollector.ReadNextEvents($eventnumber)
while($events){
$events | % {
$event = $_
switch($event.GetType().Name){
"VmBeingHotMigratedEvent" {
$migrations += New-Object PSObject -Property @{
VMname = $task.EntityName
Source = $event.Host.Name
Destination = $event.DestHost.Name
Start = $task.StartTime
Finish = $task.CompleteTime
Result = $task.State
User = $task.Reason.UserName
DRS = &{if($task.DescriptionId -like "Drm.*"){$true}else{$false}}
}
}
Default {}
}
}
$events = $eCollector.ReadNextEvents($eventnumber)
}
$ecollection = $eCollector.ReadNextEvents($eventnumber)
# By default 32 event collectors are allowed. Destroy this event collector.
$eCollector.DestroyCollector()
}
$tasks = $tCollector.ReadNextTasks($tasknumber)
}
# By default 32 task collectors are allowed. Destroy this task collector.
$tCollector.DestroyCollector()
# Handle the guests that have been vMotioned
$grouped = $migrations | Group-Object -Property VMname
$grouped | Sort-Object -Property Count -Descending | where{$vmHash.ContainsKey($_.Name)} | %{
$i = 1
$row = New-Object PSObject
Add-Member -InputObject $row -Name VM -Value $_.Name -MemberType NoteProperty
$_.Group | Sort-Object -Property Finish | %{
# The original location of the guest
if($i -eq 1){
Add-Member -InputObject $row -Name ("Time" + $i) -Value $start -MemberType NoteProperty
Add-Member -InputObject $row -Name ("Host" + $i) -Value $_.Source -MemberType NoteProperty
$i++
}
# All the vMotion destinations
Add-Member -InputObject $row -Name ("Time" + $i) -Value $_.Finish -MemberType NoteProperty
Add-Member -InputObject $row -Name ("Host" + $i) -Value $_.Destination -MemberType NoteProperty
Add-Member -InputObject $row -Name ("DRS" + $i) -Value $_.DRS -MemberType NoteProperty
Add-Member -InputObject $row -Name ("User" + $i) -Value $_.User -MemberType NoteProperty
$i++
}
$report += $row
$vmHash.Remove($_.Name)
}
# Add remaining guests to report
$vmHash.GetEnumerator() | %{
$row = New-Object PSObject
Add-Member -InputObject $row -Name VM -Value $_.Name -MemberType NoteProperty
Add-Member -InputObject $row -Name Time1 -Value $start -MemberType NoteProperty
Add-Member -InputObject $row -Name Host1 -Value $_.Value -MemberType NoteProperty
Add-Member -InputObject $row -Name DRS1 -Value $false -MemberType NoteProperty
$report += $row
}
$report | Export-Csv "C:\scripts\vMotion-history.csv" -NoTypeInformation -UseCulture

Thanks VMware to finally addressing this issue.

Here are the scripts in case you want a copy and would like the correct indents 🙂

VM-Report-vMotions-Servers-Test.ps1

VM-Update-2012-Servers-Test.ps1

Leave a Reply