HP AMS driver problem in ESXi 5.x

The last weeks several customers contacted me that one or more ESXi hosts are not manageable anymore. The ESXi host has the following symptoms:

  • Unable to manage using the vSphere Client, Web Client or SSH.
  • Unable to perform a vMotion
  • On the ESXi console there is a message: can’t fork
  • Enable services from the console does not work

The VMs are still running on the ESXi host. This is a known issue and can occur when running the following versions of the HP AMS driver:

  • hp-ams 500.9.6.0-12.434156
  • hp-ams-550.9.6.0-12.1198610
  • hp-ams 500.10.0.0-18.434156 (Added 13-11-2014)
  • hp-ams-550.10.0.0-18.1198610  (Added 13-11-2014)

In some situations the ESXi servers is still manageable in vCenter, so vMotion can be used . In all the other situations shutdown the VMs by using RDP or SSH and reboot or reset the ESXi server. The HP AMS driver must be upgraded to version 10.0.1 or can be disabled.  The VMware KB can be found here.

I developed a simple PowerCLI script to list the HP-AMS driver version of each ESXi host.

Update 17-12-2014: I developed a new PowerShell tool called VIBSearch. This this graphical tool it is easier to search a specific VIB. More information can be found here: Link

Script:

## Problem with HP AMS hp-ams 500.9.6.0-12.434156 and hp-ams-550.9.6.0-12.1198610
## http://kb.vmware.com/selfservice/microsites/search.do?language=en_US&cmd=displayKC&externalId=2085618

# Add PowerCLI module
Add-PSSnapin vmware.VimAutomation.core -ErrorAction SilentlyContinue

# VMware VirtualCenter server name
$VCserver = Read-Host "Enter your vCenter server"
$Username= Read-Host "Enter the Username"
$Password = Read-Host "Enter Password"

# Connect to the vCenter Server
Connect-VIServer -Server $VCserver -User $Username -Password $Password -port 443

# Clear screen
Clear

$GetHosts = Get-VMHost | Where {$_.ConnectionState -eq “Connected”}
ForEach ($VMHost in $GetHosts) {
$ESXCLI = Get-EsxCli -VMHost $VMHost
Write-Host $VMHost -ForegroundColor Green
$ESXCLI.software.vib.list() | Where {$_.Name -eq “hp-ams”} | Select Name,Vendor,Version | FT
}

# Disconnect session vCenter
Disconnect-VIserver -Confirm:$false

Example output:

hpams

VMware patch available for the product license to expire on 12 august 2008

VMware has released a patch for VMware ESX 3.5 and VMware ESX 3.5i.

1.Read the following Knowledge Base articles first:

Fix of virtual machine power on failure issue, refer to KB 1006716

For VI 3.5, refer to KB 1006721 for deployment consideration and instruction

For VI3.5i, refer to KB 1006670 for deployment consideration and instruction

2. Download and apply the patch according to the product(s) you have:

VMware ESXi 3.5 Update 2 Patch | VMware ESX 3.5 Update 2 Patch

ESX 3.5 Update 2 and ESXi 3.5 Update 2 issue with product license to expire

 You got the following error when powering on a VM:

Unable to Power On virtual machine with “A General System error occurred: Internal error”

An issue has been uncovered with ESX 3.5 Update 2 and ESXi 3.5 Update 2 that causes the product license to expire on August 12. VMware is alerting customers and partners of this issue. VMware regrets the inconvenience caused to customers. Updated product bits with correct licensing will be made available for download as soon as possible.

 

The work-around: turn off NTP (if you’re using it), and then manually set the date of all ESX 3.5u2 hosts back to 10th of August. This can be done either through the VI Client (Host -> Configuration -> Time Configuration) or by typing date -s “08/10/2008” at the Service Console command line on the ESX hosts.

 

This seems to affect initial VM power-on (including from suspended state) and VMotion.

On the VMTN forum there is a thread post about this problem, check it here

Update:

VMware has created a KB article:

Unable to Power On virtual machine with “A General System error occurred: Internal error”

 http://kb.vmware.com/selfservice/dynamickc.do?externalId=1006716&sliceId=1&command=show&forward=nonthreadedKC&kcId=1006716