HP AMS driver problem in ESXi 5.1

The last weeks several customers contacted me that one or more ESXi hosts are not manageable anymore. The ESXi host has the following symptoms:

  • Unable to manage using the vSphere Client, Web Client or SSH.
  • Unable to perform a vMotion
  • On the ESXi console there is a message: can’t fork
  • Enable services from the console does not work

The VMs are still running on the ESXi host. This is a known issue and can occur when running the following versions of the HP AMS driver:

  • hp-ams 500.9.6.0-12.434156
  • hp-ams-550.9.6.0-12.1198610

To solve this issue shutdown the VMs by using RDP or SSH and reboot or reset the ESXi server.  The HP AMS driver must be upgraded to version 10.0.1 or can be disabled.  The VMware KB can be found here.

I created a PowerCLI script to list the HP-AMS driver version of each ESXi host.

Script:

## Problem with HP AMS hp-ams 500.9.6.0-12.434156 and hp-ams-550.9.6.0-12.1198610
## http://kb.vmware.com/selfservice/microsites/search.do?language=en_US&cmd=displayKC&externalId=2085618

# Add PowerCLI module
Add-PSSnapin vmware.VimAutomation.core -ErrorAction SilentlyContinue

# VMware VirtualCenter server name
$VCserver = Read-Host "Enter your vCenter server"
$Username= Read-Host "Enter the Username"
$Password = Read-Host "Enter Password"

# Connect to the vCenter Server
Connect-VIServer -Server $VCserver -User $Username -Password $Password -port 443

# Clear screen
Clear

$GetHosts = Get-VMHost | Where {$_.ConnectionState -eq “Connected”}
ForEach ($VMHost in $GetHosts) {
$ESXCLI = Get-EsxCli -VMHost $VMHost
Write-Host $VMHost -ForegroundColor Green
$ESXCLI.software.vib.list() | Where {$_.Name -eq “hp-ams”} | Select Name,Vendor,Version | FT
}

# Disconnect session vCenter
Disconnect-VIserver -Confirm:$false

Example output:

hpams

Monitor VMware Horizon View environments with ControlUp

A couple of weeks ago ControlUp released version 4.0 with support for VMware Horizon View. In this blog post I share my experience with ControlUp 4.0 monitoring VMware vSphere and VMware Horizon View.

What is ControlUp

ControlUp is a real-time performance monitor for Microsoft Remote Desktop Services (RDS), Citrix, and physical- and virtual server environments. ControlUp can be used to:

  • Troubleshoot performance issues in real time
  • Analyze performance trends and usage patterns
  • Compare and manage multiple computers
  • Investigate incidents and receive email alerts

The new 4.0 version of  ControlUp has added support for:

  • VMware vSphere (4.x and 5.x)
  • Citrix XenServer (6.x)
  • VMware Horizon View desktops (5.x and 6.x)

ControlUp is tested against the following lab environment:

  • VMware ESXi 5.5 Update 2
  • VMware Horizon View 6.x environment with the composer
  • The VDI desktops are part of a floating pool
  • Windows 7 64-bit as VDI desktop OS.

Installation and configuration

The installation is very simple. On a management server execute a single executable (ControlUpConsole.exe).  It’s runs in memory, so there is no installation needed. The console is the GUI for displaying data and running tasks. As prerequisites .Net Framework 3.5 SP1, Active Directory connection and a internet connection  is required. ControlUp will default enabled in Enterprise Mode. The Enterprise Mode offers features that allow you to collaborate with team members and defining user roles and delegate administrative  tasks to different ControlUp users. The other option is to use Standalone Mode. Standalone Mode does not require an active  internet connection but is limited, for example in collaboration and delegation.

After creating an account is is time to create one or more organization(s). An organization represents groups of computers managed by the same administrators. By creating additional organizations it is possible to segment your network computers into different administrative units managed by different administrators.  After this the configuration is ready.

Monitoring VMware environments

For VMware environments you need to add the vCenter Server to the ControlUp Console.  When the vCenter is added, the cluster(s), hosts and VM’s inventory is displayed in the ControlUp console. In the following example we added a vCenter server with a 1 cluster  that contains 3 ESXi hosts.

ESXI hosts

To organize the computers a folder tree is created. For ESXi hosts, counters on CPU, Memory, Networking and storage can be monitored. For every counter a specific tresholds per folder can be configured. Here is an overview of counters that can be monitored on the vSphere layer:

stresslevel

The Stress level reflects the state of the performance metric reported to the console. In the following example we stressed  the ESXi hosts memory threshold (90% >)  by powering extra VMs.

esxmon2

The stress level jumped to high (red) within seconds.  So in real time you see what is happening to the vSphere layer. In the next example we stressed the datastore latency (> 22 ms)  and free datastore space (below 500 MB).

esxmon3

In the above examples we see when a threshold is exceeded on the VMware ESXi hosts, it is displayed almost in real time in the ControlUp Console.

Monitoring VMware Horizon View VDI Desktops

To monitor VMware Horizon VDI desktops,  a lightweight ControlUp agent needs to be installed. The ControlUp agent can be installed in the Golden Image or deployed when the VDI desktop is running. It is important that firewall ports on RCP, WMI, Windows Remote Management and the ControlUp agent firewall ports are allowed. After adding the ControlUp agent, VMware View specific settings can be displayed.

view specific session settings

In the following example we have 4 Windows 7 VDI desktops. As you can see the stress level is low or medium.

View session 1

In a Windows 7 VDI desktop we installed to 3e party tool “heavyload” to generate CPU load on the VDI desktop. When starting “Heavyload” the CPU spiked to 100%.

 view sesion 2  view session 3

In the ControlUp console we looking in the Computer tab and see the Stress level after a couple of seconds changing to “High” and the CPU graph is raising to 100%.

view session 4

To see what is causing the high CPU load we opened “Processes” tab and  sort on the Stress Level to see what process is causing the CPU load. We see that the Process “HeavyLoad.exe” is causing the high load.

View session 5

On the right menu you see some actions that be performed. Here are some examples:

  • Getting a Screenshot of the desktop
  • Ending or killing the process
  • Throttle the CPU for a process

In the following example we use the tool “heavyload” again to generate memory load.  Now we see that the Stress Level is critical again and that the Memory is red.

view session monitoring memory

To see what is causing the high memory load, open the” Processes” tab and  sort on the Stress Level to see what process is causing the high memory load. And again the process “HeavyLoad.exe” is causing the high memory load.

view session memory 1

Triggers

When a service stops or a event occur it is possible to generate an alert. Using the “Incident Triggers” feature you can configure triggers to detect the following conditions:

  • Stress level
  • Windows Event
  • Computer Down
  • Process Started or Ended
  • User Logged On/User Logged Off
  • Session State changed

When a trigger occur an real time alert can be generated such as sending an email. There are pre-configured incident triggers for Horizon View services and events that occur.

incident triggers

Testing ControlUp

ControlUp has a launched an Expert Program for vExperts and recognized VMware View Consultants. To apply for the license, register here. All other VMware View admins can simply download ControlUp from the ControlUp website and enjoy the free unlimited 30 days trial.

Conclusion

Version 4 is the first version that supports VMware vSphere and Horizon View. In this blogpost I highlighted a couple of examples that shows the strength of ControlUp and how easy it is to install and use. In real time you see what is happing in your environment and drill from the hypervisor down to the Windows process level to identify what is causing the problem. This makes troubleshooting a lot easier on VMware Horizon View environments. The VMware Horizon View specific counters are a bit limited on the moment but ControlUp is asking your feedback on this. If you miss something let them know by using the “feature request” button.

ControlUp has a lot more features then showed in this blog post. Other features are for example:

  • RDP to computer
  • Remote Assistance
  • Use RunAs accounts for actions
  • Update and killing Group Policies
  • Sending messages and chatting with users
  • Multiple computers management on the file system, registry and services.

All these features makes ControlUp a powerful tool for the VMware Horizon View Administrator.

Troubleshoot replication problems between VMware Horizon View Connection servers

In environments with multiple Horizon View Connection Servers (High Available) the Lightweight Directory Access Protocol (LDAP) directory is replicated. Configuration, Pool and desktop information is stored in the ADAM database. Problems with replication can result in:

  • Configuration changes made are not replicated
  • Authentication problems

The VMware View Administrator dashboard does not check the replication status.  So regular checking the replication status is a good thing. Checking the replication status can be done by using the following command: repadmin.exe /showrepl localhost:389 DC=vdi,DC=vmware,DC=int

When the replication is okay, it looks like something below:

image

When there are problems with the replication between the  Connection Servers errors as “8453 Replication access was denied”, “1772 The list of RPC servers available for the binding of auto handles has been exhausted” or 8457 The destination servers is currently rejecting replication requests” are displayed using the repadmin utility.

3

When having replication problems, changes are not replicated. Here is an example when logging in the VMware View Administrator  on each View Connection server, it displays differences in the ” Preparing”  ” Problem Desktops”  and ” Prepare for use” amounts.

4 5

 

Here are some tips for troubleshooting replication problems:

  • Check if port 389 is open on the View Connection Servers
  • Restart the VMware View LDAP directory service (VMware VDMS service). It will restart the View Connection Server service.

image

  • Force replication by using the following command: repadmin.exe /replicate fqdn-localhost:389 fqdn-remotehost:389 dc=vdi,dc=vmware,dc=int
  • Change the fqdn-locahost and fqdn-remote host to the View Connection Server names in your environment

image

  • Ensure that replication has not been disabled by using the following command: repadmin /options localhost:389 -DISABLE_OUTBOUND_REPL -DISABLE_INBOUND_REPL

With the checks listed above the replication can be checked and in most cases repaired if it is broken.

 

VMware vCloud suite 5.8 and Horizon View 6.0.1 product updates

Yesterday ( 9 September 2014)  VMware released updates to the vCloud Suite 5.8 suite and VMware Horizon View components.  Here is a overview of the updated products:

 

Error occurred during vCenter operation when editing a VMware Horizon View pool

When editing a VMware Horizon desktop pool in some situations the server error ‘Error occurred during vCenter operation’. appears.

1

The error hasn’t any useful information that says what is wrong. When this error occurs the vCenter Server is unable to find one or more objects listed in the desktop pool. When changing datacenter, cluster or host settings in vCenter the desktop pool must be updated to.  To resolve this error open the desktop pool, select the vCenter settings tab an check or update the Parent VM, Snapshot, Host or cluster, Resource pool and datastores. Also check if the ESXi host with local storage are not in maintenance mode! After updating these settings the error must be gone.

2