Unable to login because of a ESXi root account lockout

When starting one of my VMware ESXi 6.5 lab hosts I was unable to login using the vSphere Host Client. I tried to make an SSH session to the host but got an “Access Denied” message.

When Using the Direct Console Interface (DCUI) I was able to login using the root account. In the log folder (under /var/log) I found that the root account is locked because of many failed attempt by investigate the following log files:

vobd.log

2018-01-02T10:57:00.003Z: [GenericCorrelator] 5612887277us: [vob.user.account.locked] Remote access for ESXi local user account 'root' has been locked for 900 seconds after 58 failed login attempts.
2018-01-02T10:57:00.003Z: [UserLevelCorrelator] 5612887277us: [vob.user.account.locked] Remote access for ESXi local user account 'root' has been locked for 900 seconds after 58 failed login attempts.
2018-01-02T10:57:00.003Z: [UserLevelCorrelator] 5612887502us: [esx.audit.account.locked] Remote access for ESXi local user account 'root' has been locked for 900 seconds after 58 failed login attempts.

auth.log

2018-01-02T11:02:08Z sshd[117700]: Connection from 192.168.249.23 port 63449
2018-01-02T11:02:09Z sshd[117701]: pam_tally2(sshd:auth): user root (0) tally 72, deny 5
2018-01-02T11:02:14Z sshd[117700]: error: PAM: Authentication failure for root from 192.168.249.23
2018-01-02T11:02:14Z sshd[117710]: pam_tally2(sshd:auth): user root (0) tally 73, deny 5

By default the ESXi 6.x password requirements for lockout behavior are:

  • A maximum of ten failed attempts is allowed before the account is locked
  • Password lockout is active on SSH and the vSphere Web Service SDK
  • Password lockout is not active on the Direct Console Interface (DCUI) and the ESXi Shell

To view the number of failed login attempt use the following command:


pam_tally2 --user root

In my example the there were 58 failed root login attempts:


Login Failures Latest failure From
root 58 01/02/18 10:56:59 unknown

The clear the the password lockout use the following command:


pam_tally2 --user root --reset

After this command I was able to login the vSphere Host Client. In the vSphere Host Client I found the VM that is causing the root account lockout:

The VM was monitoring the vSphere ESXi host with the wrong root password. After changing the password the account lockout problem was solved.

Deploy an OVA/OVF fails with certificate error

When trying to deploy an OVA/OVF with the vSphere Web Client the following error is displayed:

The operation failed for an undetermined reason. Typically this problem occurs due to certificates that the browser does not trust. If you are using self-signed or custom certificates, open the URL below in a new browser tab and accept the certificate, then retry the operation.

This error occurs with vSphere 6.5 because the certificates are not trusted. The self-signed certificates are used and are not added to the trusted root certification store.

To deploy a OVF/OVA to the vCenter Server appliance trusted root CA must be added to the certificate store. The following steps will work with Chrome and Internet Explorer:

  • Open the vCenter URL: https://vcenter-FQDN

  • Select the “Download trusted root CA certificates” and save the archive(ZIP) file
  • Extract the archive (ZIP)

  • Start – Run – MMC
  • File – Add Snap-ins – Certificates – Computer Account – Local  computer
  • Open Trusted Root Certification Authories – Certificates
  • Import the two *.crt certificates

  • Close the browser and re-open the browser and navigate to the vCenter Server using the FQDN.
  • Now the URL is marked as secure (green lock) and you’re able to import the OVA/OVF

 

vMotion between two vCenter Servers with different SSO domains

Last week i did my first vMotion between two vCenter Servers with different SSO domain by using PowerCLI. This functionality is also known as “cross vCenter vMotion” and is not included in the vSphere Web Client yet. Without downtime it’s possible to live migrate VMs from one vCenter Server to another. Cool stuff!

Before starting the following requirements must be met:

  • VMware vSphere 6.0 and later for the source and destination environment
  • PowerCLI 6.5 and later
  • vSphere Enterprise Plus license per ESXi host
  • An active connection to the source and destination  vCenter Server

I created a simple script to vMotion the VMs between the two SSO domains. See the more information section for more advanced PowerCLI scripts from for example William Lam and Romain Decker.

In the script the following variables must be defined:

  • Source vCenter Server, username and password
  • Destination vCenter Server, username and password
  • VM name will be moved
  • Destination vSwitch
  • Destination Portgroup
  • Destination datastore

Only VMs with one NIC are supported

PowerCLI example script

Import-Module VMware.PowerCLI
 
#Variables
$sourceVC = 'sourcevc'
$sourceVCUsername = 'administrator@vsphere.local'
$sourceVCPassword= 'password!'
 
$destVC = 'destinationvc'
$destVCUsername = 'administrator@vsphere.local'
$destVCPassword= 'password'
$destESXi = 'destinationesxi'
 
$vmname = 'vmname'
$Switchname = 'destinationswitch'
$NetworkName = 'destinationvlan'
$datastorename = 'destinationdatastore'
 
# Connect to the vCenter Servers
$sourceVCConn = Connect-VIServer -Server $sourceVC -user $sourceVCUsername -password $sourceVCPassword
$destVCConn = Connect-VIServer -Server $destVC -user $destVCUsername -password $destVCPassword
 
$vm = Get-VM $vmname -Server $sourceVCConn
$networkAdapter = Get-NetworkAdapter -VM $vm -Server $sourceVCConn
 
$destination = Get-VMHost -name $destESXi -Server $destVCConn
$destinationPortGroup = Get-VirtualPortGroup -VirtualSwitch $Switchname -name $NetworkName -VMHost $destination
$destinationDatastore = Get-Datastore -name $datastorename -Server $destVCConn
 
Move-VM -VM $vm -Destination $destination -NetworkAdapter $networkAdapter -PortGroup $destinationPortGroup -Datastore $destinationDatastore

With this script I migrated a couple of VMs between to vSphere 6.5 environment with different SSO domains without any downtime.

More information:

  • PowerCLI blog on the Move-VM cmdlet, link
  • William Lam, Automation Cross vCenter vMotion, link
  • Romain Decker Cross vMotion script from, link