Unable to login because of a ESXi root account lockout

When starting one of my VMware ESXi 6.5 lab hosts I was unable to login using the vSphere Host Client. I tried to make an SSH session to the host but got an “Access Denied” message.

When Using the Direct Console Interface (DCUI) I was able to login using the root account. In the log folder (under /var/log) I found that the root account is locked because of many failed attempt by investigate the following log files:

vobd.log

2018-01-02T10:57:00.003Z: [GenericCorrelator] 5612887277us: [vob.user.account.locked] Remote access for ESXi local user account 'root' has been locked for 900 seconds after 58 failed login attempts.
2018-01-02T10:57:00.003Z: [UserLevelCorrelator] 5612887277us: [vob.user.account.locked] Remote access for ESXi local user account 'root' has been locked for 900 seconds after 58 failed login attempts.
2018-01-02T10:57:00.003Z: [UserLevelCorrelator] 5612887502us: [esx.audit.account.locked] Remote access for ESXi local user account 'root' has been locked for 900 seconds after 58 failed login attempts.

auth.log

2018-01-02T11:02:08Z sshd[117700]: Connection from 192.168.249.23 port 63449
2018-01-02T11:02:09Z sshd[117701]: pam_tally2(sshd:auth): user root (0) tally 72, deny 5
2018-01-02T11:02:14Z sshd[117700]: error: PAM: Authentication failure for root from 192.168.249.23
2018-01-02T11:02:14Z sshd[117710]: pam_tally2(sshd:auth): user root (0) tally 73, deny 5

By default the ESXi 6.x password requirements for lockout behavior are:

  • A maximum of ten failed attempts is allowed before the account is locked
  • Password lockout is active on SSH and the vSphere Web Service SDK
  • Password lockout is not active on the Direct Console Interface (DCUI) and the ESXi Shell

To view the number of failed login attempt use the following command:


pam_tally2 --user root

In my example the there were 58 failed root login attempts:


Login Failures Latest failure From
root 58 01/02/18 10:56:59 unknown

The clear the the password lockout use the following command:


pam_tally2 --user root --reset

After this command I was able to login the vSphere Host Client. In the vSphere Host Client I found the VM that is causing the root account lockout:

The VM was monitoring the vSphere ESXi host with the wrong root password. After changing the password the account lockout problem was solved.

Deploy an OVA/OVF fails with certificate error

When trying to deploy an OVA/OVF with the vSphere Web Client the following error is displayed:

The operation failed for an undetermined reason. Typically this problem occurs due to certificates that the browser does not trust. If you are using self-signed or custom certificates, open the URL below in a new browser tab and accept the certificate, then retry the operation.

This error occurs with vSphere 6.5 because the certificates are not trusted. The self-signed certificates are used and are not added to the trusted root certification store.

To deploy a OVF/OVA to the vCenter Server appliance trusted root CA must be added to the certificate store. The following steps will work with Chrome and Internet Explorer:

  • Open the vCenter URL: https://vcenter-FQDN

  • Select the “Download trusted root CA certificates” and save the archive(ZIP) file
  • Extract the archive (ZIP)

  • Start – Run – MMC
  • File – Add Snap-ins – Certificates – Computer Account – Local  computer
  • Open Trusted Root Certification Authories – Certificates
  • Import the two *.crt certificates

  • Close the browser and re-open the browser and navigate to the vCenter Server using the FQDN.
  • Now the URL is marked as secure (green lock) and you’re able to import the OVA/OVF

 

What to check before upgrading to vSphere 6.5

Last week vSphere 6.5 was released (GA). This release has a lot of new cool features (see this link for more information). In the past I saw vSphere environments that are upgraded without proper preparation resulting in a rollback because compatibility issues with hard-or software. So I created a simple list with steps to check before upgrading to vSphere 6.5:

  • Check the hardware against the VMware Compatibility Guide, link
    • There is a PowerCLI script to check the hardware against the VMware Compatibility Guide, link
    • Devices deprecated and unsupported in ESXi 6.5, link
  • Check if all vSphere products are supported by vSphere 6.5. The following product are not supported yet (when writing this blog):
    • VMware NSX
    • VMware Integrated OpenStack
    • vCloud Director for Service Providers
    • vRealize Infrastructure Navigator
    • App Volumes
    • Horizon Air Hybrid-Mode
    • Integrated OpenStack
    • vCloud Networking and Security
    • vRealize Business for Cloud
    • vRealize Configuration Manager
    • vRealize Hyperic
    • vRealize Networking Insight
  • Check the “Important information before upgrading to vSphere 6.5 article, link
  • Check the update sequence for vSphere 6.5 and its compatible VMware products, link
  • Check if all the third-party products are supported by vSphere 6.5. For example last week Veeam Backup & Replication 9.5 is released. This release has no support yet for vSphere 6.5. Veeam Availability Suite 9.5 Update 1 will add support for vSphere 6.5.
  • The existing vSphere 6.0 license keys are supported for vSphere 6.5. No new license key are needed. More info: link
  • Check the vSphere 6.5 upgrade documentation, link
  • Always install vSphere 6.5 first in non-production environments and test all the critical stuff for some time. vSphere 6.0 had some nasty Change Block Tracking (CBT) bugs that you don’t want in your production environment.
  • Check the supported and deprecated topologies for VMware vSphere 6.5 article, more info: link
  • The vSphere Windows (C#) Client is  deprecated. Use the vSphere Web client of the new HTML5 based Client.
  • VMFS6 is the new filesystem of vSphere 6.5. VMFS6 cannot be inline or offline upgraded from VMFS5 to VMFS6. More info: link
  • TLS protocol versions 1.0, 1.1, and 1.2 are enabled by default in vSphere 6.5. More information about disabling TLS 1.0 can be found here: link.