VMware Tools 10.3.0 recalled, check you’re vSphere environment!

Yesterday VMware released a Knowledge Base article that VMware Tools version 10.3.0 is recalled because issues with the VMXNET3 network driver for Windows on ESXi 6.5. The issues can cause a Purple Diagnostic Screen (PSOD) or guest network connectivity loss. Because of these issues, VMware Tools 10.3.0 is recalled and no longer available.

Update: September 12, 2018: VMware Tools 10.3.2. is released that fixes the VMXNET3 issue. More information can be found here, link.

Action is required if VMware Tools 10.3.0 is deployed and the following is true:

  • vSphere ESXi 6.5 hosts
  • VM Hardware Version 13
  • Windows 8/Windows Server 2012 or higher guest OSes

If this is the case uninstall VMware Tools 10.3.0 and reinstall VMware Tools 10.2.5 from the VMware Downloads page, link. For other configurations, no immediate action is required.

I created a simple PowerCLI script to identify VMware Tools version 10.3.0 and display the Hardware Version and Operating System. With this script you can do quick check if you’re vSphere 6.5 environment contains VMware Tools 10.3.0 with Hardware Version 13  and Windows 8/Windows 2012 or higher VMs.

The script ‘”identVMwaretools.ps1″ can be found on my GitHub repository, link. The KB can be found here, link.

Upgrading a vCenter Server Appliance (VCSA) to version 6.7

Last week VMware launched vSphere 6.7. In this blog post I show how easy it is to upgrade a vCenter Server 6.x appliance to a new vCenter Server 6.7 appliance using the graphical interface (GUI) upgrade. The GUI upgrade uses a two stage process:

  • Stage 1: Deploy a new vCenter Server 6.7 appliance
  • Stage 2: Transfer the services and configuration data from the old to the new appliance

Upgrading the vCenter Server Appliance includes deploying a new appliance (version 6.7). The configuration and data is transferred from old (6.0 or 6.5)  appliance to the new vCenter Server 6.7 Appliance.  The old appliance is still available in a powered down state in the vCenter Server inventory after the upgrade.

vSphere 6.7 is the last release to include vCenter Server for Windows. After this release, vCenter Server for Windows will not be available! So make sure that all new deployments and upgrades are using the the vCenter Server Appliance (VCSA)!

New enhancements

Some cool enhancements of the vCenter Server 6.7 appliance are:

  • The vCenter Server with Embedded PSC supports Enhanced Linked Mode. This gives the following benefits:
    • No load balancer required for high availability and fully supports native vCenter Server High Availability.
    • SSO Site boundary removal provides flexibility of placement.
    • Supports vSphere scale maximums.
    • Allows for 15 deployments in a vSphere Single Sign-On Domain.
    • Reduces the number of nodes to manage and maintain.
  • vSphere 6.7 supports repointing a vCenter Server to another external Platform Services Controller in the same SSO site and different SSO site within the same SSO domain
  • vSphere 6.7 supports repointing a vCenter Server (Appliance only) to another external Platform Services Controller in a different SSO domain.
  • The vSphere Appliance Management Interface (VAMI) on port 5480 has some great new enhancements:
    • Upgraded Clarity interface
    • Dedicated monitor tab
    • Services tab. See the status of the VCSA services and the ability to: stop, start and restart services. So no CLI is needed for that anymore!
    • Backup scheduler. The backup scheduler let you schedule a backup of the VCSA and select how many backups are retained. The supported protocols for backup locations are: FTP, FTPS, HTTP, HTTPS and SCP.
  • The vSphere Client (HTML5) has updated and includes new workflows on Update Manager and vSAN for example.

Before upgrading

Before upgrading make sure to check this:

  • Check the compatibility of the VMware and third party products you are using. When writing this blog the following VMware products are not compatible (yet) with vSphere 6.7:
    • NSX
    • Horizon. Horizon 7.4 is not compatible with the Instant Clone API used in vSphere 6.7. Instant Clone support for vSphere 6.7 will be available in an upcoming Horizon release.
    • VMware Integrated OpenStack (VIO)
    • VMware vSphere Integrated Containers (VIC)
    • vCloud Director
  • For the upgrade order of multiple VMware products see the “Update sequence for vSphere 6.7 and its compatible VMware products (53710)” KB, link
  • It’s only possible to upgrade the vCenter Server Appliance version 6.0 or 6.5 to 6.7.
  • It’s not supported to upgrade from 6.5 U2 to 6.7! It will be provided in a future release! With vSphere 6.7 Update 1 (not available yet) it’s possible to upgrade from vSphere 6.5 U2 to vSphere 6.7 U1.
  • For vSphere 5.5 you must first upgrade to vSphere 6 or vSphere 6.5 before upgrading to vSphere 6.7
  • Make sure you have enough capacity in the cluster to add an extra vCenter Server Appliance (VCSA). The old appliance can be removed when the upgrade is successful. Here’s an overview of the hardware specifications needed.
  • In vSphere 6.7, only TLS 1.2 is enabled by default. vSphere 6.7 disables TLS 1.0 and TLS 1.1 protocols for improved security. Some applications might support only the older protocols. To revert TLS 1.0 and TLS 1.1 protocols use the TLS reconfigurator tool. The tool can be found in the appliance under: /usr/lib/vmware-TlsReconfigurator/VcTisReconfigurator.
  • Windows 2003 and XP are no longer supported.

Platform Services Controller (PSC) hardware sizing

Option Environment vCPU Memory (GB) Default Storage (GB)
Platform Services Controller 2 4 60

vCenter Server Appliance (VCSA) hardware sizing

Option Environment vCPU Memory (GB) Default Storage (GB)
Tiny Up to 10 hosts or 100 VMs 2 10 250
Small Up to 100 hosts or 1000 VMs 4 16 290
Medium Up to 400 hosts or 4000 VMs 8 24 425
Large Up to 1000 hosts or 10000 VMs 16 32 640
X-Large Up to 2000 hosts or 35000 VMs 24 48 980
  • Use a temporary fixed IP address
  • Make sure that you have the SSO administrator and root account information of the existing VCSA
  • Have a backup of the VCSA
  • Disable Fully Automated DRS during the upgrade

The upgrade steps

In the following steps a single vCenter Server  Appliance with an embedded PSC and vCenter Server role will be upgraded to version 6.7.

  • Mount the VCSA ISO (VMware-VCSA-all-6.7.0-8217866.iso)
  • Navigate to the <drive letter>:\vcsa-ui-installer\win32\ folder and open the installer.exe
  • Choose for the upgrade option. With the option you can upgrade a PSC and vCenter Server appliance.

  • 1. The upgrading process will enter “stage 1”, deploy the appliance.

  • 2. Accept the End user License Agreement.

  • 3. Connect to the source vCenter Server 6.x appliance and ESXi server.  Enter the SSO and root username of the VCSA and the ESXi server that manages the source appliance. Accept the certificate warning.

  • 4. Select the deployment target. I use the same ESXi host where the source VCSA is running. Accept the certificate warning.

  • 5. Set up the target appliance VM name and root password. The upgrade will maintain the original FQDN name of the VCSA. This name will be used as VM name in the VCSA inventory and can be changed later!

  • 6. Select the (new) size of the new appliance.

  • 7. Select the datastore

  • 8. Configure the network settings. Make sure to use an new temporarily IP address for the upgrade. After the upgrade the new appliance will use the original IP address!

  • 9. Click finish to start stage 1

 

 

 

 

 

  • After a while the following message appears and you’re ready to continue to stage 2.

  • 1. Introduction. Stage 2 will copy data from the source vCenter Server Appliance to the new deployed appliance.

  • 2. A pre-upgrade check will run, after the pre-upgrade check has finished warning messages will be shown such as:
    • Disable Fully Automated DRS during the upgrade
    • Files that cannot be used with Update Manager 6.7 will not be copied from the source.
    • An NSX extension has been found that may not work after the upgrade

  • 3. The data types that needs to migrated can be selected. A new cool thing is that the amount of time that’s involved is displayed for the Configuration data.

  • 4. Configure the VMware Customer Experience Improvement Program (CEIP)

  • 5. Ready to start fase 2 by selecting “I have backed up the source vCenter Server and all the required data from the database.

  • A shutdown warning is displayed, the source VCSA will be shut down.

  • The data transfer and appliance setup is running

  • A couple of messages will be displayed about for example Auto Deploy and that TLS 1.0 and TLS 1.1 are disabled in vSphere 6.7.

  • Stage 2 is completed and the vCenter Server Appliance is deployed.

  • Now you can access the vCenter Server by using vSphere Client (HTML5), the vSphere Web Client or VMware Appliance Management Interface by using the original FQDN of the vCenter Server Appliance.

After the upgrade the VCSA is upgraded to version 6.7.

Unable to login because of a ESXi root account lockout

When starting one of my VMware ESXi 6.5 lab hosts I was unable to login using the vSphere Host Client. I tried to make an SSH session to the host but got an “Access Denied” message.

When Using the Direct Console Interface (DCUI) I was able to login using the root account. In the log folder (under /var/log) I found that the root account is locked because of many failed attempt by investigate the following log files:

vobd.log

2018-01-02T10:57:00.003Z: [GenericCorrelator] 5612887277us: [vob.user.account.locked] Remote access for ESXi local user account 'root' has been locked for 900 seconds after 58 failed login attempts.
2018-01-02T10:57:00.003Z: [UserLevelCorrelator] 5612887277us: [vob.user.account.locked] Remote access for ESXi local user account 'root' has been locked for 900 seconds after 58 failed login attempts.
2018-01-02T10:57:00.003Z: [UserLevelCorrelator] 5612887502us: [esx.audit.account.locked] Remote access for ESXi local user account 'root' has been locked for 900 seconds after 58 failed login attempts.

auth.log

2018-01-02T11:02:08Z sshd[117700]: Connection from 192.168.249.23 port 63449
2018-01-02T11:02:09Z sshd[117701]: pam_tally2(sshd:auth): user root (0) tally 72, deny 5
2018-01-02T11:02:14Z sshd[117700]: error: PAM: Authentication failure for root from 192.168.249.23
2018-01-02T11:02:14Z sshd[117710]: pam_tally2(sshd:auth): user root (0) tally 73, deny 5

By default the ESXi 6.x password requirements for lockout behavior are:

  • A maximum of ten failed attempts is allowed before the account is locked
  • Password lockout is active on SSH and the vSphere Web Service SDK
  • Password lockout is not active on the Direct Console Interface (DCUI) and the ESXi Shell

To view the number of failed login attempt use the following command:


pam_tally2 --user root

In my example the there were 58 failed root login attempts:


Login Failures Latest failure From
root 58 01/02/18 10:56:59 unknown

The clear the the password lockout use the following command:


pam_tally2 --user root --reset

After this command I was able to login the vSphere Host Client. In the vSphere Host Client I found the VM that is causing the root account lockout:

The VM was monitoring the vSphere ESXi host with the wrong root password. After changing the password the account lockout problem was solved.