What to know about Spectre and Meltdown with VMware environments

You probably heard about the two massive security flaws: Spectre and Meltdown (link). These security flaws allow attackers to access “secure” data by compromising privileged processor memory from major manufacturers, including Intel, AMD, and ARM. So the most CPUs are affected by Spectre and Meltdown security flaws! In this blogpost I highlight what to do in VMware environments.

 

Last Updated:

  • Januari 14, 2018. All the patches associated with VMSA-2018-0004 have been pulled back from the online and offline portal. Intel has notified VMware of recent sightings that may affect some of the initial microcode patches that provide the speculative execution control mechanism (Intel Sightings) for a number of Intel Haswell and Broadwell processors. Link
  • Januari 17, 2018. LoginVSI gives a free license to all companies who are in need of performance testing their VMware Horizon VDI environment regarding meltdown and spectre security patches. This special license will be valid until March 31, 2018, and offers unlimited users, unlimited locations, and includes all standard user workloads. More information can be found here Link.
  • Januari 22, 2018. VMSA-2018-0002.3 updated. Updated security advisory after release of ESXi 5.5 patch ESXi550-201801301-BG that has mitigation against both CVE-2017-5753 and CVE-2017-5715 on 2018-01-22. This patch does NOT include the unstable microcode mentioned in KB52345.
  • Februari 3, 2018. Added the VMware Performance Impact for CVE-2017-5753, CVE-2017-5715, CVE-2017-5754 KB. Link
  • Februari 8, 2018. VMware Virtual Appliance updates address side-channel analysis due to speculative execution (VMSA-2018-0007) is added. Link

o.. We just decided to give away @LoginVSI for free to all companies who are in need of performance testing their @vmwarehorizon VDI environment regarding #meltdown and

Currently these security flaws can be divided into the following categories:

Exploit NameExploited VulnerabilityExploit Name / CVEMicrocode update required on the host
Variant 1SpectreBounds check bypass

CVE-2017-5753

No
Variant 2SpectreBranch target injection

CVE-2017-5715

Yes
Variant 3MeltdownRogue data cache load

CVE-2017-5754

No

Operating System patches will protect against number variant 1 and 3.  With variant 2 a CPU microcode update is required.

What components needs to patched from a hypervisor perspective?

With a type 1 hypervisor such as VMware ESXi or Hyper-V the following components needs to be patched:

  • CPU microcode (BIOS/UEFI update)
  • Server firmware
  • Hypervisors
  • Operating systems
  • Virtual machines
  • Virtual appliances

So what’s the first step to perform?

The first thing to start is to develop a patch strategy. Here’s an example of  tasks to perform to develop the patch strategy:

  • Identify all the hardware components in the datacenter(s) that. Besides the hosts where the hypervisor is running there are connections to networking and storage components. There are tools available (for VMware environments) to help with this such as:
    • RVTools, Link
    • Verify Hypervisor-Assisted Guest Mitigation (Spectre) patches using PowerCLI, Link
    • Document your vSphere Environment script, Link
    • Use the Microsoft PowerShell Module “SpeculationControl” to verify that protections are enabled. See the Microsoft section below for more information.
    • PowerCLI can be used for example to identify the VM hardware version with a simple oneliner:
 Get-VM | Select Name, PowerState, Version | Out-GridView 
  • Identify per vendor what patches are available and how these patches needs to be installed.
  • Identify the hardware that can’t be patched anymore. Contact the hardware vendor for a possible solution and decide what to do.
  • Make sure your virus/anti malware solution is compatible with the new patches. Contact the antivirus software vendor for compatibility information.
  • What’s the impact after applying those patches? Test the patches first in a separate environment. Works everything after deploying? Is there a performance impact when installing these patches?
  • Identify what systems needs to be first patched (for example shared and multi tenant environments).
  • The security best practices is to install all the patches available per vendor. Communicate with the vendor so you know when patches will be released. The comming days/weeks a lot of vendors will release patches against Spectre and Meltdown.

Vendor patch information

Here’s an overview of some vendors and there current patches available.

VMware

The VMware Security Advisories webpage displays the latest remediation for security vulnerabilities . The following advisories are available when writing this blog:

  • VMSA-2018-0002.2 (updated 2018-01-13), about Hypervisor-Specific remediation
  • VMSA-2018-0004.2 about Hypervisor-Assisted Guest Remediation

To protect against hardware mitigation for branch target injection issue identified in CVE-2017-5715 (See VMware Security Advisory VMSA-2018-0004 and Hypervisor-Assisted Guest Mitigation for branch target injection (52085) ) use the following steps:

  1. Upgrade the vCenter Server to:
    1. 6.5 U1e (Build Number 7515524)
    2. 6.0 U3d (Build Number 7464194)
    3. 5.5 U3g (Build Number 7460842)
  2. Apply the VMware ESXi patches:
    1. ESXi650-201801401-BG hypervisor *
    2. ESXi600-201801401-BG hypervisor *
    3. ESXi550-201801401-BG hypervisor and microcode **
  3. Apply the Microcode/BIOS updates for CVE-2017-5715 in one of two ways:
    1. Apply the BIOS/Microcode update from your platform vendor.
      OR
    2. Apply one of the following ESXi patches to update the microcode for supported CPUs
      1. ESXi650-201801402-BG microcode *
      2. ESXi600-201801402-BG microcode *
      3. ESXi550-201801401-BG hypervisor and microcode **
 * ESXi 6.5 and ESXi 6.0 use separate patches for hypervisor and microcode.
** ESXi 5.5 uses a single patch for both hypervisor and microcode.

For each Virtual Machine (VM), enable Hypervisor-Assisted Guest mitigation via the following steps:

  1. Power down the VM
  2. Create a snapshot of the VM in case of issues
  3. Power on the VM
  4. Apply all security patches for your guest OS
  5. Ensure that all VMs are using Hardware Version 9 (available in ESXi 5.1 and above) or higher. Hardware version 9 is the minimum requirement for Hypervisor-Assisted Guest Mitigation for branch target injection (CVE-2017-5715). For best performance, Virtual Hardware Version 11 or higher is recommended. Virtual Hardware Version 11 (available in ESXi 6.0 and above) enables PCID/INVPCID. These features may reduce the performance impact of CVE-2017-5754 mitigations on CPUs that support those features. ESXi 6.5 uses hardware version 13.
  6. Test the VM if everything works as excepted. If not roll back to the snapshot
  7. Remove the snapshot

More information about the vMotion and EVC changes see the KB “Hypervisor-Assisted Guest Mitigation for branch target injection (52085)”.

  • Power down and start the VM to see the new EVC capabilities!
  • After installing all the patches check the Hyperivosr-Assisted Guest Mitigation with  William Lam’s PowerCLI script (Link). It happen  that EVC must be disabled and enabled before the guest VMs get the proper EVC instructions!

More information:

VMware Security AdvisoriesLink
Sign up to the Security-Announce mailing list to receive new and updated VMware Security Advisories and click ‘subscribe to article’ on the right side of this page to be alerted when new information is added to this document.Link
Hypervisor-Assisted Guest Mitigation for branch target injection (52085)
Link
VMware Performance Impact for CVE-2017-5753, CVE-2017-5715, CVE-2017-5754 (aka Spectre and Meltdown) (52337)
Link
VMware Response to Speculative Execution security issues, CVE-2017-5753, CVE-2017-5715, CVE-2017-5754 (aka Spectre and Meltdown) (52245)Link
Updated: januari 11 2018

vCenter Server Appliance (and PSC) 6.5 / 6.0 Workaround for CVE-2017-5753, CVE-2017-5715, CVE-2017-5754 (aka Spectre and Meltdown) (52312)

Link

Other vendor patch information

Here is an list of resources of vendors I frequently work with:

HPE

It looks like HPE G6 and G7 models will not been updated anymore!

HPE, Hewlett Packard Enterprise Product Security Vulnerability AlertsLink
Bulletin: (Revision) HPE ProLiant, Moonshot and Synergy Servers – Side Channel Analysis Method Allows Improper Information Disclosure in Microprocessors (CVE-2017-5715, CVE-2017-5753, CVE-2017-5754)Link

Dell

Dell, Microprocessor Side-Channel Vulnerabilities (CVE-2017-5715, CVE-2017-5753, CVE-2017-5754): Impact on Dell EMC products (Dell Enterprise Servers, Storage and Networking)Link

Cisco

CPU Side-Channel Information Disclosure VulnerabilitiesLink

Fortinet

CPU hardware vulnerable to Meltdown and Spectre attacksLink

NVIDIA

NVIDIA, Security Bulletin: NVIDIA GPU Display Driver Security Updates for Speculative Side ChannelsLink

Ubuntu

CVE-2017-5754Link

Microsoft

Windows Client Guidance for IT Pros to protect against speculative execution side-channel vulnerabilities.

This article includes a PowerShell script to verify that protections are enabled.

Link
Understanding the performance impact of Spectre and Meltdown mitigations on Windows Systems including performanceLink
Alternative protections for Windows Server 2016 Hyper-V Hosts against the speculative execution side-channel vulnerabilitiesLink
Protecting guest virtual machines from CVE-2017-5715 (branch target injection)Link

Citrix

Citrix Security Updates for CVE-2017-5715, CVE-2017-5753, CVE-2017-5754Link

Synology

Synology-SA-18:01 Meltdown and Spectre AttacksLink

Unable to login because of a ESXi root account lockout

When starting one of my VMware ESXi 6.5 lab hosts I was unable to login using the vSphere Host Client. I tried to make an SSH session to the host but got an “Access Denied” message.

When Using the Direct Console Interface (DCUI) I was able to login using the root account. In the log folder (under /var/log) I found that the root account is locked because of many failed attempt by investigate the following log files:

vobd.log

2018-01-02T10:57:00.003Z: [GenericCorrelator] 5612887277us: [vob.user.account.locked] Remote access for ESXi local user account 'root' has been locked for 900 seconds after 58 failed login attempts.
2018-01-02T10:57:00.003Z: [UserLevelCorrelator] 5612887277us: [vob.user.account.locked] Remote access for ESXi local user account 'root' has been locked for 900 seconds after 58 failed login attempts.
2018-01-02T10:57:00.003Z: [UserLevelCorrelator] 5612887502us: [esx.audit.account.locked] Remote access for ESXi local user account 'root' has been locked for 900 seconds after 58 failed login attempts.

auth.log

2018-01-02T11:02:08Z sshd[117700]: Connection from 192.168.249.23 port 63449
2018-01-02T11:02:09Z sshd[117701]: pam_tally2(sshd:auth): user root (0) tally 72, deny 5
2018-01-02T11:02:14Z sshd[117700]: error: PAM: Authentication failure for root from 192.168.249.23
2018-01-02T11:02:14Z sshd[117710]: pam_tally2(sshd:auth): user root (0) tally 73, deny 5

By default the ESXi 6.x password requirements for lockout behavior are:

  • A maximum of ten failed attempts is allowed before the account is locked
  • Password lockout is active on SSH and the vSphere Web Service SDK
  • Password lockout is not active on the Direct Console Interface (DCUI) and the ESXi Shell

To view the number of failed login attempt use the following command:


pam_tally2 --user root

In my example the there were 58 failed root login attempts:


Login Failures Latest failure From
root 58 01/02/18 10:56:59 unknown

The clear the the password lockout use the following command:


pam_tally2 --user root --reset

After this command I was able to login the vSphere Host Client. In the vSphere Host Client I found the VM that is causing the root account lockout:

The VM was monitoring the vSphere ESXi host with the wrong root password. After changing the password the account lockout problem was solved.

Deploy an OVA/OVF fails with certificate error

When trying to deploy an OVA/OVF with the vSphere Web Client the following error is displayed:

The operation failed for an undetermined reason. Typically this problem occurs due to certificates that the browser does not trust. If you are using self-signed or custom certificates, open the URL below in a new browser tab and accept the certificate, then retry the operation.

This error occurs with vSphere 6.5 because the certificates are not trusted. The self-signed certificates are used and are not added to the trusted root certification store.

To deploy a OVF/OVA to the vCenter Server appliance trusted root CA must be added to the certificate store. The following steps will work with Chrome and Internet Explorer:

  • Open the vCenter URL: https://vcenter-FQDN

  • Select the “Download trusted root CA certificates” and save the archive(ZIP) file
  • Extract the archive (ZIP)

  • Start – Run – MMC
  • File – Add Snap-ins – Certificates – Computer Account – Local  computer
  • Open Trusted Root Certification Authories – Certificates
  • Import the two *.crt certificates

  • Close the browser and re-open the browser and navigate to the vCenter Server using the FQDN.
  • Now the URL is marked as secure (green lock) and you’re able to import the OVA/OVF