You probably heard about the two massive security flaws: Spectre and Meltdown (link). These security flaws allow attackers to access “secure” data by compromising privileged processor memory from major manufacturers, including Intel, AMD, and ARM. So the most CPUs are affected by Spectre and Meltdown security flaws! In this blogpost I highlight what to do in VMware environments.
- Januari 14, 2018. All the patches associated with VMSA-2018-0004 have been pulled back from the online and offline portal. Intel has notified VMware of recent sightings that may affect some of the initial microcode patches that provide the speculative execution control mechanism (Intel Sightings) for a number of Intel Haswell and Broadwell processors. Link
- Januari 17, 2018. LoginVSI gives a free license to all companies who are in need of performance testing their VMware Horizon VDI environment regarding meltdown and spectre security patches. This special license will be valid until March 31, 2018, and offers unlimited users, unlimited locations, and includes all standard user workloads. More information can be found here Link.
- Januari 22, 2018. VMSA-2018-0002.3 updated. Updated security advisory after release of ESXi 5.5 patch ESXi550-201801301-BG that has mitigation against both CVE-2017-5753 and CVE-2017-5715 on 2018-01-22. This patch does NOT include the unstable microcode mentioned in KB52345.
- Februari 3, 2018. Added the VMware Performance Impact for CVE-2017-5753, CVE-2017-5715, CVE-2017-5754 KB. Link
- Februari 8, 2018. VMware Virtual Appliance updates address side-channel analysis due to speculative execution (VMSA-2018-0007) is added. Link
- Februari 15, 2018. VMSA-2018-0007 is updated. Link
- March 20, 2018. Updated KB VMSA-2018-0004.3. New patches available for vCenter Server, ESXi, Workstation and Fusion.
Currently these security flaws can be divided into the following categories:
|Exploit Name||Exploited Vulnerability||Exploit Name / CVE||Microcode update required on the host|
|Variant 1||Spectre||Bounds check bypass
|Variant 2||Spectre||Branch target injection
|Variant 3||Meltdown||Rogue data cache load
Operating System patches will protect against number variant 1 and 3. With variant 2 a CPU microcode update is required.
What components needs to patched from a hypervisor perspective?
With a type 1 hypervisor such as VMware ESXi or Hyper-V the following components needs to be patched:
- CPU microcode (BIOS/UEFI update)
- Server firmware
- Operating systems
- Virtual machines
- Virtual appliances
So what’s the first step to perform?
The first thing to start is to develop a patch strategy. Here’s an example of tasks to perform to develop the patch strategy:
- Identify all the hardware components in the datacenter(s) that. Besides the hosts where the hypervisor is running there are connections to networking and storage components. There are tools available (for VMware environments) to help with this such as:
- RVTools, Link
- Verify Hypervisor-Assisted Guest Mitigation (Spectre) patches using PowerCLI, Link
- Document your vSphere Environment script, Link
- Use the Microsoft PowerShell Module “SpeculationControl” to verify that protections are enabled. See the Microsoft section below for more information.
- PowerCLI can be used for example to identify the VM hardware version with a simple oneliner:
Get-VM | Select Name, PowerState, Version | Out-GridView
- Identify per vendor what patches are available and how these patches needs to be installed.
- Identify the hardware that can’t be patched anymore. Contact the hardware vendor for a possible solution and decide what to do.
- Make sure your virus/anti malware solution is compatible with the new patches. Contact the antivirus software vendor for compatibility information.
- What’s the impact after applying those patches? Test the patches first in a separate environment. Works everything after deploying? Is there a performance impact when installing these patches?
- Identify what systems needs to be first patched (for example shared and multi tenant environments).
- The security best practices is to install all the patches available per vendor. Communicate with the vendor so you know when patches will be released. The comming days/weeks a lot of vendors will release patches against Spectre and Meltdown.
Vendor patch information
Here’s an overview of some vendors and there current patches available.
The VMware Security Advisories webpage displays the latest remediation for security vulnerabilities . The following advisories are available when writing this blog:
- VMSA-2018-0002.2 (updated 2018-01-13), about Hypervisor-Specific remediation
- VMSA-2018-0004.4 about Hypervisor-Assisted Guest Remediation
To protect against hardware mitigation for branch target injection issue identified in CVE-2017-5715 (See VMware Security Advisory VMSA-2018-0004.3 and Hypervisor-Assisted Guest Mitigation for branch target injection (52085) ) use the following steps:
- Upgrade the vCenter Server to:
- 6.5 U1g
- 6.0 U3e
- 5.5 U3h
- Apply the VMware ESXi patches:
- For ESXi 6.5: ESXi650-201803401-BG* and ESXi650-201803402-BG**
- For ESXi 6: ESXi600-201803401-BG* and ESXi600-201803402-BG**
- For ESXi 5.5: ESXi550-201803401-BG* and ESXi550-201803402-BG**
- The ESXi patches listed above will also automatically apply these critical CPU microcode updates if the server’s BIOS/Firmware has not already applied. The ESXi patches apply the microcode updates listed in Table 1. found in VMware KB 52085.
For each Virtual Machine (VM), enable Hypervisor-Assisted Guest mitigation via the following steps:
- Power down the VM
- Create a snapshot of the VM in case of issues
- Power on the VM
- Apply all security patches for your guest OS
- Ensure that all VMs are using Hardware Version 9 (available in ESXi 5.1 and above) or higher. Hardware version 9 is the minimum requirement for Hypervisor-Assisted Guest Mitigation for branch target injection (CVE-2017-5715). For best performance, Virtual Hardware Version 11 or higher is recommended. Virtual Hardware Version 11 (available in ESXi 6.0 and above) enables PCID/INVPCID. These features may reduce the performance impact of CVE-2017-5754 mitigations on CPUs that support those features. ESXi 6.5 uses hardware version 13.
- Test the VM if everything works as excepted. If not roll back to the snapshot
- Remove the snapshot
vMotion and EVC information
vMotion and EVC Information
An ESXi host that is running a patched vSphere hypervisor with updated microcode will see new CPU features that were not previously available.
These new features will be exposed to all Virtual Hardware Version 9+ VMs that are powered-on by that host. Because these virtual machines now see additional CPU features, vMotion to an ESXi host lacking the microcode or hypervisor patches applied will be prevented.
The vCenter patches enable vMotion compatibility to be retained within an EVC cluster.
In order to maintain this compatibility the new features are hidden from guests within the cluster until all hosts in the cluster are properly updated. At that time, the cluster will automatically upgrade its capabilities to expose the new features. Unpatched ESXi hosts will no longer be admitted into the EVC cluster.
More information about the vMotion and EVC changes see the KB “Hypervisor-Assisted Guest Mitigation for branch target injection (52085)”.
- Power down and start the VM to see the new EVC capabilities!
- After installing all the patches check the Hyperivosr-Assisted Guest Mitigation with William Lam’s PowerCLI script (Link). It happen that EVC must be disabled and enabled before the guest VMs get the proper EVC instructions!
|VMware Security Advisories||Link|
|Sign up to the Security-Announce mailing list to receive new and updated VMware Security Advisories and click ‘subscribe to article’ on the right side of this page to be alerted when new information is added to this document.||Link|
|Updated Advisory VMSA-2018-004.3||Link|
VMware Performance Impact for CVE-2017-5753, CVE-2017-5715, CVE-2017-5754 (aka Spectre and Meltdown) (52337)
|VMware Response to Speculative Execution security issues, CVE-2017-5753, CVE-2017-5715, CVE-2017-5754 (aka Spectre and Meltdown) (52245)||Link|
|Updated: januari 11 2018
vCenter Server Appliance (and PSC) 6.5 / 6.0 Workaround for CVE-2017-5753, CVE-2017-5715, CVE-2017-5754 (aka Spectre and Meltdown) (52312)
Other vendor patch information
Here is an list of resources of vendors I frequently work with:
It looks like HPE G6 and G7 models will not been updated anymore!
|HPE, Hewlett Packard Enterprise Product Security Vulnerability Alerts||Link|
|Bulletin: (Revision) HPE ProLiant, Moonshot and Synergy Servers – Side Channel Analysis Method Allows Improper Information Disclosure in Microprocessors (CVE-2017-5715, CVE-2017-5753, CVE-2017-5754)||Link|
|Dell, Microprocessor Side-Channel Vulnerabilities (CVE-2017-5715, CVE-2017-5753, CVE-2017-5754): Impact on Dell EMC products (Dell Enterprise Servers, Storage and Networking)||Link|
|CPU Side-Channel Information Disclosure Vulnerabilities||Link|
|CPU hardware vulnerable to Meltdown and Spectre attacks||Link|
|NVIDIA, Security Bulletin: NVIDIA GPU Display Driver Security Updates for Speculative Side Channels||Link|
|Windows Client Guidance for IT Pros to protect against speculative execution side-channel vulnerabilities.
This article includes a PowerShell script to verify that protections are enabled.
|Understanding the performance impact of Spectre and Meltdown mitigations on Windows Systems including performance||Link|
|Alternative protections for Windows Server 2016 Hyper-V Hosts against the speculative execution side-channel vulnerabilities||Link|
|Protecting guest virtual machines from CVE-2017-5715 (branch target injection)||Link|
|Citrix Security Updates for CVE-2017-5715, CVE-2017-5753, CVE-2017-5754||Link|
|Synology-SA-18:01 Meltdown and Spectre Attacks||Link|
6 thoughts on “What to know about Spectre and Meltdown with VMware environments”
https://kb.vmware.com/s/article/52345 is ambiguous on
” it is recommended to create dynamic or static baseline excluding the bulletins ESXi650-201801401-BG, ESXi650-201801402-BG, ESXi600-201801401-BG, ESXi600-201801402-BG ,ESXi550-201801401-BG and continue with the remediation process.”
What does this mean?
Operating systems > Guest OS? Ok, aknowledged but what “Virtual machines” in this conext mean?
Seems HPE Gen 6 & 7 will have a fix at a later time after all. Info found in v10 of HPE support bulletin:
So I have an older DEV environment that is running HP DL380p G8’s. HP just released microcode for these servers (BIOS – P70 1/22/2018).
1). I have patched Vcenter to 5.5u3g.
2). I have patched 1 test host to ESXi 5.5u3h.
3). I have applied HP BIOS P70 released on 3/2/2018 with updated microcode.
4). I have applied MS patches and made registry modifications on a Windows 2012 R2 test VM.
When I run the Get-SpeculationControlSettings script it still shows Hardware Support as FALSE.
I have tried numerous reboots and power off/on to the VM and host. I have updated to VM hardware version 10 and updated tools to the latest. However, the script still shows Hardware Support as FALSE.
Any suggestions? …running out of ideas! TIA
Thanks for great article! But it seems like I am having a same problem as Justin described in a post above!
Justin, any solution? Anyone else?
Thanks in advance!