Firefox does not trusts vCenter signed CA certificates

For a vCenter Server environment I replaced the default SSL certificates with CA signed SSL certificates. The Platform Service Controller (PSC) is configured as VMCA subordinate CA. When opening the vSphere Web/HTML5 Client, Firefox displays the following warning: Your connection is not secure.

This is because Firefox does not trust root certificates in the Windows certificate store. Since Firefox 49 a new option is included which allows Firefox to trust root certificates. This option is not enabled by default.

The following steps illustrate how to configure Firefox to use the Windows certificate store:

  • Open Firefox
  • In the address bar type: about:config
  • Accept the warning
  • Navigate to Preference name: security.enterprise_roots.enabled 
  • Set the value to:  true

Firefox now trust the root certificates in the Windows certificate store.

 

Installing VMware ESXi fails on Cisco UCS Blades with FlexFlash SD cards

When implementing a new Cisco UCS environment I encountered the following error when trying to install VMware ESXi 6.5 on a Cisco UCS Blade server with a FlexFlash SD card:

Operation failed

This program has encountered an error:

partedUtil failed with message: “Error: Can’t have a a partition outside the disk!
Unable to read partition table for device /vmfs/devices/disks……………….

The solutions is simple, perform a format of the SD card in UCSM before installing VMware ESXi. The format option can be found under:

  • Equipment -> Servers -> Select the server -> Inventory -> Storage – Controller -> Select the FlexFlash controller -> Format SD Cards

 

What to know about Spectre and Meltdown with VMware environments

You probably heard about the two massive security flaws: Spectre and Meltdown (link). These security flaws allow attackers to access “secure” data by compromising privileged processor memory from major manufacturers, including Intel, AMD, and ARM. So the most CPUs are affected by Spectre and Meltdown security flaws! In this blogpost I highlight what to do in VMware environments.

 

Last Updated:

  • Januari 14, 2018. All the patches associated with VMSA-2018-0004 have been pulled back from the online and offline portal. Intel has notified VMware of recent sightings that may affect some of the initial microcode patches that provide the speculative execution control mechanism (Intel Sightings) for a number of Intel Haswell and Broadwell processors. Link
  • Januari 17, 2018. LoginVSI gives a free license to all companies who are in need of performance testing their VMware Horizon VDI environment regarding meltdown and spectre security patches. This special license will be valid until March 31, 2018, and offers unlimited users, unlimited locations, and includes all standard user workloads. More information can be found here Link.
  • Januari 22, 2018. VMSA-2018-0002.3 updated. Updated security advisory after release of ESXi 5.5 patch ESXi550-201801301-BG that has mitigation against both CVE-2017-5753 and CVE-2017-5715 on 2018-01-22. This patch does NOT include the unstable microcode mentioned in KB52345.
  • Februari 3, 2018. Added the VMware Performance Impact for CVE-2017-5753, CVE-2017-5715, CVE-2017-5754 KB. Link
  • Februari 8, 2018. VMware Virtual Appliance updates address side-channel analysis due to speculative execution (VMSA-2018-0007) is added. Link

o.. We just decided to give away @LoginVSI for free to all companies who are in need of performance testing their @vmwarehorizon VDI environment regarding #meltdown and

Currently these security flaws can be divided into the following categories:

Exploit NameExploited VulnerabilityExploit Name / CVEMicrocode update required on the host
Variant 1SpectreBounds check bypass

CVE-2017-5753

No
Variant 2SpectreBranch target injection

CVE-2017-5715

Yes
Variant 3MeltdownRogue data cache load

CVE-2017-5754

No

Operating System patches will protect against number variant 1 and 3.  With variant 2 a CPU microcode update is required.

What components needs to patched from a hypervisor perspective?

With a type 1 hypervisor such as VMware ESXi or Hyper-V the following components needs to be patched:

  • CPU microcode (BIOS/UEFI update)
  • Server firmware
  • Hypervisors
  • Operating systems
  • Virtual machines
  • Virtual appliances

So what’s the first step to perform?

The first thing to start is to develop a patch strategy. Here’s an example of  tasks to perform to develop the patch strategy:

  • Identify all the hardware components in the datacenter(s) that. Besides the hosts where the hypervisor is running there are connections to networking and storage components. There are tools available (for VMware environments) to help with this such as:
    • RVTools, Link
    • Verify Hypervisor-Assisted Guest Mitigation (Spectre) patches using PowerCLI, Link
    • Document your vSphere Environment script, Link
    • Use the Microsoft PowerShell Module “SpeculationControl” to verify that protections are enabled. See the Microsoft section below for more information.
    • PowerCLI can be used for example to identify the VM hardware version with a simple oneliner:
 Get-VM | Select Name, PowerState, Version | Out-GridView 
  • Identify per vendor what patches are available and how these patches needs to be installed.
  • Identify the hardware that can’t be patched anymore. Contact the hardware vendor for a possible solution and decide what to do.
  • Make sure your virus/anti malware solution is compatible with the new patches. Contact the antivirus software vendor for compatibility information.
  • What’s the impact after applying those patches? Test the patches first in a separate environment. Works everything after deploying? Is there a performance impact when installing these patches?
  • Identify what systems needs to be first patched (for example shared and multi tenant environments).
  • The security best practices is to install all the patches available per vendor. Communicate with the vendor so you know when patches will be released. The comming days/weeks a lot of vendors will release patches against Spectre and Meltdown.

Vendor patch information

Here’s an overview of some vendors and there current patches available.

VMware

The VMware Security Advisories webpage displays the latest remediation for security vulnerabilities . The following advisories are available when writing this blog:

  • VMSA-2018-0002.2 (updated 2018-01-13), about Hypervisor-Specific remediation
  • VMSA-2018-0004.2 about Hypervisor-Assisted Guest Remediation

To protect against hardware mitigation for branch target injection issue identified in CVE-2017-5715 (See VMware Security Advisory VMSA-2018-0004 and Hypervisor-Assisted Guest Mitigation for branch target injection (52085) ) use the following steps:

  1. Upgrade the vCenter Server to:
    1. 6.5 U1e (Build Number 7515524)
    2. 6.0 U3d (Build Number 7464194)
    3. 5.5 U3g (Build Number 7460842)
  2. Apply the VMware ESXi patches:
    1. ESXi650-201801401-BG hypervisor *
    2. ESXi600-201801401-BG hypervisor *
    3. ESXi550-201801401-BG hypervisor and microcode **
  3. Apply the Microcode/BIOS updates for CVE-2017-5715 in one of two ways:
    1. Apply the BIOS/Microcode update from your platform vendor.
      OR
    2. Apply one of the following ESXi patches to update the microcode for supported CPUs
      1. ESXi650-201801402-BG microcode *
      2. ESXi600-201801402-BG microcode *
      3. ESXi550-201801401-BG hypervisor and microcode **
 * ESXi 6.5 and ESXi 6.0 use separate patches for hypervisor and microcode.
** ESXi 5.5 uses a single patch for both hypervisor and microcode.

For each Virtual Machine (VM), enable Hypervisor-Assisted Guest mitigation via the following steps:

  1. Power down the VM
  2. Create a snapshot of the VM in case of issues
  3. Power on the VM
  4. Apply all security patches for your guest OS
  5. Ensure that all VMs are using Hardware Version 9 (available in ESXi 5.1 and above) or higher. Hardware version 9 is the minimum requirement for Hypervisor-Assisted Guest Mitigation for branch target injection (CVE-2017-5715). For best performance, Virtual Hardware Version 11 or higher is recommended. Virtual Hardware Version 11 (available in ESXi 6.0 and above) enables PCID/INVPCID. These features may reduce the performance impact of CVE-2017-5754 mitigations on CPUs that support those features. ESXi 6.5 uses hardware version 13.
  6. Test the VM if everything works as excepted. If not roll back to the snapshot
  7. Remove the snapshot

More information about the vMotion and EVC changes see the KB “Hypervisor-Assisted Guest Mitigation for branch target injection (52085)”.

  • Power down and start the VM to see the new EVC capabilities!
  • After installing all the patches check the Hyperivosr-Assisted Guest Mitigation with  William Lam’s PowerCLI script (Link). It happen  that EVC must be disabled and enabled before the guest VMs get the proper EVC instructions!

More information:

VMware Security AdvisoriesLink
Sign up to the Security-Announce mailing list to receive new and updated VMware Security Advisories and click ‘subscribe to article’ on the right side of this page to be alerted when new information is added to this document.Link
Hypervisor-Assisted Guest Mitigation for branch target injection (52085)
Link
VMware Performance Impact for CVE-2017-5753, CVE-2017-5715, CVE-2017-5754 (aka Spectre and Meltdown) (52337)
Link
VMware Response to Speculative Execution security issues, CVE-2017-5753, CVE-2017-5715, CVE-2017-5754 (aka Spectre and Meltdown) (52245)Link
Updated: januari 11 2018

vCenter Server Appliance (and PSC) 6.5 / 6.0 Workaround for CVE-2017-5753, CVE-2017-5715, CVE-2017-5754 (aka Spectre and Meltdown) (52312)

Link

Other vendor patch information

Here is an list of resources of vendors I frequently work with:

HPE

It looks like HPE G6 and G7 models will not been updated anymore!

HPE, Hewlett Packard Enterprise Product Security Vulnerability AlertsLink
Bulletin: (Revision) HPE ProLiant, Moonshot and Synergy Servers – Side Channel Analysis Method Allows Improper Information Disclosure in Microprocessors (CVE-2017-5715, CVE-2017-5753, CVE-2017-5754)Link

Dell

Dell, Microprocessor Side-Channel Vulnerabilities (CVE-2017-5715, CVE-2017-5753, CVE-2017-5754): Impact on Dell EMC products (Dell Enterprise Servers, Storage and Networking)Link

Cisco

CPU Side-Channel Information Disclosure VulnerabilitiesLink

Fortinet

CPU hardware vulnerable to Meltdown and Spectre attacksLink

NVIDIA

NVIDIA, Security Bulletin: NVIDIA GPU Display Driver Security Updates for Speculative Side ChannelsLink

Ubuntu

CVE-2017-5754Link

Microsoft

Windows Client Guidance for IT Pros to protect against speculative execution side-channel vulnerabilities.

This article includes a PowerShell script to verify that protections are enabled.

Link
Understanding the performance impact of Spectre and Meltdown mitigations on Windows Systems including performanceLink
Alternative protections for Windows Server 2016 Hyper-V Hosts against the speculative execution side-channel vulnerabilitiesLink
Protecting guest virtual machines from CVE-2017-5715 (branch target injection)Link

Citrix

Citrix Security Updates for CVE-2017-5715, CVE-2017-5753, CVE-2017-5754Link

Synology

Synology-SA-18:01 Meltdown and Spectre AttacksLink