NVMe SSDs are not recognized anymore after upgrading to VMware ESXi 7

Last week VMware released vSphere 7. The first thing I did was upgrading my homelab. My homelab has two  hosts (Shuttle SH370R6 Plus and a Shuttle SH370R8 Plus). After upgrading my hosts from VMware ESXi 6.7 Update 3 to VMware ESXi 7, the NVMe SSDs are not recognized anymore. I Have the following NVMe SSD disks in the hosts:

  • Samsung SSD 970 EVO 1TB NVMe
  • Samsung SSD 950 PRO 512GB NVMe

My college and fellow vExpert Jesper Alberts encountered the same problem with his Supermicro X9DRL-iF and Samsung 970 EVO MZ-V7E1T0BW NVMe SSD homelab.

To fix this, perform a clean installation of VMware ESXi 7. After the clean installation, the NVMe disks are recognized.

Update May 16, 2020: Jeffrey Kusters experienced the same problem and was able to provide William Lam the support bundle of the host that failed the upgrade. William Lam and the VMware support team discovered quickly that we used the wrong esxcli update command (esxcli software vib update). The correct command to upgrade an ESXI host is: esxcli software profile update. William explains the commands in his blog found here, link.

So the steps when upgrading a standalone ESXi host are:

  • Stop the VMs on the ESXi host
  • Enter Maintenance Mode
  • Copy to VMware ESXI offline bundle to a datastore with enough space available. In this example I use the “nfs01” datastore
  • SSH to the host and perform the following command: esxli software sources profile list -d /vmfs/volumes/<datastore>/<ESXofflinebundle.zip>
[root@esxi01:~] esxcli software sources profile list -d /vmfs/volumes/nfs01/ISO/VMware-ESXi-7.0.0-15843807-depot.zip
Name                          Vendor        Acceptance Level  Creation Time        Modification Time
----------------------------  ------------  ----------------  -------------------  -----------------
ESXi-7.0.0-15843807-standard  VMware, Inc.  PartnerSupported  2020-03-16T10:48:54  2020-03-16T10:48:54
ESXi-7.0.0-15843807-no-tools  VMware, Inc.  PartnerSupported  2020-03-16T10:48:54  2020-03-16T10:48:54
  • This list the image profiles available from the offline bundle
  • Execute the command: esxcli software sources profile update -d /vmfs/volumes/<datastore>/<ESXiOfflineBundle.zip> -p <Image profile>
[root@esxi01:~] esxcli software sources profile update -d /vmfs/volumes/nfs01/ISO/VMware-ESXi-7.0.0-15843807-depot.zip -p ESXi-7.0.0-15843807-no-tools
  • When the upgrade is completed successfully, reboot the ESXi host
  • Log in and exit the maintenance mode on the ESXi host

Thanks, to  Jeffrey and William for solving this problem. Great community work!

7 thoughts on “NVMe SSDs are not recognized anymore after upgrading to VMware ESXi 7”

  1. I was able to vSphere & vSAN 7.0 from 6.7 using lifecycle manger. I have Samsung 250GB 960 EVO NVMe M.2 Internal SSD in each of my 3 SuperMicro E300 servers. vSAN 7.0 is reporting the vSAN cluster is healthy.

    http://www.bitofsnow.com

    [root@lab1-esxi:~] esxcli nvme device list
    HBA Name Status Signature
    ——– —— ———
    vmhba1 Online nvmeMgmt-nvmhba0
    [root@lab1-esxi:~] esxcli nvme device get -A vmhba1
    Controller Identify Info:
    PCIVID: 0x144d
    PCISSVID: 0x144d
    Serial Number: S3ESNX0K190151K
    Model Number: Samsung SSD 960 EVO 250GB

  2. I ran through the updates yesterday on a single host I’ve got at home:
    970 EVO NVM 1 TB for cache
    860 EVO 1 TB SSD for capacity

    I ran the upgrade from a flash drive, and didn’t note any issues with the drives disappearing

  3. hi,
    any idea if it’s working for Samsung 970 Evo Plus 1 TB NVMe with vSphere/vSAN 7?
    or any experience with XPG 8200 Pro 1 TB NVMe and vSphere/vSAN 7?

  4. Hi,

    I have installed Samsung 970 Evo Plus 2TB NVMe on my vSphere 7.
    I copied a file of 10Gb from NVMe to NVMe and it too 100 secs.
    I should be much quicker than 100 secs.
    Any idea how to improve the performance?

    Regards,
    Remo

  5. Great work, this helped me a lot!

    Just a quick note – your command for the actual update is incorrect.

    The correct command is like this – without the parameter “sources”.
    [root@esxi01:~] esxcli software profile update -d /vmfs/volumes/nfs01/ISO/VMware-ESXi-7.0.0-15843807-depot.zip -p ESXi-7.0.0-15843807-no-tools

  6. 1. Upload the following files to esxi “/tmp” directory
    VMware_bootbank_vmware-esx-esxcli-nvme-plugin_1.2.0.32-0.0.8169922.vib
    VMW_bootbank_nvme_1.2.1.34-1vmw.670.0.0.8169922.vib
    (Files can be download from http://221.126.249.9/nvme.7z)

    2. SSH run the following commands to install 2 files:

    esxcli software vib install -v /tmp/VMware_bootbank_vmware-esx-esxcli-nvme-plugin_1.2.0.32-0.0.8169922.vib
    esxcli software vib install -v /tmp/VMW_bootbank_nvme_1.2.1.34-1vmw.670.0.0.8169922.vib

    3. Reboot > Done !! Enjoy 🙂

    ————————————————————————–

    login as: root
    Keyboard-interactive authentication prompts from server:
    | Password:
    End of keyboard-interactive prompts from server
    The time and date of this login have been sent to the system logs.

    WARNING:
    All commands run on the ESXi shell are logged and may be included in
    support bundles. Do not provide passwords directly on the command line.
    Most tools can prompt for secrets or accept them from standard input.

    VMware offers supported, powerful system administration tools. Please
    see http://www.vmware.com/go/sysadmintools for details.

    The ESXi Shell can be disabled by an administrative user. See the
    vSphere Security documentation for more information.
    [root@localhost:~] esxcli software vib install -v /tmp/VMware_bootbank_vmware-es
    x-esxcli-nvme-plugin_1.2.0.32-0.0.8169922.vib
    Installation Result
    Message: Operation finished successfully.
    Reboot Required: false
    VIBs Installed: VMware_bootbank_vmware-esx-esxcli-nvme-plugin_1.2.0.32-0.0.81 69922
    VIBs Removed: VMware_bootbank_vmware-esx-esxcli-nvme-plugin_1.2.0.36-2.48.130 06603
    VIBs Skipped:
    [root@localhost:~] esxcli software vib install -v /tmp/VMW_bootbank_nvme_1.2.1.3
    4-1vmw.670.0.0.8169922.vib
    Installation Result
    Message: The update completed successfully, but the system needs to be rebooted for the changes to be effective.
    Reboot Required: true
    VIBs Installed: VMW_bootbank_nvme_1.2.1.34-1vmw.670.0.0.8169922
    VIBs Removed: VMW_bootbank_nvme_1.2.2.28-1vmw.670.3.73.14320388
    VIBs Skipped:
    [root@localhost:~]

Leave a Comment