homelab

NVMe SSDs are not recognized anymore after upgrading to VMware ESXi 7

Last week VMware released vSphere 7. The first thing I did was upgrading my homelab. My homelab has two  hosts (Shuttle SH370R6 Plus and a Shuttle SH370R8 Plus). After upgrading my hosts from VMware ESXi 6.7 Update 3 to VMware ESXi 7, the NVMe SSDs are not recognized anymore. I Have the following NVMe SSD disks in the hosts:

  • Samsung SSD 970 EVO 1TB NVMe
  • Samsung SSD 950 PRO 512GB NVMe

My college and fellow vExpert Jesper Alberts encountered the same problem with his Supermicro X9DRL-iF and Samsung 970 EVO MZ-V7E1T0BW NVMe SSD homelab.

To fix this, perform a clean installation of VMware ESXi 7. After the clean installation, the NVMe disks are recognized.

Update May 16, 2020: Jeffrey Kusters experienced the same problem and was able to provide William Lam the support bundle of the host that failed the upgrade. William Lam and the VMware support team discovered quickly that we used the wrong esxcli update command (esxcli software vib update). The correct command to upgrade an ESXI host is: esxcli software profile update. William explains the commands in his blog found here, link.

So the steps when upgrading a standalone ESXi host are:

  • Stop the VMs on the ESXi host
  • Enter Maintenance Mode
  • Copy to VMware ESXI offline bundle to a datastore with enough space available. In this example I use the “nfs01” datastore
  • SSH to the host and perform the following command: esxli software sources profile list -d /vmfs/volumes/<datastore>/<ESXofflinebundle.zip>
[root@esxi01:~] esxcli software sources profile list -d /vmfs/volumes/nfs01/ISO/VMware-ESXi-7.0.0-15843807-depot.zip
Name                          Vendor        Acceptance Level  Creation Time        Modification Time
----------------------------  ------------  ----------------  -------------------  -----------------
ESXi-7.0.0-15843807-standard  VMware, Inc.  PartnerSupported  2020-03-16T10:48:54  2020-03-16T10:48:54
ESXi-7.0.0-15843807-no-tools  VMware, Inc.  PartnerSupported  2020-03-16T10:48:54  2020-03-16T10:48:54
  • This list the image profiles available from the offline bundle
  • Execute the command: esxcli software sources profile update -d /vmfs/volumes/<datastore>/<ESXiOfflineBundle.zip> -p <Image profile>
[root@esxi01:~] esxcli software sources profile update -d /vmfs/volumes/nfs01/ISO/VMware-ESXi-7.0.0-15843807-depot.zip -p ESXi-7.0.0-15843807-no-tools
  • When the upgrade is completed successfully, reboot the ESXi host
  • Log in and exit the maintenance mode on the ESXi host

Thanks, to  Jeffrey and William for solving this problem. Great community work!

VMware homelab build of materials and configuration

William Lam has started a great initiative. William asked (link) everyone who owns a homelab to share there build of materials (BOM) and configuration so the vCommunity can benefit and learn from. I have a simple homelab configuration, the materials I used and configuration are listed below:

Internet

Cable modem in bridge mode with 250 Mbit/s down and 25 Mbit/s upload.

Router 

Ubiquiti EdgeRouter Lite 3-Port router

Access Point

Ubiquiti UniFi AP AC PRO

Layer 2 switches

2 x HP ProCurve 1810G (8 x 1GbE) manageable switches.

Compute

Shuttle SH370R6 Plus and a Shuttle SH370R8 Plus. Each barebone has:

  • 500 W Plus Silver PSU
  • Intel Core i7 8700 with 6 cores and 12 threads
  • 64 GB memory
  • Samsung 970 EVO 1 TB m.2
  • Kingston datatraveler 100 G3 32 GB USB disk
  • 2 x 1 GbE Network cards

Network Attached Storage (NAS)

QNAP TS-251+ NAS with two Western Digital (WD) Red 8 TB disk in a RAID-1 configuration.

Software

  • VMware vSphere (ESXi, vCenter)
  • VMware vSAN
  • VMware Horizon
  • VMware NSX-V and NSX-T
  • vRealize products
  • IoT stuff

Build of materials (BOM)

Components Costs ~  Links to blog posts
Ubiquiti EdgeRouter Lite 3-Poort Router € 93
Ubiquiti UniFi AP AC PRO € 136
HP ProCurve 1810G € 75 each. Not available anymore.
Shuttle SH370R6 Plus € 1200 Link
Shuttle SH370R8 € 1200 Link
QNAP TS-251+ € 314
2 x Western Digital (WD) Red 8 TB € 258 each. Total € 516

An overview of all the submitted community homelabs can be found here, link.

DHCP problems after Ubiquiti EdgeRouter firmware upgrade

In my homelab I use a Ubiquiti EdgeRouter Lite 3-port and UniFi AC Access Points for some time now. After upgrading the Ubiquiti EdgeRouter to the latest firmware (EdgeOS 1.10.1) my WIFI devices where unable to get an IP address. I have different VLANs defined on the EdgeRouter for the WIFI networks. Each VLAN has it’s own DHCP scope configured.

In the EdgeRouter GUI I didn’t find any clue why the WIFI devices didn’t get an IP address anymore, so I opened a SSH session to the EdgeRouter and start troubleshooting. First I tried to start the DHCP service by using this command.

 
sudo service dhcpd start

The following error is displayed:

[….] Cannot start the DHCP server because configuration file /opt/vyatta/etc/d [FAILconf is absent. … failed!

The DHCP service cannot be started, that’s the problem why the WIFI devices didn’t get an IP address anymore. Then i looked in the following log files:

  • cat /var/log/messages
  • cat /var/log/vyatta/vyatta-commit.log

In the vyatta-commit.log the following error is displayed under the [service dhcp-server] section:

[ service dhcp-server ]
Static DHCP lease IP ‘192.168.249.11’ under static mapping ‘Chromecast’
under shared network name ‘WIFI’ is already is in by static-mapping ”.
DHCP server configuration commit aborted due to error(s).

In the DHCP scope for the WIFI VLAN there was a static IP mapping called “Chromecast”. I removing the “Chromecast” static IP mapping in the GUI of the EdgeRouter. In the SSH session tried to start the DHCP service by using the following command.

 
sudo service dhcpd start

Starting DHCP server daemon…

The DHCP service is started.  In the vyatta-commit.log no new errors are displayed and the WIFI devices were able to get an IP address.  Removing the “Chromecast” static mapping cleared the duplicate static IP error.