Home Lab

How to build an ESXi on ARM Pi cluster?

Shortly after VMworld 2020, VMware released (after years of announcing and demoing) the ESXi On ARM fling (*1). On Social media and the community ESXi on ARM is a very hot topic. The ESXi ARM fling makes it possible to run the VMware ESXi hypervisor on ARM platforms such as:

    • Avantek Workstation and server (Ampere eMAG)
    • Lenovo ThinkSystem HR330A and HR350A (Ampere eMAG)
    • SolidRun Honeycomb LX2
    • Raspberry Pi (rPi) 4b model (4GB and 8GB only).

Because it supports the Raspberry Pi 4b model is very interesting for the home labbers.

(*1) A fling shows an early stage of software to the VMware community. There is no official (only community) support available. The ESXi on ARM fling can be download for the following location: link.

Use cases

Some use cases for ESXi On ARM are:

  • vSAN Witness node, link
  • Automation environment for PowerCLI and Terraform and packer, link.
  • Security at the edge
  • Other home lab projects such as running Home Assistant (blog post is coming).

For my home lab environment, I wanted to build an ESXi ARM cluster for my IoT stuff (such as Home Assistant) with two Pi nodes attached to my existing QNAP NAS. With the two ESXi ARM nodes, a vCenter Server, and shared storage, cluster functions such as vMotion, High Availability, DRS, and even FT are available. How cool is that!

Every day there are new use cases created in the community. That’s one reason why ESXi on ARM is such a cool technology!

My Environment build

Here is a simple diagram of how my setup looks like:

 

build of materials (BOM)

In this blog article, I will mention the build of materials (BOM). The following components I use:

Number Component ~Cost €  Link (cheapest Pi shop in the Netherlands)
1 Raspberry Pi 4 Model B with 8GB memory. 87,50 (per Pi) Link
2 Raspberry Pi 15W USB-C Power Supply  9,95 (per Pi) Link
3 Argon One Pi 4 case 28,95 (per Pi) Link
4 Official Raspberry Pi USB keyboard 17,95 Link
5 Micro SD card, 32 GB 13,95 (per Pi) Link
6 Delock USB 3.2 16 GB flash drive  8,99 (per Pi) I reused the USB drives
7 Micro-HDMI to HDMI cable 1,5m.  7,95 Link

1. Raspberry Pi 4 Model B with 8GB memory.

This Pi model has the following specifications:

    • 1.5GHz quad-core ARM Cortex-A72 CPU
    • VideoCore VI graphics
    • 4kp60 HEVC decode
    • True Gigabit Ethernet
    • 2 × USB 3.0
    • 2 × USB 2.0 ports
    • 2 × micro-HDMI ports (1 × 4kp60 or 2 × 4kp30)
    • USB-C for input power, supporting 5V 3A operation
    • 8Gb LPDDR4 memory

2. Raspberry Pi 15W USB-C Power Supply.

The power Supply uses the USB-C for charging the Pi. Make sure to use a decent power supply such as this one.

3. Argon One Pi 4 case

This case (which looks like the Tesla Cybertruck) has an aluminum enclosure for passive cooling and a fan inside for active cooling. Proper cooling is very important for the Pi because when running VMware ESXi it can get hot. You can control the FAN by software or enable the always-on mode. In software mode when the CPU temp reaches 55 degrees, the fan will run at 10%, at 60 degrees it will run at 55%, and at 60 degrees it will run 100%. The driver does not work on VMware ESXi, it is designed for the Pi OS. Hopefully, there will be a VIB available in the future that makes software control of the fan possible.  For VMware ESXi, you need to enable the always-on mode by switching the jumper pen next to the fan.

The assembly of the Pi and case is very easy:

  • Next to the fan, you see two cooling blocks (grey ones), one for the CPU and the other for the RAM chip
  • Add some terminal paste to the cooling blocks
  • Plug the PCB board into the Pi and the case. With the PCB board, all the ports and buttons are accessed from the back!
  • Tighten the screws.

The GPIO pins are still available when removing the magnetic cap from the top of the case.

4. Official Raspberry Pi USB keyboard.

This is a 78-key QWERTY keyboard with a built-in 3 ports hub on the back. It has a small form factor.

5 & 6. Micro SD card and USB disk.

The SD card is for storing the UEFI firmware that is required to boot the VMware ESXi-ARM installer. I used 32 GB SD. The USB drive is for installing the VMware ESXi ARM ISO.

7. Micro-HDMI to HDMI cable 1,5m.

The following components I already have in my home lab environment and will be re-used:

  • Netgear switch
  • 2 x Delock USB 3.2 16 GB flash drives
  • 2 x UTP CAT 5e cables
  • QNAP NAS

After the assembly of the case, connect the USB drive, SD card, Power-Supply, Monitor, keyboard, UTP cable, and you’re ready to install the VMware ESXi for ARM fling.

In the next ESXi on ARM blog, I will highlight the ESXi on ARM installation process and how to install and configure Home Assistant.

Here are some great links to follow:

Thanks to the Raspberry Store for the quick delivery.

NVMe SSDs are not recognized anymore after upgrading to VMware ESXi 7

Last week VMware released vSphere 7. The first thing I did was upgrading my homelab. My homelab has two  hosts (Shuttle SH370R6 Plus and a Shuttle SH370R8 Plus). After upgrading my hosts from VMware ESXi 6.7 Update 3 to VMware ESXi 7, the NVMe SSDs are not recognized anymore. I Have the following NVMe SSD disks in the hosts:

  • Samsung SSD 970 EVO 1TB NVMe
  • Samsung SSD 950 PRO 512GB NVMe

My college and fellow vExpert Jesper Alberts encountered the same problem with his Supermicro X9DRL-iF and Samsung 970 EVO MZ-V7E1T0BW NVMe SSD homelab.

To fix this, perform a clean installation of VMware ESXi 7. After the clean installation, the NVMe disks are recognized.

Update May 16, 2020: Jeffrey Kusters experienced the same problem and was able to provide William Lam the support bundle of the host that failed the upgrade. William Lam and the VMware support team discovered quickly that we used the wrong esxcli update command (esxcli software vib update). The correct command to upgrade an ESXI host is: esxcli software profile update. William explains the commands in his blog found here, link.

So the steps when upgrading a standalone ESXi host are:

  • Stop the VMs on the ESXi host
  • Enter Maintenance Mode
  • Copy to VMware ESXI offline bundle to a datastore with enough space available. In this example I use the “nfs01” datastore
  • SSH to the host and perform the following command: esxli software sources profile list -d /vmfs/volumes/<datastore>/<ESXofflinebundle.zip>
[root@esxi01:~] esxcli software sources profile list -d /vmfs/volumes/nfs01/ISO/VMware-ESXi-7.0.0-15843807-depot.zip
Name                          Vendor        Acceptance Level  Creation Time        Modification Time
----------------------------  ------------  ----------------  -------------------  -----------------
ESXi-7.0.0-15843807-standard  VMware, Inc.  PartnerSupported  2020-03-16T10:48:54  2020-03-16T10:48:54
ESXi-7.0.0-15843807-no-tools  VMware, Inc.  PartnerSupported  2020-03-16T10:48:54  2020-03-16T10:48:54
  • This list the image profiles available from the offline bundle
  • Execute the command: esxcli software sources profile update -d /vmfs/volumes/<datastore>/<ESXiOfflineBundle.zip> -p <Image profile>
[root@esxi01:~] esxcli software sources profile update -d /vmfs/volumes/nfs01/ISO/VMware-ESXi-7.0.0-15843807-depot.zip -p ESXi-7.0.0-15843807-no-tools
  • When the upgrade is completed successfully, reboot the ESXi host
  • Log in and exit the maintenance mode on the ESXi host

Thanks, to  Jeffrey and William for solving this problem. Great community work!

VMware homelab build of materials and configuration

William Lam has started a great initiative. William asked (link) everyone who owns a homelab to share there build of materials (BOM) and configuration so the vCommunity can benefit and learn from. I have a simple homelab configuration, the materials I used and configuration are listed below:

Internet

Cable modem in bridge mode with 250 Mbit/s down and 25 Mbit/s upload.

Router 

Ubiquiti EdgeRouter Lite 3-Port router

Access Point

Ubiquiti UniFi AP AC PRO

Layer 2 switches

2 x HP ProCurve 1810G (8 x 1GbE) manageable switches.

Compute

Shuttle SH370R6 Plus and a Shuttle SH370R8 Plus. Each barebone has:

  • 500 W Plus Silver PSU
  • Intel Core i7 8700 with 6 cores and 12 threads
  • 64 GB memory
  • Samsung 970 EVO 1 TB m.2
  • Kingston datatraveler 100 G3 32 GB USB disk
  • 2 x 1 GbE Network cards

Network Attached Storage (NAS)

QNAP TS-251+ NAS with two Western Digital (WD) Red 8 TB disk in a RAID-1 configuration.

Software

  • VMware vSphere (ESXi, vCenter)
  • VMware vSAN
  • VMware Horizon
  • VMware NSX-V and NSX-T
  • vRealize products
  • IoT stuff

Build of materials (BOM)

Components Costs ~  Links to blog posts
Ubiquiti EdgeRouter Lite 3-Poort Router € 93
Ubiquiti UniFi AP AC PRO € 136
HP ProCurve 1810G € 75 each. Not available anymore.
Shuttle SH370R6 Plus € 1200 Link
Shuttle SH370R8 € 1200 Link
QNAP TS-251+ € 314
2 x Western Digital (WD) Red 8 TB € 258 each. Total € 516

An overview of all the submitted community homelabs can be found here, link.