Upgrading a vCenter Server Appliance (VCSA) to version 6.7

Last week VMware launched vSphere 6.7. In this blog post I show how easy it is to upgrade a vCenter Server 6.x appliance to a new vCenter Server 6.7 appliance using the graphical interface (GUI) upgrade. The GUI upgrade uses a two stage process:

  • Stage 1: Deploy a new vCenter Server 6.7 appliance
  • Stage 2: Transfer the services and configuration data from the old to the new appliance

Upgrading the vCenter Server Appliance includes deploying a new appliance (version 6.7). The configuration and data is transferred from old (6.0 or 6.5)  appliance to the new vCenter Server 6.7 Appliance.  The old appliance is still available in a powered down state in the vCenter Server inventory after the upgrade.

vSphere 6.7 is the last release to include vCenter Server for Windows. After this release, vCenter Server for Windows will not be available! So make sure that all new deployments and upgrades are using the the vCenter Server Appliance (VCSA)!

New enhancements

Some cool enhancements of the vCenter Server 6.7 appliance are:

  • The vCenter Server with Embedded PSC supports Enhanced Linked Mode. This gives the following benefits:
    • No load balancer required for high availability and fully supports native vCenter Server High Availability.
    • SSO Site boundary removal provides flexibility of placement.
    • Supports vSphere scale maximums.
    • Allows for 15 deployments in a vSphere Single Sign-On Domain.
    • Reduces the number of nodes to manage and maintain.
  • vSphere 6.7 supports repointing a vCenter Server to another external Platform Services Controller in the same SSO site and different SSO site within the same SSO domain
  • vSphere 6.7 supports repointing a vCenter Server (Appliance only) to another external Platform Services Controller in a different SSO domain.
  • The vSphere Appliance Management Interface (VAMI) on port 5480 has some great new enhancements:
    • Upgraded Clarity interface
    • Dedicated monitor tab
    • Services tab. See the status of the VCSA services and the ability to: stop, start and restart services. So no CLI is needed for that anymore!
    • Backup scheduler. The backup scheduler let you schedule a backup of the VCSA and select how many backups are retained. The supported protocols for backup locations are: FTP, FTPS, HTTP, HTTPS and SCP.
  • The vSphere Client (HTML5) has updated and includes new workflows on Update Manager and vSAN for example.

Before upgrading

Before upgrading make sure to check this:

  • Check the compatibility of the VMware and third party products you are using. When writing this blog the following VMware products are not compatible (yet) with vSphere 6.7:
    • NSX
    • Horizon. Horizon 7.4 is not compatible with the Instant Clone API used in vSphere 6.7. Instant Clone support for vSphere 6.7 will be available in an upcoming Horizon release.
    • VMware Integrated OpenStack (VIO)
    • VMware vSphere Integrated Containers (VIC)
    •  vCloud Director
  • For the upgrade order of multiple VMware products see the “Update sequence for vSphere 6.7 and its compatible VMware products (53710)” KB, link
  • It’s only possible to upgrade the vCenter Server Appliance version 6.0 or 6.5 to 6.7.
  • For vSphere 5.5 you must first upgrade to vSphere 6 or vSphere 6.5 before upgrading to vSphere 6.7
  • Make sure you have enough capacity in the cluster to add an extra vCenter Server Appliance (VCSA). The old appliance can be removed when the upgrade is successful. Here’s an overview of the hardware specifications needed.
  • In vSphere 6.7, only TLS 1.2 is enabled by default. vSphere 6.7 disables TLS 1.0 and TLS 1.1 protocols for improved security. Some applications might support only the older protocols. To revert TLS 1.0 and TLS 1.1 protocols use the TLS reconfigurator tool. The tool can be found in the appliance under: /usr/lib/vmware-TlsReconfigurator/VcTisReconfigurator.
  • Windows 2003 and XP are no longer supported.

Platform Services Controller (PSC) hardware sizing

OptionEnvironmentvCPUMemory (GB)Default Storage (GB)
Platform Services Controller2460

vCenter Server Appliance (VCSA) hardware sizing

OptionEnvironmentvCPUMemory (GB)Default Storage (GB)
TinyUp to 10 hosts or 100 VMs210250
SmallUp to 100 hosts or 1000 VMs416290
MediumUp to 400 hosts or 4000 VMs824425
LargeUp to 1000 hosts or 10000 VMs1632640
X-LargeUp to 2000 hosts or 35000 VMs2448980
  • Use a temporary fixed IP address
  • Make sure that you have the SSO administrator and root account information of the existing VCSA
  • Have a backup of the VCSA
  • Disable Fully Automated DRS during the upgrade

The upgrade steps

In the following steps a single vCenter Server  Appliance with an embedded PSC and vCenter Server role will be upgraded to version 6.7.

  • Mount the VCSA ISO (VMware-VCSA-all-6.7.0-8217866.iso)
  • Navigate to the <drive letter>:\vcsa-ui-installer\win32\ folder and open the installer.exe
  • Choose for the upgrade option. With the option you can upgrade a PSC and vCenter Server appliance

  • The upgrading process will enter “stage 1”, deploy the appliance

  • Connect to the source vCenter Server 6.x appliance

  • Enter the SSO and root username of the VCSA and the ESXi server that manages the source appliance

  • Set up the target appliance VM name and root password. The upgrade will maintain the original FQDN name of the VCSA. This name will be used as VM name in the VCSA inventory and can be changed later!

  • Select the (new) size of the new appliance.

  • Configure the network settings. Make sure to use an new temporarily IP address for the upgrade. After the upgrade the new appliance will use the original IP address!

  • Click on finish to start stage 1

 

 

 

 

 

  • After a while the following message appears and you’re ready to continue to stage 2

  • Stage 2 will copy data from the source vCenter Server Appliance to the new deployed appliance.
  • A pre-upgrade check will be run with informational messages will be shown such as:
    • Disable Fully Automated DRS during the upgrade
    • An NSX extension has been found that may not work after the upgrade

  • The data that needs to migrated can be selected. A new cool thing is that the amount of time that’s involved is displayed.

  • Configure the VMware Customer Experience Improvement Program (CEIP)

  • Ready to start and complete fase 2

  • The source VCSA will be shut down.

  • The data transfer and appliance setup is running

  • A couple of messages will be displayed about for example Auto Deploy and that TLS 1.0 and TLS 1.1 are disabled in vSphere 6.7.

  • Stage 2 is completed and the vCenter Server Appliance is deployed.

  • Now you can access the vCenter Server by using vSphere Client (HTML5), the vSphere Web Client or VMware Appliance Management Interface by using the original FQDN of the vCenter Server Appliance.

After the upgrade the VCSA 6.7 version is: 6.7.10000 build 8217866

DHCP problems after Ubiquiti EdgeRouter firmware upgrade

In my homelab I use a Ubiquiti EdgeRouter Lite 3-port and UniFi AC Access Points for some time now. After upgrading the Ubiquiti EdgeRouter to the latest firmware (EdgeOS 1.10.1) my WIFI devices where unable to get an IP address. I have different VLANs defined on the EdgeRouter for the WIFI networks. Each VLAN has it’s own DHCP scope configured.

In the EdgeRouter GUI I didn’t find any clue why the WIFI devices didn’t get an IP address anymore, so I opened a SSH session to the EdgeRouter and start troubleshooting. First I tried to start the DHCP service by using this command.

 
sudo service dhcpd start

The following error is displayed:

[….] Cannot start the DHCP server because configuration file /opt/vyatta/etc/d [FAILconf is absent. … failed!

The DHCP service cannot be started, that’s the problem why the WIFI devices didn’t get an IP address anymore. Then i looked in the following log files:

  • cat /var/log/messages
  • cat /var/log/vyatta/vyatta-commit.log

In the vyatta-commit.log the following error is displayed under the [service dhcp-server] section:

[ service dhcp-server ]
Static DHCP lease IP ‘192.168.249.11’ under static mapping ‘Chromecast’
under shared network name ‘WIFI’ is already is in by static-mapping ”.
DHCP server configuration commit aborted due to error(s).

In the DHCP scope for the WIFI VLAN there was a static IP mapping called “Chromecast”. I removing the “Chromecast” static IP mapping in the GUI of the EdgeRouter. In the SSH session tried to start the DHCP service by using the following command.

 
sudo service dhcpd start

Starting DHCP server daemon…

The DHCP service is started.  In the vyatta-commit.log no new errors are displayed and the WIFI devices were able to get an IP address.  Removing the “Chromecast” static mapping cleared the duplicate static IP error.

Top vCenter Server Appliance (VCSA) troubleshooting commands

During the configuration and troubleshooting of vCenter Server Appliances (VCSA) I maintain a list of commands that I frequently use. This list contains my top configuration and troubleshooting VCSA commands:

  • Enable access the Bash shell:
shell.set --enabled true
  • Permanently configure the default Shell to BASH for Root:
chsh -s /bin/bash root
  • Log location of the VCSA:
/var/log/vmware/vsphere-client/logs/
  • VCSA service management:
Check status
service-control --status --all
List services
service-control --list
Stop all services
service-control --stop --all
Start all services
service-control --start --all
  • Join the AD domain from PSC:
/opt/likewise/bin/domainjoin-cli join domain.nl aduser password

After the AD domain join reboot the appliance

  • Check the AD domain join status:
/opt/likewise/bin/domainjoin-cli query
  • Leave AD domain join:
/opt/likewise/bin/domainjoin-cli leave

After the AD domain leave reboot the appliance

  • Certificate Manager location:
/usr/lib/vmware-vmca/bin/certificate-manager
  • Test port connectivity from the VCSA:
curl -v telnet://target ip address:port
Example:
curl –v telnet://mypsc.domain.local:443
  • Identity which PSC the VCSA is pointing to:
/usr/lib/vmware-vmafd/bin/vmafd-cli get-ls-location --server-name localhost
  • Repoint the VCSA to another PSC:
cmsso-util repoint --repoint-psc "PSC01"
  • Check the PSC replication partner:
/usr/lib/vmware-vmdir/bin/vdcrepadmin -f showpartners -h localhost -u administrator -w password
  • Check the PSC replication status:
/usr/lib/vmware-vmdir/bin/vdcrepadmin -f showpartnerstatus -h localhost -u administrator -w password

Output:

Partner: psc02
Host available: Yes
Status available: Yes
My last change number: 4274
Partner has seen my change number: 4274
Partner is 0 changes behind.
  • VDC Admin tool test LDAP and force replication
/usr/lib/vmware-vmdir/bin/vdcadmintool