Objective 4 – Upgrade a vSphere Deployment to 6.x

Objective 4.1: Perform ESXi Host and Virtual Machine Upgrades

  • vSphere 6.0 upgrade 
    • for vanilla sso->VC->VUM->esxi->vmware tools->
  • vSphere 6.0 upgrade best practices
  • VUM – know it
    • 3 main components
    • server runs on windows server
      • 64 bit windows server
      • client runs in web client or fat client to actually manage VUM
    • database
      • requires sql server or oracle
  • Installing VUM
    • prepare database
    • install VUM server
    • Install the update manager client plugin in the vSphere client
    • enable the vSphere update manager plugin for the vSphere web client
  • VUM basic order of operations
    • configure downloads, get downloads
    • create baselines or baseline groups
      • baselines are groups of 1 or more patches, extensions or upgrades
      • baseline groups consist of one upgrade baseline
    • attach baselines
    • scan
    • re-mediate
    • re-mediating multiple objects at once proceeds sequentially by default
  • new VUM web client
    • attach baselines
    • scan against baselines
    • VUM 6 is only compatible with vCenter server 6

 

Identify upgrade requirements for ESXi hosts

  • ESXi Requirements in the vSphere Upgrade Guide on page 37.
  • The following minimum requirements must be met to upgrade or install ESXi 6.0:
    • Supported Server Platform – http://www.vmware.com/go/hcl
    • At least 2 CPU cores required.
    • 64-bit x86 processor released after September 2016.
    • NX/XD bit enabled in the BIOS
    • Minimum of 4 GB of physical RAM, at least 8 GB of physical RAM recommended.
    • SCSI disk or a local, non-network, RAID LUN with unpartitioned space for the virtual machines.
    • At least one Gigabit or faster Ethernet controller.
    • Minimum 1 GB boot device.
  • vSphere 6.0 supports booting ESXi hosts from the Unified Extensible Firmware Interface (UEFI).
  • VMware Auto Deploy requires the legacy BIOS.
  • Changing between legacy BIOS and UEFI is not supported after installing ESXi 6.0.
  • For environments that boot from a SAN or use Auto Deploy, you need not allocate a separate LUN for each ESXi host.
  • Configure download source(s)
    • VUM admin view -> configuration -> download sources
      • add URL’s to the list
  • Stage patches & Extensions
    • after scanning, click stage…
  • Remediate an object
    • For hosts being upgraded that have been provisioned with a custom certificates, those certificates stay in place on upgrade. For hosts that have been provisioned with thumbprint  certs, they are automatically assigned certs from the VMCA at upgrade.
    • ESXi hardware requirements
    • 2 cores minimum, only 64bit
    • requires NX/XD bit to be enabled in bios
    • 4GB ram minimum to install, 8GB to run vm’s
    • To support 64-bit virtual machines, support for hardware virtualization (Intel VT-x or AMD RVI) must be enabled on x64 CPUs.
    • 1 or more 1GB controller
    • can boot from UEFI (HD, CD, usb, SD)
      • when using USB or SD, /scratch is left in ramdisk and should be migrated to persistent storage
    • Be sure that you have attach baseline privilege
  • Upgrade a vSphere Distributed Switch
    • go the the dvSwitch in networking and click actions->upgrade. choose version
  • Upgrade VMware Tools
    • manual
    • set remediation options
    • VUM
  • Upgrade Virtual Machine hardware
    • via VUM or web client
    • can be done in conjunction with a VM host upgrade if using VUM orchestrated upgrade
  • Stage multiple ESXi Host upgrades
    • turn off FT before performing upgrades on a cluster. primary and secondary VM’s  cannot reside on hosts of different ESXi version/patch levels
    • single baseline or baseline group
    • after upgrading hosts, they are in evaluation mode. You must manually assign to the appropriate license prior to eval expiration.
      • eval=60days
  • Align appropriate baselines with target inventory objects.
    • uhh…. know what you’re doing?

Objective 4.2: Perform vCenter Server Upgrades

Knowledge

  • Compare the methods of upgrading vCenter Server
    • vCenter 5, there are no common services, can upgrade to either embedded or external deployment
      • if you will be moving to an external PSC, you must install the PSC prior to upgrading vCenter, otherwise install will use embedded
        • if upgrading an external SSO to external PSC, custom certificates are maintained as machine certs and the VMCA assigns a cert to each solution user
      • during upgrade of simple->embedded custom certs are retained
    • 5.1 or 5.5 simple installs will be upgraded to an embedded PSC
    • 5.1 or 5.5 custom install with external SSO will be upgraded to external PSC.
      • External PSC allows you to have enhanced linked mode. Enhanced linked mode allows you to have multiple vCenters connected to same domain. Can search & manage across vCenters
      • If auto deploy exists it is upgraded as part of the VC upgrade. If auto deploy is separate, it is migrated to the VC and old auto deploy shut down.
        • settings are migrated
        • hosts must be reconfigured to point to the new auto deploy server
      • if web client is external, it is upgrades as part of the VC upgrade and migrated to the VC
      • inventory service is migrated to VC.
        • Data only is migrated
        • legacy inventory server remains and must be uninstalled manually.
      • Basically everything migrates to the new VC except for SSO.
  • Backup vCenter Server database, configuration and certificate datastore
    • Make a full backup of the vCenter Server database and the vCenter Inventory Service database. For the vCenter Server database, see the vendor documentation for your vCenter Server database type. For the Inventory Service database, see the topics “Back Up the Inventory Service Database on Windows” and “Back Up the Inventory Service Database on Linux” in the vSphere Installation and Setup documentation.
    • %ALLUSERSPROFILE%\VMware\VMware VirtualCenter\SSL
    • http://pubs.vmware.com/vsphere-60/topic/com.vmware.vsphere.install.doc/GUID-539B47B4-114B-49BC-9736-F14058127ECA.html
    • Full backups only, no FT.
    • use VDP
    • can restore each component separately
    • or do an emergency restore if vcenter is unavailable
      • disconnect host from vcenter choose to do
    • after restoring PSC you must run psc-restore script before starting services or restarting the PSC computer.
    • Backup SSL certificates
      • To keep your current SSL certificates, back up the SSL certificates that are on the vCenter Server system before you upgrade to vCenter Server 6.0. The default location of the SSL certificates is %allusersprofile%\Application Data\VMware\VMware VirtualCenter.
    • ***read all about certificates. upgrade guide page 59
  • Perform update as prescribed for Appliance or Installable
    • windows
      • download iso
      • check md5sum
      • mount iso (or extract)
    • VCSA
      • download the VCSA iso
      • install the client integration plug-in
      • launch the vcsa-setup.html, which leverages the client integration plugin and launches an installation wizard. The wizard takes necessary input for the deployment and deploys the host.
    • Installing via command line
  • Upgrade vCenter Server Appliance (vCSA)
    • “The vCenter Server Appliance is always upgraded to vCenter Server Appliance with an embedded Platform Services Controller. If you want to use the vCenter Server Appliance with an external Platform Services Controller, you must deploy a new vCenter Server Appliance. Version 6.0 of the vCenter Server Appliance uses the embedded PostgreSQL database, which is suitable for environments with up to 1,000 hosts and 10,000 virtual machines”
    • Upgrading the vCenter Server Appliance in the vSphere Upgrade Guide on page 92.
    • Deploying the vCenter Server Appliance requires the Client Integration Plug-In. This is an HTML installer for Windows which can be used to connect directly to an ESXi 5.x or ESXi 6.x host to deploy the vCenter Server Appliance on the host.
    • vCenter Server Appliance Upgrade Tasks:
      • Download the .iso installed for the vCenter Server Appliance and Client Integration Plug-in
      • Install the Client Integration Plug-in from the vcsa directory on the .iso
      • Upgrade the vCenter Server Appliance using vcsa-setup.html on the .iso
    • During the vCenter Server Appliance upgrade the following tasks are performed:
      • Export of the existing vCenter Server Appliance Configuration
      • Deployment of the vCenter Server Appliance 6.0
      • Migration of services and configuration data to the new vCenter Server Appliance 6.0
      • The legacy vCenter Server Appliance is powered off
    • A few really great blog posts on the vCenter Server Appliance upgrade process:

Given a scenario, determine the upgrade compatibility of an environment

    • Identify upgrade requirements for vCenter
    • DNS needs to be working and time must be synchronized.
      • FQDN must be resolvable.
    • vCenter Server for Windows Requirements in the vSphere Upgrade Guide on page 30.
      • vCenter Server requires a 64-bit operating system.
      • The earliest Windows version supported for vCenter Server is Windows 2008 SP2.
      • vCenter Server 6.0 for Windows can run on a physical or virtual machine.
      • A 64-bit system DSN is required to connect to an external vCenter database. vCenter Supports Oracle and Microsoft SQL Server databases.
        • Microsoft SQL Server 2008 R2 SP2 or higher, Microsoft SQL Server 2012, Microsoft SQL Server 2014, Oracle 11g, and Oracle 12c are supported database types.
        • vCenter Server databases require a UTF code set.
        • If the Microsoft SQL Express databases was used it will be migrated to the PostgreSQL database during the upgrade.
      • The bundled PostgreSQL database for windows based vCenter can be used for environments of up to 20 hosts and 200 virtual machines.
      • The user account the vCenter Server service is running as requires the following permissions:
        • Member of the Administrators group
        • Log on as a service
        • Act as part of the operating system
      • vCenter Server Appliance Requirements in the vSphere Upgrade Guide on page 33.
        • Synchronize clocks
        • Verify DNS matches
        • Do not install vcenter on a AD controller
        • Verify that local service account has read permission on the folder where vCenter is installed & HKLM
        • Verify connectivity to AD controller
      • vCenter Server Appliance 5.1U3 and vCenter Server Appliance 5.5 can be upgraded to vCenter Server Appliance 6.
      • VMware vCenter Server Appliance can be deployed only on hosts that are running ESXi version 5.0 or later.
      • If an external vCenter SSO is used, the vCenter Server Appliance cannot be upgraded. Upgrade is only supported if the embedded vCenter SSO is used.
      • The vCenter Server Appliance PostgreSQL database supports up to 1000 hosts and 10,000 virtual machines.
      • An Oracle 11g database or an Oracle 12c database are the only external databases supported by the vCenter Server Appliance.
      • recommended topologies: https://kb.vmware.com/selfservice/microsites/search.do?language=en_US&cmd=displayKC&externalId=2108548
      • for VCSA upgrade to version 6 the source and target esxi hosts must not be in lockdown or maintenance mode
      • VCSA can only be deployed only by using the Client Integration Plug-in
        • html installer for windows that connects directly to esxi host and deployes the appliance
    • Determine correct order of steps to upgrade a vSphere implementation
      • Diagrams of the process are located on page 69 of the upgrade guide.
      • From the upgrade guide 
        • 1. Read the vSphere release notes.
        • 2 Verify that your system meets vSphere hardware and software requirements. See Chapter 2, “Upgrade Requirements,” on page 29.
        • 3 Verify that you have backed up your configuration.
        • 4 If your vSphere system includes VMware solutions or plug-ins, verify that they are compatible with the vCenter Server or vCenter Server Appliance version to which you are upgrading. See VMware Product Interoperability Matrix at http://www.vmware.com/resources/compatibility/sim/interop_matrix.php vSphere Upgrade 18 VMware, Inc.
        • 5 Upgrade vCenter Server
        • 6. post upgrade tasks
          • if you had separately deployed services like auto-deploy you may need to do post-configuration tasks. Ex.\ update DHCP & TFTP
          • any 5.x services which were migrated but not removed during the upgrade process can be shutdown & removed
          • upgrade vsphere client
          • update “vcenter server – tc”, the vmware tomcat server
            • JVM heap settings can be tuned (typically upwards) to reduce garbage collection -> improve performance
              • java option = maxmemorysize
          • Add vCenter server administrator role
            • If the PSC is deployed externally the local admin group is not automagically assigned vcenter server administrator rights as in earlier versions. To enable users other than administrator, you must add them
            • logon to vsphere web client as administrator@domainname (typically administrator@vsphere.local). Click on Administration -> single sign on. If necessary add an identity source and then configure the additional administrator users under users and groups.
            • NOTE: localos is still acceptable as an identity source, but only for the PSC server. LocalOS is not recommended as an identity source for anything but the most basic of deployments.
          • Deploy a secondary PSC.
            • deploy secondary PSC and join it to the same SSO domain that’s on PSC #1.
            • with a secondary PSC, you can repoint the VCSA or vCenter server to PSC#2 at any time
            • review the vSphere upgrade guide for additional post-upgrade tasks

 

 

        • 7. upgrade VUM
          • can upgrade VUM 4.x & 5.X 64bit to VUM6.0
          • to move a 32-bit VUM instance you cannot do an in place upgrade
          • VM patch baselines are removed during upgrade
          • stop VUm and backup the database before
          • don’t forget to upgrade/enable the VUM client plugin after
        • 8. upgrade esxi hosts
          • if you have solutions/plugins/extensions make sure they are compatible prior to proceeding
          • make sure the hardware complies with the VMware compatibility guide
          • know the supported methods of upgrading various versions to 6.0. Page 115 of the vsphere upgrade guide, typically:
            • vSphere Update Manager. ***vmware recommended
            • Interactive upgrade from CD, DVD, or USB drive. OR PXE*** appropriate for a small number of hosts
            • Scripted upgrade.
              • shift-o during boot loader to get to options and point to the kickstart. boot options are on page 147 of the upgrade guide
            • vSphere Auto Deploy. If the ESXi 5.0.x host was deployed by using vSphere Auto Deploy, you can use vSphere Auto Deploy to reprovision the host with a 6.0 image.
              • custom certs are (re)configured after the upgrade
            • The esxcli command.
          • If the host being upgraded has a supported VIB, the VIB is migrated. If the VIB is incompatible an error identifies the VIB and you can either remove it or build a custom ISO that contains the VIB
          • DON’t use DHCP for your hosts (ever) during VUM upgrades
        • 9. reconnect hosts to vcenter and re-apply licenses
        • 10 (optional) set up syslog
        • 11. upgrade VM’s and appliances
  • gotchas
    • vc6 can’t manage 4.x
    • can’t go direct from vc4 -> vc6
    • VCSA includes vpostgres for embedded database. VCSA only supports Oracle for external database.
    • can continue to use VMFS3, but cannot create new VMFS3
    • if you use a user account for running vcenter service, must be a member of the local administrators.
      • needs log on as service
      • same user for DSN
      • also means can use windows authentication for SQL -> greater security
  • Identify/troubleshoot vCenter upgrade errors
    • Troubleshooting a vSphere Upgrade in the vSphere Upgrade Guide on page 185.
    • Windows vCenter Server Upgrade Logs
      • Installation/upgrade log files are located in %TEMP% and %PROGRAMDATA%\VMware\CIS\logs
        • vminst.log
        • pkgmgr.log
        • pkgmgr-comp-msi.log
        • vim-vcs-msi.log
    • To Collect vCenter Server Appliance Logs from the bash shell run vc-support.sh script to generate the log bundle. A .tgz file will be generated in /var/tmp
    • To determine which firstboot script failed cat the /var/log/firstbootStatus.json file. This will assist in identifying the firstboot script which failed. Log files for the firstboot scripts are also located in /var/log.