CTVMUG – Ditch the UI

2018-03-01 21_33_09-ClipboardFirst off, if you attended my session at the CT Usercon, thank you for attending and engaging. If you weren’t there, I have to ask WHY WEREN’T YOU!?!

I honestly tried to jam a bit much content into the presentation, so as promised here is all of the code I presented today, complete with more detailed comments (after I get some rest). The code is way too much for one post, so I’m going put the code out now and expand on the posts more in the coming days.

Create a new vDS and add a server   (updated March 3, 2018)

Update from .msi based PowerCLI to module-based PowerCLI from the PowerShell Gallery (updated March 13, 2018)

Building Blocks: increasingly larger samples of VM deployment

Building Blocks: modify disks via PowerCLI #1

Building Blocks: PowerCLI HardDisk #2, introducing parameters and functions

Cleanup snapshots

Tag based SPBM policy

Checking SPBM compliance

Install programs via Invoke-VMScript (updated March 3, 2018)

Performance Reports via vCenter statistics

Get-EsxCli oh my! (updated March 2, 2018)

vRops & Remove-VM


This VM is getting long in the tooth…

Leverage Operations Manager to check the “summary|oversized metric and if the VM crosses thresholds… So long, farewell, auf Wiedersehen, good night

### Connect to vROps server
Connect-OMServer -Server "Demo" -Credential $cred -AuthSource "ADdomain"

### set some things that will be used later when we do some things
$when = $(get-date).AddDays(-20)
$vkey = "Summary|Oversized"
$threshold = ".75"

foreach ($vm in $(get-vm|Select-Object -First 33)){
    ### get-stats on each VM from vROps
    $avg = $vrstat|Get-OMStat -Key $vkey -from $when|Select-Object -ExpandProperty Value|Measure-Object -Average

    ### Remove VM's that surpass the threshold
    write-host $vm.name, $avg.average
    if($avg.Average -gt $threshold){
        write-host $vm.name, $avg.average
        if($vm.PowerState -eq "PoweredOn"){
            ### Confirm and WhatIf have been set to true in order to protect the innocent
            stop-vm -vm $vm -Confirm:$true -WhatIf:$true
    Start-Sleep 3
    ### Confirm and WhatIf have been set to true in order to protect the innocent
    Remove-VM -vm $vm -DeletePermanently -RunAsync -Confirm:$true -WhatIf:$true

Get-EsxCli oh my!

This post is the first detailed post coming out of my presentation Ditch the UI – vSphere Management with PowerCLI at the CT VMUG Usercon. Stay tuned for the full bakers dozen of code posts!

esxcli, oh my…

I was finishing a task the day before my presentation, at the CT VMUG usercon – Ditch the UI when I happened to come across the cmdlet Get-ESXCli. My first thought was “No. Really? This command exists and I’m just finding out about it now?” sigh…

But yes it’s true, the fine folks on the PowerCLI provided us this cmdlet some time ago, and I’m just learning about it now. Well, better late than never as they say.

Before we go any further, let’s take a minute to talk about esxcli. ESXCLI is a command line interface (duh) for ESXi systems. It’s a means to modify the VMHost systems, mostly but not entirely, as it relates to the hardware systems. It’s an early & essential, but limited, way of configuring some of the management elements for your ESXi hosts.

Now I mention it’s limitations because they are essential to our story. The first big limitation is the structure of tool. It’s structure is that of namespaces with a hierarchical tree. This means that when you want to access, oh I don’t know, VMFS information/properties/methods you had to know that VMFS exists as a sub-namespaces under the esxcli storage namespace. Not a big deal and some folks actually like the structure, although it’s never really work for me.

Of bigger concern with esxcli, is locality. What I mean is that it was difficult to manage multiple systems. Of course you had options like the vMA or vCLI, but again you’re limited with connectivity options. And it was ugly to programatically try to get around these limitations.

Needless to say I was stoked to come across get-esxcli the other day.

Get-ESXCli! Oh My!!!

I’m a tinkerer. When I see a new toy, I just want to fiddle with it. That being said, I also know from experience (good and bad) what you can do with esxcli, so my first stop on Wednesday was get-help.

2018-03-02 21_27_50-Windows PowerShell

Not terribly helpful… I guess we have no choice but to dive in. To start off, I saved the output of get-esxcli to a variable.

$esxcli=get-esxcli -VMHost $vmhost -V2

Now that we’ve got esxcli (see what I did there?) stored as an object, let’s see what we can do. First things first, echo it back to the screen to see what we get right out of the gate.


Ok… well, this gives me something to go off of. It looks just like the structure of esxcli. I suppose that this shouldn’t be shocking, but it certainly is reassuring that we’re treading in semi-familiar territory. So it’s probably save to start moving down the tree.


Thank you Jeffrey Snover and team for allowing us to experience the wonders of things like tab complete and accessing sub-elements via the dot operator. So looking at a the output of $esxcli.network.nic we see that we can access things like TCP Segmentation Offload and VLAN’s. We get a few direct methods against our current level of the tree in addition to a Help() method.


OK, so this is starting to make a little more sense now. We’re basically taking the same namespaces and methods, but just making them look like PowerShell. At this point I figured I was good to go and just started firing off commands… which resulted in a lot of red text. After one of my frustrating attempts I actually paid attention to what tab-complete was showing me


Hey there “Invoke”? What are you doing there? How come I haven’t seen you in these parts before? After a few tests to get the syntax down, I learned that the .Invoke() method is how we actually get work done, and I can get down to the task I’d originally been so happy to automate.

Accessing the VAAI primitive

Ok, so I wasn’t quite ready to get to work yet.


After checking out help, just to make sure, I was ready to go…

Here’s my first script using the get-esxcli cmdlet. My whole intent was to programmatically go through my datastores, which resided on a thin-provisioned SAN, to free up any space that needed reclamation.

### Disclaimer. This one needs work, I was just so excited to learn about this cmdlet! To be continued....
Set-PowerCLIConfiguration -WebOperationTimeoutSeconds -1 -Scope Session -Confirm:$false

foreach ($cluster in $(get-cluster)){
  $dsarray=get-cluster $cluster|get-datastore
  $vmhost=get-cluster $cluster|Get-VMHost|select -First 1
  $esxcli=get-esxcli -VMHost $vmhost -V2
  foreach ($ds in $dsarray|where{$_.Type -eq 'VMFS'}){
      $esxcli.storage.vmfs.unmap.Invoke(@{reclaimunit = 60; volumelabel =$ds.name})

I went at this script knowing that my storage volumes were mapped to all hosts in each of my clusters. If you weren’t in a similar vanilla situation you’d have to get a bit tricksier with your selection logic. However with knowledge of my environment at hand it was pretty simple to:

line 4: Iterate through each of my clusters.
line 5: Get each datastore for the specified cluster
line 6: Get the first VMHost in the cluster. This is ok because I know that all datastores are associated with all hosts in a given cluster.
line 7: Use our new friend get-esxcli against the previously retrieved VMHost.
lines 8-10: Iterate through each datastore in the cluster and run leverage the invoke method against the VMFS UNMAP primative
$esxcli.storage.vmfs.unmap.Invoke(@{reclaimunit = 60; volumelabel =$ds.name})

With that I was able to return a not insignificant amount of freed up space back to my array. After writing PowerShell code leveraging PowerCLI for the last few years, I’m feeling like I have a new array of opportunities open to me after newly discovering the Get-ESXCli cmdlet.

Happy scripting!



Performance Reports via vCenter statistics

Leverage get-stat to pull statistics from vCenter, build performance report based on averages

$myCol = @()
$start = (Get-Date).AddDays(-30)
$finish = Get-Date

$objServers = get-cluster cluster | Get-VM
foreach ($server in $objServers) {
    if ($server.guest.osfullname -ne $NULL){
        if ($server.guest.osfullname.contains("Windows")){
            $stats = get-stat -Entity $server -Stat "cpu.usage.average","mem.usage.average" -Start $start -Finish $finish

            $ServerInfo = ""|Select vName, OS, Mem, AvgMem, MaxMem, CPU, AvgCPU, MaxCPU, pDisk, Host
            $ServerInfo.vName  = $server.name
            $ServerInfo.OS     = $server.guest.osfullname
            $ServerInfo.Host   = $server.vmhost.name
            $ServerInfo.Mem    = $server.memoryGB
            $ServerInfo.AvgMem = $("{0:N2}" -f ($stats | where {$_.MetricId -eq "mem.usage.average"} |Measure-Object -Property Value -Average ).Average)
            $ServerInfo.MaxMem = $("{0:N2}" -f ($stats | where {$_.MetricId -eq "mem.usage.average"} |Measure-Object -Property Value -Maximum ).Maximum)
            $ServerInfo.CPU    = $server.numcpu
            $ServerInfo.AvgCPU = $("{0:N2}" -f ($stats | where {$_.MetricId -eq "cpu.usage.average"} |Measure-Object -Property Value -Average ).Average)
            $ServerInfo.MaxCPU = $("{0:N2}" -f ($stats | where {$_.MetricId -eq "cpu.usage.average"} |Measure-Object -Property Value -Maximum ).Maximum)
            $ServerInfo.pDisk  =[Math]::Round($server.ProvisionedSpaceGB,2)


$myCol |Sort-Object vName| export-csv "VM_report.csv" -NoTypeInformation

Install programs via Invoke-VMScript

This script is a sample from a presentation I gave at the Connecticut VMUG UserCon. You can see all of the scripts from that presentation here. CTVMUG – Ditch the UI

I first wrote about the wonder of Invoke-VMScript just about a year ago. If you were to search this blog, you’d see that it turns up quite often. Honestly if the fine folks that work on PowerCLI had stopped after Invoke-VMScript, I’d probably have been OK with that.

Cutting to the chase Invoke-VMScript allows you to run scripts within a Virtual Machine’s OS. It’s a very powerful tool in that you can run PowerShell, batch or bash scripts within the context of the VM’s Windows or Linux OS.

Why would you want to automate this amazing tool? Why wouldn’t you want to?!? By leveraging this beautiful cmdlet, you can in many cases fully automate the deployment of a VM or installation of a VM within an app. Let’s be honest, as awesome as templates are they can only take you so far. Automation tools like vRA essentially operate off the same behavior, and not everyone has the funding to get those tools. For people like me Invoke-VMScript can be an extremely powerful tool that can be leveraged to great effect.

2018-03-03 22_27_45-Tom Angleberger quote_ Didn't Gandalf say _With great power comes great responsiHowever, with great power comes great responsibility. Because of the power presented with this tool, I’d recommend that you pay close attention to Role Based Access Control (among many other reasons). If you have a VM admin or operator with overly generous permissions, not only can they negatively impact your vSphere environment but they can also impact your VM’s as well, never mind compliance/auditing bag of worms that you can open.

OK, so we got the bad out of the way, here’s one of my favorite pieces of code I’ve ever written. It’s my favorite because it’s not pretty, but it sure gets the job done and moves the ball forward. That’s what we’re trying to do here, so here’s an example of how to leverage Invoke-VMScript to interact with VM OS.

write-host "Installing .NET Framework 4.5.2. If this step errors, you may be able to run the install script at c:\temp\.net4.5.2install.ps1"

    $installScript = @(
        'if (!$(test-path c:\temp\.NET4.5.2)) { New-Item -ItemType directory -path c:\temp\.NET4.5.2 }',
        '$NetFxURL = "http://download.microsoft.com/download/E/2/1/E21644B5-2DF2-47C2-91BD-63C560427900/NDP452-KB2901907-x86-x64-AllOS-ENU.exe"',
        '$NetFxPath = "c:\temp\.NET4.5.2\NDP452-KB2901907-x86-x64-AllOS-ENU.exe"',
        '$WebClient = New-Object System.Net.WebClient',
        '$process = (Start-Process -FilePath $NetFxPath -ArgumentList "/q /norestart" -Wait -Verb RunAs -PassThru)',
        'wuauclt /detectnow /reportnow',
        'wuauclt /updatenow'

    foreach ($line in $installScript) {
        Invoke-VMScript `
			-ScriptText "echo ""$line"">>C:\temp\.net4.5.2install.ps1" `
			-ScriptType Bat `
			-vm $serverData.Hostname `
			-GuestCredential $creds.LocalAdmin

    Invoke-VMScript `
		-ScriptText "echo ""$line"">>C:\temp\.net4.5.2install.ps1" `
		-ScriptType Bat `
		-vm $serverData.Hostname `
		-GuestCredential $creds.LocalAdmin

Please note this script could be condensed considerably by removing the backtick ` that allows us to continue a script command onto the next line, as exampled on lines 14-27

line 3-12: Store a series of commands into the array $installScript
line 14-20: Iterate through the $installScript array. For each line make use of the Install-VMScript command to echo the value out to a local file within the VM.
line 22-27: Use Invoke-VMScript to call the script that was created in the previous block.

It’s that simple. But I guess that’s the beauty of the solution. There are way more complicated ways of accomplishing the same ends. BUT if you have the tool, it’s easy and it’s free… Well I guess I’ll end this post similarly to how I started it: Why wouldn’t you make use of this amazing tool….

Checking SPBM compliance

Use New-VIProperty to save SPBM object within VirtualMachine objects.
Test to see if VM’s are compliant with their assign policy. If not, migrate to compliant storage.

### warning, highly inefficient code follows. simply an example for how to use VIproperty
New-VIProperty -Name SpbmCompliance -ObjectType VirtualMachine -Value {
  $Args[0]| Get-SpbmEntityConfiguration -CheckComplianceNow
} -Force
### end of inefficient code. well, kind of...

$targetobject=get-cluster NCFCU-SB-DC-cl|get-vm|Where-Object {$_.name -match "sql" -or $_.name -match "application" -or $_.name -match "linux"}

### Iterate through the targetobjects and check compliance status.
foreach ($vm in $targetobject){
  if ($vm.SpbmCompliance.ComplianceStatus -ne "compliant"){
       write-host "uh oh, $vm not compliant with $($($vm.SpbmCompliance).StoragePolicy)" -ForegroundColor Red -BackgroundColor White
       ### do something about nonCompliance
        $tag=$($vm|Get-TagAssignment|Where-Object{$_.tag.Category -match "StorageTiers"}).Tag.Name
        $destStorage=Get-SpbmCompatibleStorage -StoragePolicy $tag"Policy"|Select-Object -First 1
        move-vm $vm -Datastore $destStorage
        $(get-vm $vm).SpbmCompliance.ComplianceStatus
       write-host "Hooray $VM is super duper compliant!" -ForegroundColor Black -BackgroundColor Green

Tag based SPBM policy


$targetobject=get-cluster NCFCU-SB-DC-cl|get-vm|Where-Object {$_.name -match "sql" -or $_.name -match "application" -or $_.name -match "linux"}

### Setup tags, rules, SPBM policy
New-TagCategory -Name "StorageTiers" -Cardinality Multiple -EntityType Datastore, VirtualMachine 

foreach ($category in @("Gold","Silver","Whatever")){
  New-Tag -Name $category"Tier" -Category "StorageTiers" 
  $rule = New-SpbmRule -AnyOfTags $(get-tag -Name $category"Tier") 
  $ruleset =New-SpbmRuleSet -Name $category"RuleSet" -AllOfRules $rule  
  New-SpbmStoragePolicy -Name $category"TierPolicy" -AnyOfRuleSets $ruleset 
  ###one line version
  ###New-SpbmStoragePolicy -Name $category"Policy" -RuleSet (New-SpbmRuleSet -Name $category"RuleSet" -AllOfRules @(New-SpbmRule -AnyOfTags $(new-tag -Name $category"Tier" -Category "StorageTiers")))

### assign tags to disks
New-TagAssignment -Tag "GoldTier" -Entity $(get-datastore ServerSL1)
New-TagAssignment -Tag "SilverTier" -Entity $(get-datastore ServerSL2)
New-TagAssignment -Tag "WhateverTier" -Entity $(get-datastore ServerSL3)

### iterate through targetobject, setting tags & policy
foreach ($vm in $targetobject){
  switch -Wildcard ($vm.name){

  ### If there's a tag, assign the appropriate policy that matches tags.
    New-TagAssignment -Entity $vm -Tag $tag  
    Set-SpbmEntityConfiguration -Configuration $(Get-SpbmEntityConfiguration $vm) -StoragePolicy $tag"Policy"
  Clear-Variable tag

Cleanup snapshots

Nobody likes old or bloated VM snapshots, but we’ve all had to deal with them from time to time. If you were feeling super crotchety or perhaps confident, you could run this simple one-liner and just take out all of the snapshots in your cluster.

get-cluster cluster|get-vm|get-snapshot|remove-snapshot -whatif

First off, I left the -whatif parameter in this snip to protect the innocent. But I guess if you’re really feeling that cranky…

Chances are you don’t want to be that guy, so perhaps a kinder approach might be warranted?

Get all snapshots in the targetobject and if surpassing the specified thresholds email your lazy admin
$targetobject=get-cluster cluster|get-vm


foreach($snap in $snaps) {
  if ($snap.created -lt (get-date).adddays($datethreshold) -or $snap.SizeGB -gt $sizethreshold) {
    $subject="Clean up your room!"
      You've got some housecleaning to do:
      VM:           $($snap.VM)
      Snap name:    $($snap.name)
      Created date: $($snap.created)
      Size (GB):    $([math]::Round($snap.SizeGB,2))
    Send-MailMessage -to $dest -From $source -Subject $subject -Body $body -SmtpServer $relay 

    #if you don't care about other peoples snapshots, just uncomment the next line and boom!


Building Blocks: PowerCLI HardDisk #2, introducing parameters and functions

Build upon the previous Build Block example and introduce defining parameters and functions

Also introduce New-VIProperty

### Define input parameter

### Function to get server ready for disk expansion
function Set-SnapReady{
  if (get-snapshot -vm $Guest  -ea SilentlyContinue){
    if($RemoveSnapshot = $false){
      write-host "Cannot expand disks due to snapshots. Exiting"
        get-snapshot -VM $Guest |remove-snapshot

### Function to extend virtual disk
function Set-DiskSize{
  if($Guest.Staaatus -ne "GuestToolsRunning" ){
    Write-Host "Too bad so sad, you'll have to manually extend the OS partition. Maybe you should fix VMtools..."
    get-vm $Guest|Get-HardDisk -name "Hard disk $DiskNumber"|Set-HardDisk -CapacityGB $DiskSize -Confirm:$false
  else {
    get-vm $Guest|Get-HardDisk -name "Hard disk $DiskNumber"|Set-HardDisk -CapacityGB $DiskSize -ResizeGuestPartition -Confirm:$false
  Get-HardDisk -VM $Guest -name "Hard disk $DiskNumber"

### main loop
New-VIProperty -ObjectType InventoryItem -Name "Staaatus" -ValueFromExtensionProperty 'Guest.ToolsRunningStatus' -Force

foreach ($Guest in $Guests){
  $Guest=get-vm $Guest
  Set-SnapReady $Guest
  Set-DiskSize $Guest $DiskSize $DiskNumber $Volume

Building Blocks: PowerCLI HardDisk #1

Building Blocks: Increase the capacity of vDisk via PowerCLI HardDisk commands.
Leverage Invoke-VMScript to write a file and extend the OS volume


$Guest = Get-VM -Name "Sql01"
$DiskSize = 40
$Disk = "Hard Disk 2"
$Volume = "D"

Invoke-VMScript -vm $Guest -ScriptText "Get-PSDrive -Name $volume" -ScriptType PowerShell -GuestCredential $cred

$objDisk = Get-HardDisk -VM $Guest -Name $disk
$objDisk | Set-HardDisk -CapacityGB $DiskSize -Confirm:$false
$scriptBlock = @"
    echo rescan > c:\Temp\diskpart.txt
    echo select vol $Volume >> c:\Temp\diskpart.txt
    echo extend >> c:\Temp\diskpart.txt
    diskpart.exe /s c:\Temp\diskpart.txt
"@<span id="mce_SELREST_start" style="overflow:hidden;line-height:0;">&#65279;</span>
Invoke-VMScript -vm $Guest -ScriptText $scriptBlock -ScriptType BAT -GuestCredential $cred 

Invoke-VMScript -vm $Guest -ScriptText "Get-PSDrive -Name $volume" -ScriptType PowerShell -GuestCredential $cred

PowerCLI Building Blocks – VM Deployment

### Build larger samples of ways to deploy a VM.

### First building block ###
### Just build a VM
new-vm -name "sql02" -ResourcePool "DemoPool"
### Second Building Block.
### Use a template and set some additional parameters. Set a variable, denoted by the preceeding '$' sign
$varRP = "DemoPool"
New-VM -Name "WinApplication01" -Template "base-w12-01" -OSCustomizationSpec "Win2k12" -ResourcePool $varRP
get-vm "WinApplication01"
### Third Building Block ###
 ### Create a temporary OS Specification with a custom nic mapping for setting VM's IP. Set all parameters as variables
 $osSpec = "Win2k12"
 $temposSpec = $osSpec + "_temp"
 $template = "base-w12-01"
 $Mask = ""
 $GW = ""
 $DNS1 = ""
 $IP = ""
 $vmname = "WinApplication02"

### Create a Temporary, nonpersistent OsSpec
 Get-OSCustomizationSpec $osSpec | New-OSCustomizationSpec -Name $temposSpec -Type NonPersistent

### Create a custom nic mapping with Static IP within the temporary OsSpec
 $osNicMapp = Get-OSCustomizationNicMapping -OSCustomizationSpec $temposSpec
 Set-OSCustomizationNicMapping -OSCustomizationNicMapping $osNicMapp -IpMode UseStaticIP -IpAddress $IP -SubnetMask $mask -DefaultGateway $GW -Dns $DNS1

### Create a VM using the temporary OsSpec
 New-VM -Name $vmname -Template $template -OSCustomizationSpec $temposSpec -ResourcePool $varRP


### Fourth Building Block ###
### Create a temporary OS Specification with a custom nic mapping for setting VM's IP. Set all parameters as variables

$InputFile = "C:\Temp\linux_servers.json"
$Servers = $(Get-Content $InputFile -Raw | ConvertFrom-Json).Servers

Get-OSCustomizationSpec $Server.OSCustomizationSpec | New-OSCustomizationSpec -Name $Server.temposSpec -Type NonPersistent

foreach ($Server in $Servers) {
    ###   2. Using Json output create custom splat tables, denoted by  @{}
    $nicsplat = @{
        OSCustomizationNicMapping = Get-OSCustomizationNicMapping -OSCustomizationSpec $Server.temposSpec
        IpMode                    = $Servers.IpMode
        IpAddress                 = $Server.IpAddress
        SubnetMask                = $Server.SubnetMask
        DefaultGateway            = $Server.DefaultGateway
        Dns                       = $Server.Dns

    ### 3. Use the $nicsplat table as parameters to set a custom OSNicMap and $newVMsplat as the parameters for a new VM
    Set-OSCustomizationNicMapping @nicsplat

    $newVMsplat = @{
        Name                = $Server.Name
        Template            = $Server.template
        OSCustomizationSpec = $Server.temposSpec
        ResourcePool        = $Server.ResourcePool
    New-VM @newVMsplat
    ### compare above with: New-VM -Name $vmname -Template $template -OSCustomizationSpec $temposSpec -ResourcePool $varRP


    "Servers": [
            "Name": "LinuxApp01",
            "Template": "Ubuntu16.04.3",
            "OSCustomizationSpec": "generic_app",
            "temposSpec": "generic_app_temp",
            "ResourcePool": "DemoPool",
            "IpMode": "UseStaticIP",
            "IpAddress": "",
            "SubnetMask": "",
            "DefaultGateway": "",
            "DNS": ""
            "Name": "LinuxApp02",
            "Template": "Ubuntu16.04.3",
            "OSCustomizationSpec": "generic_app",
            "temposSpec": "generic_app_temp",
            "ResourcePool": "MyOtherDemoPool",
            "IpMode": "UseStaticIP",
            "IpAddress": "",
            "SubnetMask": "",
            "DefaultGateway": "",
            "DNS": ""