Have you ever found it a nuisance to coordinate the update of firmware on Cisco UCS servers running VMware? Cisco UCS Central has a powerful ability to update firmware across many servers quickly. The problem I always ran into was how to do this without impacting the production environment with as little manual input as possible. In the past, I would:
login to VMware vCenter and put a host into maintenance mode
login to Cisco UCS Central and acknowledge the alert for the server allowing the firmware to update
login to Cisco UCS Manager to track the progress of the firmware install
wait for the server to come back online
wait for the server to start communicating with vCenter
take the server out of maintenance mode
move on to the next server
Painfully slow! This article will show how I resolved this problem using PowerShell.
The technologies I am using include VMware vCenter, PowerShell, PowerCLI, and Cisco UCS Central. Since all these technologies are evolving quickly, below is the current revisions of each of the technologies so you can compare this with your environment.
VMware vCenter – version 6.5 update 1
PowerShell – version 5.1
PowerCLI – version 6.5 Release 1 build 4624819
Cisco UCS Central – version 2.0(1d)
The high level process I used in my PowerShell script contains the following sections
login to vCenter, if not already logged in
get the hostname of the host targeted for the firmware upgrade
put the host in maintenance mode
login to Cisco UCS Central and acknowledge the pending firmware update the host
give the host some time to update
check in vCenter to see if the host is communicating again
take the host out of maintenance mode
Building the PowerShell Script
1. Load the PowerCLI module
The prerequisite for this is to have PowerCLI installed.
Import-Module -Name VMware.VimAutomation.Core2. Login to VMware vCenter
Use the IP address or DNS name of your vCenter server in the following piece of code to login to your vCenter server. I’ve added a simple if statement before the login command to make sure I’m not already logged into the vCenter server. You can also add the -user/password switches to avoid the popup asking you to login.
###CONNECT TO VCENTER SERVER IF NOT ALREADY CONNECTED
If ($global:DefaultVIServers.Count -eq 0) {
#VCENTER SERVER VERSION 6.5 IS 10.10.10.10
Connect-VIServer -Server 10.10.10.10 -Protocol https}3. Get the hostname of host you are upgrading
There are a lot of different ways to get the hostname of the host you are upgrading. I decided to do my upgrades by cluster, so my script gets the names of all the hosts in a particular cluster. Then the script continues by iterating through each of the hosts in the cluster. I had a little challenge in that the hostnames in vCenter were fully qualified domain names (FQDN) and the hostnames in Cisco UCS Central were the short name. I used a regex function in PowerShell to strip off the domain name. Now I have two variables; one with the short name and one with the FQDN.
###SET THE CLUSTER FOR MAINTENANCE$cluster = "VCENTER CLUSTER NAME"$clusterhosts = Get-Cluster $cluster | Get-VMHost | Sort Name foreach ($clusterhost in $clusterhosts) { #THIS FOREACH LOOP CONCLUDES IN STEP 7 $vmcompname = $clusterhost.name $vmcompname -match "^(?<host>S[^.]*)" $compname = $matches["host"]
4. Make Sure the Host is Actually in Maintenance Mode Before Proceeding
This if statement makes sure the host is in maintenance mode before proceeding. I would hate to apply firmware updates to a host that didn’t successfully make it to maintenance mode.
#CHECK TO MAKE SURE THE HOST IS IN MAINTENANCE MODE BEFORE PROCEEDING
If ((Get-VMHost -Name $vmcompname).ConnectionState -eq "Maintenance") { #THIS IF STATEMENT CONCLUDES IN STEP 7
5. Connect to Cisco UCS Central RESTful XML API and Acknowledge Pending Firmware Update
This was the most difficult part of the script. I was avoiding making this part work for a few months. Although I have been in IT for over 20 years and writing system administration scripts for about 15 years, I would still consider myself a newbie when it comes to RESTful API’s. The biggest breakthrough I had when researching this was the logging function in Cisco UCS Central. UCS Central uses its XML API for everything including the it’s web interface. This means if you know how to do something in the GUI of UCS Central you can capture the URI and the XML that is sent via UCS Central’s logging mechanism. In UCS Central under “System Tools” all you need to do is click “Start Logging Session” then when you are done click “Stop Logging Session” and a log file will automatically download. Contained in the log file is all the URI’s called and the XML that is sent and received from the RESTful API. Armed with this information, the rest of the of the pieces fell into place easily.
5.1 Login to UCS Central and Get a Cookie to Use for Subsequent Requests
The Invoke-WebRequest command in PowerShell returns a string. The contents of the string are xml but PowerShell doesn’t see it that way. I used a regex match to pull out the outCookie parameter and assign it to a variable to be used later.
#LOGIN AND GET A COOKIE TOKEN
$body = "<aaaLogin inName='username' inPassword='password'></aaaLogin>"
$r = Invoke-WebRequest -Uri "https://ciscoUCScentralServerName.fqdn.com/xmlIM" -Method Post -Body $body -ContentType 'application/xml'
$r.Content -match 'outCookie="(?<a>S*)"'
$cookie = $matches['a']
5.2 Find Messages that Need Acknowledgement
The only way I found the configGetAckables XML element was by using the session logging technique in described above in UCS Central. I have not found any Cisco documentation online pointing to this method. This pulls back some XML with an ackDn parameter that can be used to acknowledge the message. This assumes you have already made either a template or profile change in UCS Central to initiate a firmware upgrade on the server with a “user acknowledge” maintenance policy applied.
#PULL BACK ALL MESSAGES THAT NEEDS ACKNOWLEDGEMENT IN CENTRAL AND FIND THE ONE FOR THE CURRENT SERVER$body = "<configGetAckables cookie='" + $cookie + "'></configGetAckables>"
$r = Invoke-WebRequest -Uri "https://ciscoUCScentralServerName.fqdn.com/xmlIM/resource-mgr" -Method Post -Body $body -ContentType 'application/xml'
$r.Content -match 'ackDn="(?<ack>S*' + $compname + 'S*)"'
$ack = $matches['ack']
5.3 Acknowledge the Firmware Message in UCS Central
This web call uses the ackDn parameter retrieved from the previous call to acknowledge the firmware update action in UCS Central.
#USING THE ACKDN IN CENTRAL - ACKNOWLEDGE THE MESSAGE TO BEGIN FIRMWARE UPDATE
$body = '<configConfMo dn="" cookie="' + $cookie + '" inHierarchical="false"><inConfig><lsmaintAck adminState="trigger-immediate" dn="' + $ack + '"></lsmaintAck></inConfig></configConfMo>'
$r = Invoke-WebRequest -Uri "https://ciscoUCScentralServerName.fqdn.com/xmlIM/resource-mgr" -Method Post -Body $body -ContentType 'application/xml'
6. Wait for the Firmware to Install
Tell the PowerShell script to go to sleep and wait for the firmware to install. In our environment it takes about 15 minutes. After the waiting period, I check to see if the host is communicating with vSphere. If the host is not communicating, log some text to the screen and wait another 60 seconds.
### SLEEP TO ALLOW FOR FIRMWARE TO INSTALL$dt = get-dateWrite-host "Sleeping for 15 Minutes" $dtSleep 900 ### CHECK VMWARE AND MAKE SURE HOST IS RESPONDING PRIOR TO MOVING ONwhile ((Get-VMHost -Name $vmcompname).ConnectionState -ne "Maintenance") { $dt = get-date write-host "Connection state is: " (Get-VMHost -Name $vmcompname).ConnectionState write-host "Waiting for $compname to come back online - sleeping 1 minute" $dt sleep 60}
7. Take the Host Out of Maintenance Mode
### BRING HOST OUT OF MAINTENANCE MODE IN VMWARE
Get-VMHost -Name $vmcompname | set-vmhost -State "Connected"
$dt = get-date
write-host "$compname is back" $dt }} #THIS IS THE END OF THE FOREACH LOOP IN THAT BEGAN IN STEP 2
That’s it. A nice programmatic way to apply firmware updates with minimal manual effort using PowerShell.
Comments