Thursday, October 2, 2014

Guide to Cloning/backing up ESXi Servers

This has been a major headache, and it wasnt really fully documented, so I thought it's better I post it for others if they run into a similar project:

I was looking through an instructions page, which was over 10 pages long most of which was configuring a very complicated Standard Switch.  The decision was made to "clone" the ESXi host,
thereby copying the network configuration and other variables on the server.  This is on a ESXi 5.5 Update 2, however it applies all the way back to 4.0 I think.  Make sure you have the exact same version of ESXi by doing this on the command line, or the other methods:

~ # vmware -vl



Ok, now that you make sure you have the same version and build on both ESXi hosts, let's move on:

VMware has two/three tools to do a backup of an ESXi hosts, or as in my case below, to clone it:

1.)  PowerCLI:

To Create a backup:
Get-VMHostFirmware -VMHost $host -BackupConfiguration -DestinationPath C:\HostBackups

To Restore that backup:
Set-VMHostFirmware -VMHost $Host -Restore -SourcePath c:\Hostbackups\backupfile.tgz -HostUser user -HostPassword password

Another important point is that the ESXi versions have to be EXACTLY the same, otherwise you will not really get an error other than:  

    + FullyQualifiedErrorId : Client20_SystemManagementServiceImpl_RestoreVmHo
   stFirmware_ViError,VMware.VimAutomation.ViCore.Cmdlets.Commands.Host.SetVM

  HostFirmware


I had to dig in the logs (/var/log/vmware/hostd.log) to get this error below, which shows that there's a ESXi version mismatch:

2014-10-01T08:57:01.770Z [606C2B70 info 'Hostsvc.FirmwareSystem' opID=hostd-cee0 user=root] RestoreConfiguration failed with status 1. Output : Mismatched Bundle: Host release level: VMware ESXi 5.5.0 Update 1 Bundle release level: VMware ESXi 5.5.0 Update 2
-->

* Note: this tar archive is also used in method #3 to restore.  
Method #2 creates a binary file which is not at all like the tar archives.  

2.)  Command line on the VMware Management Appliance (vMA)

to create a Backup:  

vicfg-cfgbackup --server=ESXi_host_IP_address --username=root -s /tmp/ESXi_test1_backup.txt
(the -s flag is to save it)

This will create a BINARY file in /tmp/ even though we called it *.txt.  You can look at this file if you want in VIM, by doing this through VIM or any HEX editor:

vim /tmp/ESXi_test1_backup.txt
in VIM,  type :%!xxd to turn it into a hexeditor
:%!xxd -r to go back to normal mode
xxd is present in any vim installation. 

to restore that backup: 
vicfg-cfgbackup --server=ESXi_host_IP_address --username=root -l=/tmp/ESXi_test1_backup.txt -f


that last "f" in there, is to force it, as if you are restoring it to a different hardware as I was, the backup will not proceed.


3.)  On the ESXi shell


Put the host into maintenance mode by running the command:

      vim-cmd hostsvc/maintenance_mode_enter

Copy the backup configuration file to a location on the host and run the command:

      vim-cmd hostsvc/firmware/restore_config /tmp/configBundle.tgz

(this is if you copied the configBundle.tgz file that you created with PowerCLI to this host /tmp directory) 


Ok, after we did all this, you now have a clone of the ESXi server that you made the backup for.  This is ok if you have a failed server, and you get a new server and restore that configuration.  

However, if you now have 2 servers with this same configuration, besides the IP conflict, you will also have duplicate MAC addresses on all the interfaces.  


If you look at the /etc/vmware/esx.conf file, ESXi maps all the hard-coded MAC addresses to virtual ones, and so your clone although having different actual MAC addresses, will have the same VIRTUAL MAC addresses:


once you clone, you will have EXACTLY the same file on the clone.  So you need to shut down the original machine and change these values.  You can do this by deleting all the lines circled in red, and you can change the IP whereever it appears in that file (3 places)

Next step, reboot the server, and you now have a clone of the original server, which brings me to another point:

THIS WILL COPY OVER THE VMWARE LICENSE!!!

You HAVE To have an actual license on the destination server in order for this to work.  the Free version or trial version won't work.  If you are using free ESXi, the remote commands are only available for "read-only" operations. For more details, please refer to this article here.

This is what you will get if you try it:


What you CAN do if you want, is put in a license, and then change it after the box is done, this is the case for me, as these boxes by design use the ESXi free version.