Replace a faulty boot media without losing ESXi configuration

One of the ESXi’s I have in the lab started reporting bootbank errors.

Simultaneously I noticed that power operations, VM snapshots and some other tasks stopped going through. This is when I knew that it was time to replace a decade old USB stick that this host was booting from.

Issue detected on 192.168.1.150 in Datacenter : Bootbank cannot be found at path '/bootbank' (2022-02-19T02:07:40.880Z cpu1:744685)

After a reboot, host would usually work normally for at least a day so it was enough to back up its configuration via PowerCLI:

My servers don’t have any monitors or peripherals plugged in, so I wanted to make it work without having to look around for cables. I performed all the steps below on a different ESXi but it should be possible to use the same host with minimal adaptations.

So the next step is to create a VM, insert a new USB stick in the server and pass it through to the VM:

Next we boot from ESXi installation ISO which has the same build number as the original host and go through installation steps:

The issue now that there doesn’t seems to be a native way to boot a VM from a pass-through USB device. So the workaround is to use Plop boot manager as intermediary. Which is basically a bootable CD that allows you to select a USB device to continue the boot with. The latest available version is 5.0.15 that was released almost a decade ago.

Nested ESXi successfully booted and picked an IP via DHCP. Next step is to apply the configuration that has been extracted from the original host:

Connect-VIServer 192.168.1.224
Set-VMHost -VMHost 192.168.1.224 -State ‘Maintenance’
Set-VMHostFirmware -VMHost 192.168.1.224 -Restore -SourcePath "C:/configBundle-192.168.1.150.tgz"

PowerCLI warned me about some mismatch between the original host and the new one but ultimately applied the configuration:

Set-VMHostFirmware : 2022-02-19 3:59:00 PM	Set-VMHostFirmware		fault.MismatchedBundle.summary	
At line:1 char:1
VMHost               UploadUrl                                                   
------               ---------                                                   
192.168.1.224        http://192.168.1.224/tmp/configBundle.tgz

I removed the original ESXi from vSphere inventory, replaced its faulty USB drive with the one that we just prepared and booted the server. Then I re-added the host in the inventory and it has been automatically placed in its original cluster. Local datastores and all other configurations were restored to their original values.