The EEPROM you have reached is no longer in service, #FreeBSD recovery of em(4) devices

Ran into a very strange error with legacy em(4) devices on FreeBSD this week.  Something I was doing while working on the EM_MULTIQUEUE kernel config option for em(4) seems to have blown up the PXE configuration in the EEPROM of one of my lem(4) devices.  Else, cosmic rays did it:

em3: <Intel(R) PRO/1000 Legacy Network Connection 1.1.0> port 0xe880-0xe8bf mem 0xfeb80000-0xfeb9ffff,0xfeb60000-0xfeb7ffff irq 19 at device 6.0 on pci5
em3: The EEPROM Checksum Is Not Valid
device_attach: em3 attach returned 5

It took me a while, but it seems when this happens, you can sometimes recover the device with Intel’s Ethernet Connections Boot Utility.

There is a DOS utility in this self-extracting CAB file called “bootutil” that can be used to reset your device to defaults.  After prepping a DOS usb stick, I booted my host from it and ran the tool against all devices as I couldn’t quite tell which device was actually having the issues:

bootutil -all -defcfg

This cleared the error and now I get a successful attach.  I literally have no idea what the problem was in the first place, but solved.

Props to the IPXE folks for having a great HOWTO on generically manipulating the firmware flash that gave me the clue on what to do.