Oh IPMI … *sigh*

A rough and frustrating week trying to diagnose the silliest problem with our workplace’s deployment of IPMI.  I swear, I never needed a device driver for a DB-9 connector and cable to get this to work before.  I can’t for the life of me figure out why anyone would intentionally use this garbage.

IPMI, in theory, is one of those bits of computer faff that is in every single server now-a-days as a bullet point entry on a saleperson’s power point presentations.  Everyone asks for it, nobody really uses it.  I mean, come on.  This nonsense is so ridiculous, its not even funny.

The idea, is that you have a tiny computer, in your computer (yo dawg, I herd u lik computers …) to manage the power, sensors and serial console of the machine.  This is sold as a good thing, but in practice is one of those paths that a sysadmin or engineer will find themselves perpetually pulled down, with no hope of escape.  This has happened to me and my coworkers, and we are not pleased.

This week’s debacle revolved around a strange alignment of settings and hardware that yeilded results that were confounding and frustrating.  Let’s start with item #1 and see if you can deduce from it what madness we inflicted upon ourselves.

Exhibit A: http://svnweb.freebsd.org/base?view=revision&revision=236216

A very nice patch, sort of based on the on-line Broadcom open-source documentation, but the Management Firmware Interface to deal with IPMI and other things isn’t 100% clearly documented.  So, the fact that this works for people in the FreeBSD community is kind of awesome.

Short explanation of that code:  It tries to save the settings on the IPMI interface before the driver resets everything and configures for operations.  The IPMI controller is its own computer and negotiates Ethernet settings on its own.  So, its “nice” that we try to not punch it in the face and take its lunch money.

Exhibit B:  IPMI interface from the Dell R410, Enterprise DRAC connected to Cisco 3560 on a port that is set to auto/FULL

The IPMI controllers on Dell boxes are called “DRAC”.  The “DRAC” connection is separate from the server interfaces and is *only* capable of 10/100.  Now think about what I just said about the port settings of auto/FULL.  ew.  There’s drama about 10/100 and “full” settings from history that we can go into at a later date when I more fully understand it, but needless to say, it doesn’t make the DRAC very happy.

Exhibit C:  The IPMI Serial Over Lan (SOL) interface on the Dell R410 allows you to still use the DB-9 interface on the server as a second TTY or serial communications interface for things like modems, GPS receivers, etc.  Unless of course you actually *connect* to the SOL interface and try to use it with ipmitool/freeipmi etc.  In this case, the second TTY (/dev/ttyu1 in the BSD case):

STOPS RESPONDING. FOR EVER.  REALLY?  ARE YOU KIDDING ME?

If you haven’t table flipped at this point, go read the IPMI standard at some point.  Prepare to be amazed and astounded by all the wizardry that it does for you.  REALLY?  I CAN READ FAN SPEEDS?  ZOMG!!@!@WTFBBQ11111

Faff.  Complete and utter FAFF.

I’d like to apologize to Dell for using their gear as an example of the nonsense in this universe.  I really feel bad for their engineers in this case.  They took a crap-ass design and made something that sort of works.

I can’t really say if the other major server folks are more/less solid at this point.  But wow, taking a completely functional design known as a “serial cable” and making it less reliable.  Amazing.