Software Updates for Serial Cables

Oh IPMI … *sigh*

A rough and frustrating week trying to diagnose the silliest problem with our workplace’s deployment of IPMI.  I swear, I never needed a device driver for a DB-9 connector and cable to get this to work before.  I can’t for the life of me figure out why anyone would intentionally use this garbage.

IPMI, in theory, is one of those bits of computer faff that is in every single server now-a-days as a bullet point entry on a saleperson’s power point presentations.  Everyone asks for it, nobody really uses it.  I mean, come on.  This nonsense is so ridiculous, its not even funny.

The idea, is that you have a tiny computer, in your computer (yo dawg, I herd u lik computers …) to manage the power, sensors and serial console of the machine.  This is sold as a good thing, but in practice is one of those paths that a sysadmin or engineer will find themselves perpetually pulled down, with no hope of escape.  This has happened to me and my coworkers, and we are not pleased.

This week’s debacle revolved around a strange alignment of settings and hardware that yeilded results that were confounding and frustrating.  Let’s start with item #1 and see if you can deduce from it what madness we inflicted upon ourselves.

Exhibit A: http://svnweb.freebsd.org/base?view=revision&revision=236216

A very nice patch, sort of based on the on-line Broadcom open-source documentation, but the Management Firmware Interface to deal with IPMI and other things isn’t 100% clearly documented.  So, the fact that this works for people in the FreeBSD community is kind of awesome.

Short explanation of that code:  It tries to save the settings on the IPMI interface before the driver resets everything and configures for operations.  The IPMI controller is its own computer and negotiates Ethernet settings on its own.  So, its “nice” that we try to not punch it in the face and take its lunch money.

Exhibit B:  IPMI interface from the Dell R410, Enterprise DRAC connected to Cisco 3560 on a port that is set to auto/FULL

The IPMI controllers on Dell boxes are called “DRAC”.  The “DRAC” connection is separate from the server interfaces and is *only* capable of 10/100.  Now think about what I just said about the port settings of auto/FULL.  ew.  There’s drama about 10/100 and “full” settings from history that we can go into at a later date when I more fully understand it, but needless to say, it doesn’t make the DRAC very happy.

Exhibit C:  The IPMI Serial Over Lan (SOL) interface on the Dell R410 allows you to still use the DB-9 interface on the server as a second TTY or serial communications interface for things like modems, GPS receivers, etc.  Unless of course you actually *connect* to the SOL interface and try to use it with ipmitool/freeipmi etc.  In this case, the second TTY (/dev/ttyu1 in the BSD case):

STOPS RESPONDING. FOR EVER.  REALLY?  ARE YOU KIDDING ME?

If you haven’t table flipped at this point, go read the IPMI standard at some point.  Prepare to be amazed and astounded by all the wizardry that it does for you.  REALLY?  I CAN READ FAN SPEEDS?  ZOMG!!@!@WTFBBQ11111

Faff.  Complete and utter FAFF.

I’d like to apologize to Dell for using their gear as an example of the nonsense in this universe.  I really feel bad for their engineers in this case.  They took a crap-ass design and made something that sort of works.

I can’t really say if the other major server folks are more/less solid at this point.  But wow, taking a completely functional design known as a “serial cable” and making it less reliable.  Amazing.

The midlife crisis mobile

image

Finally got the ok to grab “my” car today fron Scion. Thr FR-S (and Subaru BR-Z) have been on my mind for months now after warching Jeremy Clarkson’s Top Gear review.

I wouldn’t say that I’ve gone and broken any speed limits yet, that’s not what this car is for. I *can* say thay I’m no longer afraid of corners. I truly look forward to them now.

However, it is a bit of a workout driving now. The 6 speed manual is super easy to drive, but compared to the automatic transmission in my Honda CR-V, it is a workout.

How to fry a router, in X easy steps: The Serial Console

DIR 825I’ve started to hack on my DLink DIR-825 B1 to get it running FreeBSD and have had some great success.  Adrian Chadd, adrian@freebsd.org put together a build system for this router and committed some of the needed changes to get this Atheros MIPS 24k based router working for us and I’ll try to document my steps in getting it working.

These things come with a variant of Linux installed, so the first step is to get a working serial console on it.  By default the serial console is 115200, 8N1.  There are 4 pins providing a 3.3v TTL RS-232 connection on the board, but you have to solder the pins on yourself.  Removing the board requires removal of the 2 screws underneath the small, black rubber feet on the bottom of the case.TTL Adapter

Opening the case requires a bit of pressure, but it will come apart and the top half of the case will detach, giving you access to the interior.  Next remove the two screws securing the LED shroud and main board to the case.  There may be bits of tape holding the DIR-825 B1 interior and shroud in place, so remove them and you should be able to remove the router from its case.

Your goal now is to get 4 pins soldered onto the board at JP1.  This will give you a VCC, Ground, TX and RX connection of a TTL serial adapter.  I recommend purchasing the cheap, Open Source Hardware version, either from Fry’s or online.  This will give you the flexibility to do 3.3v or 5v TTL serial and let you use any mini USB cable you have lying around to get the console working.

Once you have soldered your pins, connect them to your serial adapter.  OpenWRT has the pinouts you need.  Pin #1 is the connection closest to the label JP1 on my board.  The VCC connection is identified with a full square around the pin.  You will not need to connect up the VCC connection in any way, so leave it alone.

At this point, connect your TTL serial adapter to the USB port of your PC and open a connection to it via your favourite serial port application (minicom, cu, etc).  You should be able to power on the DIR-825 and watch the system boot up now.

Why is this important?  We have no way of interacting with the unit if we fail to boot.  Developing the O/S images for deployment require many mistakes, most of which end up requiring reboots.  There’s no easy way to determine what went wrong without some kind of log, the serial console is your most important interface to managing your router.  Besides which, its TOTALLY RAD to hack on your equipment and see the results of your work when you get the serial console working.  You don’t have to know how to code, nor do you have to even be an o/s developer to think that this is cool.

Refilling the radio percolator

Tomorrow is my last Radio-Kol show before I go on hiatus.  When I logged on to my secondary KoL account this morning, I was greeted with a 5 year anniversay Concrete Fedora.  I was amused to say the least, since I created *that* account 1 day before my first show back in 2008.

Taking a hiatus from my three days a week, 5am shows was a pretty hard decision for me to make.  I enjoy having a small block of time each week that is “my” time and let’s me play a fun little video game while listening to new/old music.  There’s something to be said for waking up early and not having anyone ask anything of you and not need anything from you.

One thing I did early on and took it as my schtick, was to play records on my show.  This was something I was always quite proud of as a computer nerd.  Spinning a record and streaming it across the internet while keeping the ID3 tags updated is a bit challenging until you get the hang of it.  The IDJC software platform made it possible for me to insert small, 1 second of silence audio tracks into a playlist and then tag them with the appropriate information.  I still feel like a real DJ only when the turntable is spinning.  Playing MP3 files on the internet isn’t as satisfying to be honest, it doesn’t feel like work.

My morning listeners seem to enjoy the comedy and serialized dramas that I would scrounge up at the record store for the shows.  To be honest, these are super cheap records to collect, which is why my collection is primarily composed of them.

But, work is getting a bit intense this summer/fall.  I need the extra time for either sleep or work and I can’t fit Radio KoL into my schedule for the next few months.  Heck, I don’t even know if I’ll be having the time to spin records for my own personal enjoyment.  Without the impetus to setup a show for others to enjoy, I may not listen to another record for several months.  🙁

Its not sad though.  It makes me look forward to the time when I’ll have the mental bandwidth to come back to the shoutcast show and once again inflict The Decaffeination Zone on your earholes.

The sacrifices we must make for science!

Interesting day on the phone with DLink.  When I say “day”, I do mean DAY.  More or less, Monday must be DLink’s crazy time when everyone comes back from work and finds their office routers dead or something.

The DIR-825 model C1 that I initially bought to test out MIPS on FreeBSD died a noble death for science last month, so I thought I’d see if I could get DLink to replace it for me.  Long story short, they gave me an RMA for it and the new one will be here next week.  Short story long, wow.

I think I was on hold for close to an hour to get to the level 1 technician.  This tech was super nice but awfully frustrating to deal with.  She didn’t ask if I had an existing case number, she didn’t ask what I had already done to try and resolve the issue nor did she understand that I had already gone through all the reset procedures on their web site in order to resolve this issue in the first place.  However, in the technician’s defence, I am kind of a tool.  Not to mention that she absolutely has a script that she must follow AND most people calling in are not nearly as technically savvy as I was to detonate the router in the first place.

So 1 hour on hold, then 2 hours with the level 1 technician.  In frustration, I started to ask about getting a replacement.  I assume that this is a key phrase that triggers escalation to level 2.  That’s right, LEVEL 2.  Speaking with the senior technician at level 2, it became very clear that we were speaking the same language.  Heck, the technician actually asked me what my Unix boxes see from a DHCP request.  AMAZING.  When I passed on the information that, no its quite dead now, getting to the RMA stage was almost trivial.

I wonder, what could make this situation easier for DLink and easier for their users.  I mean, most cases they deal with are just simple configuration mistakes, not real hardware issues.  My case was the exception, not the rule in their universe.  I think this means that Internet is hard.

I’ll go into more details on my science tomorrow, when I get the RMA sent off and can deal with the DIR-825 Model B1 that is totally working with FreeBSD MIPS now.  By the way, totally awesome.  SCIENCE!

Getting super social with Bacon

After getting this blog online, I was able to hook it to my twitter account, supposedly.  The entire purpose of getting a http://rootbsd.net virtual machine, IP, domain name and trying to understand how computers work was so that I could post things in more than 140 characters.

I was quite suprised at the quality of the how-to I found on http://forums.freebsd.org for setting up WordPress, with the exception of using mysql 5.6 for the installation, all seemed to be well.  https://forums.freebsd.org/showthread.php?t=30268

Now, and only now, do I realize the folly of all this though.  One of the crutches I’ve been heavily leaning on was ZFS, The Bacon Of Filesystems:

I use it for snap shotting my file systems before doing any kind of buildworld or port upgrades.  That has given me numerous changes to rollback before any serious damage has occured, at least until now.  I only NOW realize that my VM is a small, UFS limited thing without the safety net of BACON.  I LOVE BACON.  It makes EVERYTHING better.  Seriously.

How do you even do backups reliably on UFS?  With ZFS you set a snap shot via “zfs snapshot -r zroot” and you’re done.  (apply “bam” noise here)

If you have an issue with the current state of affairs, no worry, zfs rollback and you’re done.  DUN.

I guess I need to learn more about backups in this non-bacon world.

Open for business

Wow, I’m on the internet and everything.

Thanks to some industrious people who made it ridiculously easy to get FreeBSD and WordPress working together smoothly, I now have the ability to inflict slightly less random things on the internet.  Normally, such ramblings were limited to 140 characters or less due to twitter, but NO MORE.

Welcome to my blog everyone, I’m sorry in advance.