AllMost There — Intel AMT on my ThinkPad T520

Spent some time this week screwing around with the “serial” port on my laptop.  Serial consoles are one of those things that you seem to always need when doing o/s development.  Its the only reliable way to get debugging information and on PC architectures, its really the best way to find and resolve serious problems in the kernel or drivers.

Laptops dropped their DB-9, RS-232 connection years ago and up until recently, you couldn’t really do much with debugging except take pictures of your laptop’s console when it had crashed.  I discovered that my laptop has AMT features built into it and decided to see if I could make FreeBSD use it as a serial console for debugging and diagnostics.  Turns out, its mostly functional and seems to do what I want.

Turning AMT on is a bit goofy, mainly do to some fairly strict password requirements, but lets start from the beginning.  I’m using my Lenovo T520 as an example, so your BIOS will vary based on your laptop vendor.

Hit your “enter the bios key command” which on my ThinkPad is the blue “ThinkVantage” button, but for the rest of the universe is probably F10, DEL or some other normal key.  This should bring up your bios selection menu and now enter your bios configuration menus.

Turn AMT on in BIOS

AMT_BIOS2 I left the values, more or less, at the defaults here.

AMT_BIOS1Save, reboot.  Now when you hit your bios configuration button:


Hit <ctrl-P> now to enter the AMT configuration screens.  AMT_Boot2

The first time you do this, there will only be one option, so hit “1” to enter the AMT configuration screens.  We’ll come back to this menu list later.

AMT_MainIf this is your first time entering this machine’s AMT device, its going to force you to reset the password from “admin” to something else.  The password requirements on the AMT device are pretty strict, but this should explain it for you.

You’ll need to adjust one or two settings and activate the AMT configuration’s IPv4 settings.

AMT_TimeoutSelect SOL/IDRR/KVM.  We’re going to activate Legacy Serial Redirection here

AMT_Conf0Select SOL

AMT_Conf1Obviously, Enable is what we want here.

AMT_Conf2Select Legacy Redirection Mode, then select Enable


Back to Main Menu and Select Network Setup

AMT_Conf3TCP/IP Settings and Setup Wired Lan IPv4 Configuration

AMT_Conf4I’m going to set DHCP mode for this test.  You can set static IP if you wish.

AMT_Conf5Yet another Enable here.

AMT_Conf6At this point, you’re ready to save/exit/reboot.  To test that the DHCP settings are working, toggle your BIOS button again (F10/DEL/ThinkVantage) and hit <ctrl-P> to get the AMT menu again.

AMT_DHCPHit “2” to check that the AMT can indeed get an IP address.  Note that this address is the same as your laptop IP, at least it should be.  There is some magic going on here that allows the AMT device to share the MAC address and the IP address of your host.  Its interesting but also causes odd things to happen that we’ll dig into a little bit later in the post.

Now that you’ve gotten an IP address, you can boot normally into FreeBSD.  We now need to modify FreeBSD to boot up onto a non-standard (COM1,2,3,4) and get the serial console working.  This is pretty standard stuff, /etc/ttys, /boot/loader.conf, /etc/make.conf things.

Determine what the pci bus address of the UART is in your system via:
pciconf -l | grep uart
uart0@pci0:0:22:3: class=0x070002 card=0x21cf17aa chip=0x1c3d8086 rev=0x04 hdr=0x00
dmesg | grep uart
uart0: <Intel AMT – KT Controller> port 0x6070-0x6077 mem 0xf522c000-0xf522cfff irq 19 at device 22.3 on pci0
uart0: console (115200,n,8,1)
uart0: fast interrupt


comconsole_pcidev=”0:22:3″ #Determine this value from the pciconf -l

If you have a device.hints file, comment out the hints for hw.uart settings.  Leave only the setting for the flags:”isa”

add the -Dh for dual console support

Enable ttyu0
ttyu0 “/usr/libexec/getty std.115200” vt100 on secure

Issue a kill -HUP 1 to let init fire up your getty instance.  If all is well, you can now go to the computer you want to use to as a console device for this laptop.  Using FreeBSD ports, install comms/amtterm. In order to connect to your laptop and validate that its working correctly, issue the following command:
amtterm -p <password you set in the AMT> <IP address of laptop>

If you get a tty login, you’ve suceeded!amtterm: NONE -> CONNECT (connection to host)
ipv4 [] 16994 open
amtterm: CONNECT -> INIT (redirection initialization)
amtterm: INIT -> AUTH (session authentication)
amtterm: AUTH -> INIT_SOL (serial-over-lan initialization)
amtterm: INIT_SOL -> RUN_SOL (serial-over-lan active)
serial-over-lan redirection ok
connected now, use ^] to escape

FreeBSD/amd64 (powernoodle) (ttyu0)



The last steps are fairly trivial.  You need to rebuild your kernel to pickup the changes to /etc/make.conf.  If you do an installkernel and reboot, you’ll now be able to capture the boot up sequence (no boot0 support) once loader starts.  You’ll get to see the beastie menu in all its glory.

The machine I’m using uses a Intel Ethernet chipset via the em(4) driver, this has some very strange behavior that I will take a look at over the next couple of weeks.

With AMT enabled, the ethernet connection only negotiates to 100Mbit.  This is very amusing to me as it acts a lot like IPMI in this regard.  When em(4) initializes it *will* drop your amtterm connection.  Irritating, but em(4) hasn’t been taught enough logic to deal with the connection to the AMT device.

Rebooting the laptop will disconnect your AMT session.  AMT is not a server grade thing like IPMI, so there’s no garauntee of reliability at all.  AMT is sniffing packets that are addressed to the laptop and snipping them out of the air.  Its a very delicate hardware hack but it does to the trick.

Since AMT presents itself as a non-standard COM port, I’m pretty sure that the boot0 and boot2 loaders can’t use AMT as a serial console.  This means that things like gptzfsboot might have issues that you cannot debug.  As far as I can tell however, this is the one of the most likely to work mechanisms for administration of your laptop and development for things like suspend/resume support.


Vertically Integrating Synergy … no really, this is a technical post

Because I couldn’t find any help from the Internet-O-Tron today, let me put this little bit of quality suffering out there for your entertainment.

I just completed murdering my linux box at the office and converted it over to FreeBSD Current with all the lovely Xorg goodness that I need it for.  Really, I only use it to run finch in a screen session and ssh into it from outside, but it does have its value in that.

The last … LAST little bit of my desktop at the office is the synergy tool from ports.  Its a tiny little bit of computer software glue that makes using a laptop as my primary work machine much more useful.  When I dock my T520 (also running FreeBSD Current), the synergy application connects over to my work desktop and I can control/login to it with my mouse and keyboard from my laptop.  Super Effective!

One tiny nit today that really burned out my brain cells.  I had changed the hostname of the box without restarting X.  This caused all kinds of grief that I had never seen before and was super confusing.  I would have been completely lost except for the wonder that is “telnet <hostname> <port>” that I learned way back in the mists of time to troubleshoot firewall and networking issues.

Try it for yourself on a *nix box.  While logged into Gnome, XFCE, KDE or whatever, change the hostname to something other than the one that was set when X started.  <takei>OH MY</takei>

synergy cannot open secondary screen

Breaking down and running the synergy server with -f so that I could see what’s going on, I noted that there appeared to never be a live connection to the host from my client.  Icky.  I assumed initially that there was a firewalling/network problem and started looking at hosts.allow and running truss on synergyc to see what was going on.  I noted, after a long time of grinding through truss output, that synergyc was never even opening a socket to begin with, I fired off a quick “telnet <server> 24800”

When that *DID* connect to the synergys server and I saw the logs it generated, I’ll be honest, I was super confused for like 15 minutes.  The only clue that drug me out of the confusion was the check for .Xauthority stuffs in the truss output.  Geezus, if that hadn’t have been there, I’d have never remembered that I had changed my hostname after I logged in.

The simple answer, not unlike the three finger salute in windows, was to log out of X and back in.  And just like Disco in the 90s, synergy was back.

To whomever in my distant past taught me the trick to telnet as a trouble shooting aid, THANK GOD FOR YOUR EXISTENCE ON THIS PLANET.


Software Updates for Serial Cables

Oh IPMI … *sigh*

A rough and frustrating week trying to diagnose the silliest problem with our workplace’s deployment of IPMI.  I swear, I never needed a device driver for a DB-9 connector and cable to get this to work before.  I can’t for the life of me figure out why anyone would intentionally use this garbage.

IPMI, in theory, is one of those bits of computer faff that is in every single server now-a-days as a bullet point entry on a saleperson’s power point presentations.  Everyone asks for it, nobody really uses it.  I mean, come on.  This nonsense is so ridiculous, its not even funny.

The idea, is that you have a tiny computer, in your computer (yo dawg, I herd u lik computers …) to manage the power, sensors and serial console of the machine.  This is sold as a good thing, but in practice is one of those paths that a sysadmin or engineer will find themselves perpetually pulled down, with no hope of escape.  This has happened to me and my coworkers, and we are not pleased.

This week’s debacle revolved around a strange alignment of settings and hardware that yeilded results that were confounding and frustrating.  Let’s start with item #1 and see if you can deduce from it what madness we inflicted upon ourselves.

Exhibit A:

A very nice patch, sort of based on the on-line Broadcom open-source documentation, but the Management Firmware Interface to deal with IPMI and other things isn’t 100% clearly documented.  So, the fact that this works for people in the FreeBSD community is kind of awesome.

Short explanation of that code:  It tries to save the settings on the IPMI interface before the driver resets everything and configures for operations.  The IPMI controller is its own computer and negotiates Ethernet settings on its own.  So, its “nice” that we try to not punch it in the face and take its lunch money.

Exhibit B:  IPMI interface from the Dell R410, Enterprise DRAC connected to Cisco 3560 on a port that is set to auto/FULL

The IPMI controllers on Dell boxes are called “DRAC”.  The “DRAC” connection is separate from the server interfaces and is *only* capable of 10/100.  Now think about what I just said about the port settings of auto/FULL.  ew.  There’s drama about 10/100 and “full” settings from history that we can go into at a later date when I more fully understand it, but needless to say, it doesn’t make the DRAC very happy.

Exhibit C:  The IPMI Serial Over Lan (SOL) interface on the Dell R410 allows you to still use the DB-9 interface on the server as a second TTY or serial communications interface for things like modems, GPS receivers, etc.  Unless of course you actually *connect* to the SOL interface and try to use it with ipmitool/freeipmi etc.  In this case, the second TTY (/dev/ttyu1 in the BSD case):


If you haven’t table flipped at this point, go read the IPMI standard at some point.  Prepare to be amazed and astounded by all the wizardry that it does for you.  REALLY?  I CAN READ FAN SPEEDS?  ZOMG!!@!@WTFBBQ11111

Faff.  Complete and utter FAFF.

I’d like to apologize to Dell for using their gear as an example of the nonsense in this universe.  I really feel bad for their engineers in this case.  They took a crap-ass design and made something that sort of works.

I can’t really say if the other major server folks are more/less solid at this point.  But wow, taking a completely functional design known as a “serial cable” and making it less reliable.  Amazing.

Embedded FreeBSD

How to fry a router, in X easy steps: The Serial Console

DIR 825I’ve started to hack on my DLink DIR-825 B1 to get it running FreeBSD and have had some great success.  Adrian Chadd, put together a build system for this router and committed some of the needed changes to get this Atheros MIPS 24k based router working for us and I’ll try to document my steps in getting it working.

These things come with a variant of Linux installed, so the first step is to get a working serial console on it.  By default the serial console is 115200, 8N1.  There are 4 pins providing a 3.3v TTL RS-232 connection on the board, but you have to solder the pins on yourself.  Removing the board requires removal of the 2 screws underneath the small, black rubber feet on the bottom of the case.TTL Adapter

Opening the case requires a bit of pressure, but it will come apart and the top half of the case will detach, giving you access to the interior.  Next remove the two screws securing the LED shroud and main board to the case.  There may be bits of tape holding the DIR-825 B1 interior and shroud in place, so remove them and you should be able to remove the router from its case.

Your goal now is to get 4 pins soldered onto the board at JP1.  This will give you a VCC, Ground, TX and RX connection of a TTL serial adapter.  I recommend purchasing the cheap, Open Source Hardware version, either from Fry’s or online.  This will give you the flexibility to do 3.3v or 5v TTL serial and let you use any mini USB cable you have lying around to get the console working.

Once you have soldered your pins, connect them to your serial adapter.  OpenWRT has the pinouts you need.  Pin #1 is the connection closest to the label JP1 on my board.  The VCC connection is identified with a full square around the pin.  You will not need to connect up the VCC connection in any way, so leave it alone.

At this point, connect your TTL serial adapter to the USB port of your PC and open a connection to it via your favourite serial port application (minicom, cu, etc).  You should be able to power on the DIR-825 and watch the system boot up now.

Why is this important?  We have no way of interacting with the unit if we fail to boot.  Developing the O/S images for deployment require many mistakes, most of which end up requiring reboots.  There’s no easy way to determine what went wrong without some kind of log, the serial console is your most important interface to managing your router.  Besides which, its TOTALLY RAD to hack on your equipment and see the results of your work when you get the serial console working.  You don’t have to know how to code, nor do you have to even be an o/s developer to think that this is cool.

Embedded FreeBSD

The sacrifices we must make for science!

Interesting day on the phone with DLink.  When I say “day”, I do mean DAY.  More or less, Monday must be DLink’s crazy time when everyone comes back from work and finds their office routers dead or something.

The DIR-825 model C1 that I initially bought to test out MIPS on FreeBSD died a noble death for science last month, so I thought I’d see if I could get DLink to replace it for me.  Long story short, they gave me an RMA for it and the new one will be here next week.  Short story long, wow.

I think I was on hold for close to an hour to get to the level 1 technician.  This tech was super nice but awfully frustrating to deal with.  She didn’t ask if I had an existing case number, she didn’t ask what I had already done to try and resolve the issue nor did she understand that I had already gone through all the reset procedures on their web site in order to resolve this issue in the first place.  However, in the technician’s defence, I am kind of a tool.  Not to mention that she absolutely has a script that she must follow AND most people calling in are not nearly as technically savvy as I was to detonate the router in the first place.

So 1 hour on hold, then 2 hours with the level 1 technician.  In frustration, I started to ask about getting a replacement.  I assume that this is a key phrase that triggers escalation to level 2.  That’s right, LEVEL 2.  Speaking with the senior technician at level 2, it became very clear that we were speaking the same language.  Heck, the technician actually asked me what my Unix boxes see from a DHCP request.  AMAZING.  When I passed on the information that, no its quite dead now, getting to the RMA stage was almost trivial.

I wonder, what could make this situation easier for DLink and easier for their users.  I mean, most cases they deal with are just simple configuration mistakes, not real hardware issues.  My case was the exception, not the rule in their universe.  I think this means that Internet is hard.

I’ll go into more details on my science tomorrow, when I get the RMA sent off and can deal with the DIR-825 Model B1 that is totally working with FreeBSD MIPS now.  By the way, totally awesome.  SCIENCE!


Getting super social with Bacon

After getting this blog online, I was able to hook it to my twitter account, supposedly.  The entire purpose of getting a virtual machine, IP, domain name and trying to understand how computers work was so that I could post things in more than 140 characters.

I was quite suprised at the quality of the how-to I found on for setting up WordPress, with the exception of using mysql 5.6 for the installation, all seemed to be well.

Now, and only now, do I realize the folly of all this though.  One of the crutches I’ve been heavily leaning on was ZFS, The Bacon Of Filesystems:

I use it for snap shotting my file systems before doing any kind of buildworld or port upgrades.  That has given me numerous changes to rollback before any serious damage has occured, at least until now.  I only NOW realize that my VM is a small, UFS limited thing without the safety net of BACON.  I LOVE BACON.  It makes EVERYTHING better.  Seriously.

How do you even do backups reliably on UFS?  With ZFS you set a snap shot via “zfs snapshot -r zroot” and you’re done.  (apply “bam” noise here)

If you have an issue with the current state of affairs, no worry, zfs rollback and you’re done.  DUN.

I guess I need to learn more about backups in this non-bacon world.