Using qemu-user to chroot and bootstrap other architectures on #FreeBSD

My last post spawned enough feedback that I thought I would dump some notes here for those interested in building a chroot on FreeBSD that allows you to test and prototype architectures, e.g. ARMv6 on AMD64.

The FreeBSD buildsys has many targets used for many things, the two we care about here are buildworld and distribution.  We will also be changing the output architecture through the use of TARGET and TARGET_ARCH command line variables.  I’ll assume csh is your shell here, just for simplicity.  You’ll need 10stable or 11current to do this, as it requires the binary activator via binmiscctl(8) which has not appeared in a release version of FreeBSD yet.

Checkout the FreeBSD source tree somewhere, your home directory will be fine and start a buildworld.  This will take a while, so get a cup of tea and relax.

make -s -j <number of cpus on your machine> buildworld TARGET=mips TARGET_ARCH=mips64 MAKEOBJDIRPREFIX=/var/tmp

Some valid combinations of TARGET/TARGET_ARCH are:

mips:mips

mips:mip64

arm:armv6

sparc64:sparc64

powerpc:powerpc

powerpc:powerpc64

i386:i386

amd64:amd64

Once this is done, you have an installable tree in /var/tmp.  You need to be root for the next few steps, su now and execute these steps:

make -s installworld TARGET=mips TARGET_ARCH=mips64 MAKEOBJDIRPREFIX=/var/tmp DESTDIR=/opt/test

DESTDIR is where you intend on placing the installed FreeBSD system.  I chose /opt/test here only because I wanted to be FAR away from anything in my running system.  Just to be clear here, this will crush and destroy your host computer without DESTDIR set.

Next, there are some tweaks that have to be done by the buildsys, so run this command as root:

make -s distribution TARGET=mips TARGET_ARCH=mips64 MAKEOBJDIRPREFIX=/var/tmp DESTDIR=/opt/test

Now we need to install the emulator tools (QEMU) to allow us to use the chroot on our system.  I suggest using emulators/qemu-user-static for this as Juergen Lock has set it up for exactly this purpose.  It will install only the tools you need here.

Once that is installed, via pkg or ports, setup your binary activator module for the architecture of your chroot.  Use the listed options on the QEMU user mode wiki page for the architecture you want.  I know the arguments are not straight forward, but there should be examples for the target that you are looking for.

For this mips/mips64 example:

binmiscctl add mips64elf –interpreter “/usr/local/bin/qemu-mips64-static”
–magic “x7fx45x4cx46x02x02x01x00x00x00x00x00x00x00x00x00x00x02x00x08”
–mask “xffxffxffxffxffxffxffx00xffxffxffxffxffxffxffxffxffxfexffxff”
–size 20 –set-enabled

Copy the binary qemu that you setup in this step *into* the chroot environment:

mkdir -p /opt/tmp/usr/local/bin

cp /usr/local/bin/qemu-mips64-static /opt/tmp/usr/local/bin/

Mount devfs into the chroot:

mount -t devfs devfs /opt/tmp/dev

Want to try building ports in your chroot?  Mount the ports tree in via nullfs:

mkdir /opt/tmp/usr/ports

mount -t nullfs /usr/ports /opt/tmp/usr/ports

And now, through the QEMU and FreeBSD, you can simply chroot into the environment:

chroot /opt/tmp

Hopefully, you can now “do” things as though you were running on a MIPS64 or whatever architecture machine you have as a target.

arm:armv6, mips:mips, mips:mips64 are working at about %80-90 functionality.  powerpc:powerpc64 and powerpc:powerpc are still a work in progress and need more work.  sparc64:sparc64 immediately aborts and probably needs someone with an eye familiar with the architecture to give QEMU a look.  If you are interested in further development of the qemu-user targets, please see my github repo and clone away.

If you are looking to see what needs to be done, Stacey Son has kept an excellent log of open item on the FreeBSD Wiki

Sometimes you have to sit down and write #FreeBSD documentation

When working on new projects or hacks, sometimes you just have to stop, think and start writing down your processes and discoveries. While working on bootstrapping the DLink DIR-825C1, I realized that I had accumulated a lot of new (to me) knowledge from the FreeBSD Community (namely, Adrian Chadd and Warner Losh).

There is a less than clear way of constructing images for these embedded devices that has an analogue in the Linux community under the OpenWRT project. Many of the processes are the same, but enough are different that I thought it wise to write down some of the processes into the beginning of a hacker’s guide to doing stuff and/or things in this space.

The first document I came up with was based on the idea that we can netboot these little devices without touching the on-board flash device. This is what you should use to get the machine bootstrapped and figure out where all the calibration data for the wireless adapters exist. This is crucial to not destroying your device. The wireless calibration data (ART) is unique to each device, destroying it will mean you have to RMA this device.

The second document I’ve created is a description of how to construct the flash device hints entries in the kernel hints file for FreeBSD. I found the kernel hints file to be cumbersome in comparison to the linux kernel way of using device specific C files for unique characteristics.

Its interesting stuff if you have the hankering to dig a bit deeper into systems that aren’t PC class machines.

Wait … Why is this stuff so hard? — Hacking the Dell DRAC

We recently “destroyed” an Dell Poweredge R720 at the office and it was to be retired to the dust bin.  We ran the Dell updates for firmware things and the machine now is completely unbootable.  Too bad, it was a very nice machine.

But, this led me down an interesting road.  I cracked open the server and started removing components one by one.  Lo and behold I noted that there was a very interesting 4 pin connector on the motherboard labeled “DRAC UART”.  Hmmm … I thought this sounded interesting.  After scrounging up a multimeter, I verified that it was indeed a 3.3 volt connection and seemed to have a ground pin as well.

I grabbed a TTL to USB converted off of my coworkers desk and proceeded to hook it up.  I was surprised and amused to find out that the Dell DRAC 7 seems to be nothing more than a fancy OpenWRT instance running on a Hitachi SH4A 32bit microprocessor.

How to fry a router in X-1 steps, The Serial Console complete

If you saw my first post  regarding my attempts to use a DLink DIR-825 with FreeBSD, you noted that I was busily trying to get the serial console working.  I won’t go over the reasons for it in this new post, but get straight into the fun and games.

My motivation here is to make the serial console available without ripping open the case every time there is an issue.  FreeBSD in a OpenWRT/DD-WRT style configuration is a very open universe right now and there will be lots of changes to come.

For now, let’s start with the ingredient list:

  1. DLink DIR-825 rev. B1
  2. Soldering Iron, solder, solder braid, good tweezers, scissors, exacto-knife
  3. Jumper wire leads, F-F
  4. OSEPP FTDI Breakout Board kit
  5. Circuit Board safe superglue

Step 1 of this operation is to open the DIR-825, remove the main board and solder on pins to JP1 on the board.

Once that is secure and you’ve validated that you can use your USB TTL adapter to get a serial console then you can move forward to setup the internal mounting and give yourself an external connector.

The case is used in 2 and 3 antenna models.  This unit leaves us the center antenna mount as a  very convient place to secure our TTL USB adapter.  Get your exacto-knife and cut out the black label area covering this hole.  Then gently cut out the top half of the innner plastic ring as I have shown in this photo.

DLink_USB_Mod_CaseSee how nicely the USB connector presents itself?  Its as though we planned this out!

Next, I modified the TTL adapter itself.  I needed to remove the jumper at the center of the adapter as it collides with the capacitor directly adjacent to the mounting point.

DIR825-USB-solderingI then was able to solder a very small length of wire to short the 3.3v selector permanently.  You’ll need a steady hand and the tweezers to pull out the pins and insert the wire.DIR825-USB-33v

You’ll note that I’ve removed the pin header that comes with the adapter.  This was accomplished with a sharp pair of scissors and then using the soldering iron to extract the remainder of the pins with judicious use of tweezers.

Once that was done, I soldered on individual pins for TX, RX and ground. Note that I’ve bent the pins at a 45 degree angle.  This will avoid any issues with the pins making contact with the mainboard.

Now that you have all the pieces ready to go, its time to break out the superglue and mount the thing.  When choosing a glue, you’ll need to get one that is safe for PCB and plasitc in general, your local electronics and hobby shop can direct you to the proper stuffs.

I mounted the TTL adapter upside down so that the connector can reach the opening on the case and there are two lower points of contact for the glue to set correctly.DIR825-USB-mount

I applied glue to the two corners next to the USB header, flipped it over and brought it into firm contact with the antenna mount.  After about 1 minute, I released pressure and applied a line of glue across the surface of the TTL adapter such that it sealed against the case of the router.

DIR825-USB-GLUE2Now … WALK AWAY for at least 6 hours.  If you’re using the right kind of glue, there’s some science happening here and you need to let the glue/plastic cure to get a secure mount.DIR825-USB-FINAL

Wire up your TX, RX and ground leads, reattach the case housing and connect up your new USB serial interface to a PC.  On FreeBSD, you should see a new serial port appear.

ugen5.2: <FTDI> at usbus5
uftdio0: <FT232R USB UART> on usbus5

Go ahead, open it with your handy, dandy comm utility, cu:

cu -l /dev/ttyU0 -s 115200

If you’ve done everything right, congrats, you’ll now see the embedded linux distribution boot up.  You are now at Step 0 getting your FreeBSD based home router running.

How to fry a router, in X easy steps: The Serial Console

DIR 825I’ve started to hack on my DLink DIR-825 B1 to get it running FreeBSD and have had some great success.  Adrian Chadd, adrian@freebsd.org put together a build system for this router and committed some of the needed changes to get this Atheros MIPS 24k based router working for us and I’ll try to document my steps in getting it working.

These things come with a variant of Linux installed, so the first step is to get a working serial console on it.  By default the serial console is 115200, 8N1.  There are 4 pins providing a 3.3v TTL RS-232 connection on the board, but you have to solder the pins on yourself.  Removing the board requires removal of the 2 screws underneath the small, black rubber feet on the bottom of the case.TTL Adapter

Opening the case requires a bit of pressure, but it will come apart and the top half of the case will detach, giving you access to the interior.  Next remove the two screws securing the LED shroud and main board to the case.  There may be bits of tape holding the DIR-825 B1 interior and shroud in place, so remove them and you should be able to remove the router from its case.

Your goal now is to get 4 pins soldered onto the board at JP1.  This will give you a VCC, Ground, TX and RX connection of a TTL serial adapter.  I recommend purchasing the cheap, Open Source Hardware version, either from Fry’s or online.  This will give you the flexibility to do 3.3v or 5v TTL serial and let you use any mini USB cable you have lying around to get the console working.

Once you have soldered your pins, connect them to your serial adapter.  OpenWRT has the pinouts you need.  Pin #1 is the connection closest to the label JP1 on my board.  The VCC connection is identified with a full square around the pin.  You will not need to connect up the VCC connection in any way, so leave it alone.

At this point, connect your TTL serial adapter to the USB port of your PC and open a connection to it via your favourite serial port application (minicom, cu, etc).  You should be able to power on the DIR-825 and watch the system boot up now.

Why is this important?  We have no way of interacting with the unit if we fail to boot.  Developing the O/S images for deployment require many mistakes, most of which end up requiring reboots.  There’s no easy way to determine what went wrong without some kind of log, the serial console is your most important interface to managing your router.  Besides which, its TOTALLY RAD to hack on your equipment and see the results of your work when you get the serial console working.  You don’t have to know how to code, nor do you have to even be an o/s developer to think that this is cool.

The sacrifices we must make for science!

Interesting day on the phone with DLink.  When I say “day”, I do mean DAY.  More or less, Monday must be DLink’s crazy time when everyone comes back from work and finds their office routers dead or something.

The DIR-825 model C1 that I initially bought to test out MIPS on FreeBSD died a noble death for science last month, so I thought I’d see if I could get DLink to replace it for me.  Long story short, they gave me an RMA for it and the new one will be here next week.  Short story long, wow.

I think I was on hold for close to an hour to get to the level 1 technician.  This tech was super nice but awfully frustrating to deal with.  She didn’t ask if I had an existing case number, she didn’t ask what I had already done to try and resolve the issue nor did she understand that I had already gone through all the reset procedures on their web site in order to resolve this issue in the first place.  However, in the technician’s defence, I am kind of a tool.  Not to mention that she absolutely has a script that she must follow AND most people calling in are not nearly as technically savvy as I was to detonate the router in the first place.

So 1 hour on hold, then 2 hours with the level 1 technician.  In frustration, I started to ask about getting a replacement.  I assume that this is a key phrase that triggers escalation to level 2.  That’s right, LEVEL 2.  Speaking with the senior technician at level 2, it became very clear that we were speaking the same language.  Heck, the technician actually asked me what my Unix boxes see from a DHCP request.  AMAZING.  When I passed on the information that, no its quite dead now, getting to the RMA stage was almost trivial.

I wonder, what could make this situation easier for DLink and easier for their users.  I mean, most cases they deal with are just simple configuration mistakes, not real hardware issues.  My case was the exception, not the rule in their universe.  I think this means that Internet is hard.

I’ll go into more details on my science tomorrow, when I get the RMA sent off and can deal with the DIR-825 Model B1 that is totally working with FreeBSD MIPS now.  By the way, totally awesome.  SCIENCE!