One pxeboot to find them and in the darkness bind them

Irritatingly, I couldn’t find a decent cut-n-paste how to on the Internet for using a much loved and mostly overlooked FreeBSD installation and boot up process using a fully cooked diskless image fetched via TFTP instead of the really well documented and fantastic howto on NFS booting.

Primarily, I wanted to switch off of NFS booting as it tends to be really troublesome if one is doing development or debugging of the FreeBSD network stack or Ethernet drivers. I couldn’t reliably screw around with the network interfaces without disturbing the NFS root mount and inflicting relatively long timeouts on my tests. I decided to try and switch to MFS root based testing so that I didn’t run into these types of problems, however I didn’t realize that the boot loader had gone through some overhauls recently and the documentation that *is* floating around on the Internet, is a bit out of date. So, here is the beginning of what I hope will be the inception of a new FreeBSD handbook entry before version 12 is released.

I want to cover the actual construction of the entire process, so I will not be hand copying binaries from a release and all files will be constructed from source where possible.  This means that I can fully automate a build and regression from source to ensure that this document stays relatively up to date.  I’m an active FreeBSD contributor, so I have many checkouts of FreeBSD lying around.  If you’re a sysadmin, you probably use /usr/src for most of the projects like this, which is fine for our purposes.

Assuming that you have a fresh checkout of FreeBSD somewhere, lets start by crafting a “src.conf” and a “make.conf” to minimize a bit of build time and a bit of space.

I turn a lot of stuff off in the build here.  Since I’m running the same major revision of FreeBSD that I’ll be building, I can get away with some shenanigans regarding not building toolchains.  If you are trying to build 12-CURRENT on 11-STABLE, you may have to play around with the knobs a bit in src.conf:


My make.conf is much shorter, and only set’s a few things to make life easier:


Set/export a couple of variables to your local shell to use these:

setenv SRCCONF /path/to/src.conf
setenv __MAKE_CONF /path/to/make.conf

Since I’m not building as root here, I set one more variable:

setenv MAKEOBJDIRPREFIX /var/tmp/mfsroot

Execute a buildworld and turn some electrons into heat for a while.  You’ll probably note a big difference in build times here.  Also note that you’re building a much smaller FreeBSD system with a lot of missing pieces.  No local mail agent, no ntp services, no compilers/debuggers, no wireless bits … etc.  Your mileage may vary here, but I needed a small world as I’m debugging/testing kernel/network drivers, not userland.

You can make a choice here of kernels to build.  Most folks will want GENERIC-NODEBUG, but if you want, you can base a customized kernel configuration from MINIMAL that will do the trick.  You’ll need to include drivers specific to your systems and this will void all your warantees if you were given one.  Here is an example that I’m using to build and test various bits and things.  Note that I’ve added em(4) to my configuration as that’s the network interface I’m using.

include MINIMAL

nomakeoptions DEBUG # Build kernel with gdb(1) debug symbols
nomakeoptions WITH_CTF # Run ctfconvert(1) for DTrace support

nooptions COMPAT_FREEBSD32
nooptions COMPAT_FREEBSD10
nooptions COMPAT_FREEBSD11
nooptions KTRACE
nooptions AUDIT
nooptions MAC
nooptions DDB_CTF

# Debugging support. Always need this:
nooptions KDB
nooptions KDB_TRACE
# For full debugger support use (turn off in stable branch):
nooptions DDB
nooptions GDB
nooptions DEADLKRES
nooptions INVARIANTS
nooptions WITNESS
# Xen HVM Guest Optimizations
# NOTE: XENHVM depends on xenpci. They must be added or removed together.
nooptions XENHVM # Xen HVM kernel infrastructure
nodevice xenpci # Xen HVM Hypervisor services driver

options GEOM_RAID
options GEOM_LABEL
options GEOM_UZIP

options IFLIB
device em

# Serial (COM) ports
device uart # Generic UART driver

# Needed to actually boot from an MFS root
device md # Memory “disks”

This will do the bare minimum and give you a functioning system, at least it does for me on my 2core Atom box.  There’s little to no debugging enabled here, so if you chose to go this route remember that all of the facilities to debug are deactivated.

Allow me a short digression here.  One should NEVER create their own kernel configs from scratch and not derive them from one of the base kernel configs (GENERIC for example).  When you hand craft a kernel config, you allow yourself to get into situations where new options and devices can be added or removed.  This can lead to scenarios where your systems can be unusable and cost you many hours of hair pulling and gnashing of teeth.  It is far simpler to include a base configuration and unset the devices and options you don’t want.  Judicious use of noptions, nodevice and nomakeoptions can be used to remove things that you don’t want.  Use them, please.

Anywho, back to the task at hand.

FreeBSD has shipped a directory to disk image creation tool for quite some time.  makefs(8) does a great job of turning a directory tree into a disk image without all the the magical hand waving of creating a file backed md(4) device.  Its a shortcut and you should use it.  Embedded folks have been using it to boot up their MIPS and ARM devices for a while now.

We’ll need to install the build somewhere that we can then access with makefs.  I tend to just put it in a throw-away directory in /var/tmp, e.g. /var/tmp/netboot_mfs

So, something like ‘make installworld DESTDIR=/var/tmp/netboot_mfs’ should do the trick here.  At this point, we can modify this installed directory with whatever configuration changes needed for your system boot up.  At a minimum, I modify /boot/loader.conf and /boot.config.  These may not be needed for your environment, but I make the following changes:



You can now build your disk image:
makefs -M 192m /var/tmp/mfs_root.img /var/tmp/netboot_mfs
Calculated size of `mfs_root.img’: 201326592 bytes, 3800 inodes
Extent size set to 8192
mfs_root.img: 192.0MB (393216 sectors) block size 8192, fragment size 1024
using 4 cylinder groups of 54.38MB, 6960 blks, 1152 inodes.
super-block backups (for fsck -b #) at:
32, 111392, 222752, 334112,
Populating `mfs_root.img’
Image `mfs_root.img’ complete

I add the “-M 192m” to the command line here to give my image a bit of space to do stuff and things on the running system without running out of space.  makefs(8) will create an image big enough to fit the files regardless of the command line arguments, but the -M will “pad” the image to 192MB if its smaller.

Go ahead and compress your diskimage with either bzip2 or gzip. I won’t judge.

-rw-r–r– 1 root sbruno 35235255 Feb 7 16:14 mfs_root.img.bz2
-rw-r–r– 1 root sbruno 39516593 Feb 7 16:17 mfs_root.img.gz
(use bzip2).

Don’t think that I haven’t forgotten about the kernel either.  We’ll compress that beastie directly.  One does not install it however, one does not need it, one does not simply install a kernel into Mordor.  Recall that we set a MAKEOBJDIRPREFIX when doing out kernel and world builds.

/var/tmp/sbruno % find . -name kernel

In this example, I’ve built both so you kind of have an idea of what you’re looking for, it can be a bit strange to *not* install a kernel, but as we are fond of saying these days, EVERYTHING IS FINE. 

You’re also going to need the pxeboot binary.  On 12-CURRENT nowadays you don’t have to recompile it for TFTP installations as it’ll do the right thing depending on DHCP options.

/var/tmp/sbruno % find . -name pxeboot

Speaking of DHCP and TFTP, we should probably set those up now.  Remember when I linked you to The Outstanding FreeBSD Handbook?  It has the correct setting for the setup of inetd(8) and tftp(8) on your sever.  Ignore all the NFS bits, those don’t apply.

Install your dhcp server, you can use the same one from The Outstanding FreeBSD Handbook as I did.

The dhcpd.conf settings are a little different here.  We don’t need to pass in a root-path option, but need a custom dhcp option code 150:

option tftp-server code 150 = { ip-address };

subnet netmask {
option routers;
option domain-name-servers;
host router {
hardware ethernet xx:xx:xx:xx:xx:xx;
option tftp-server;
filename “/boot/pxeboot”;

Now we can populate /tftpboot with the files we’ve created and add *one* last configuration file, this time with FORTH to ensure we’ve done the right thing.

/tftpboot/boot # ls -l /tftpboot/boot
total 34844
drwxr-xr-x 2 root wheel 3 Feb 7 16:45 kernel
-rw-r–r– 1 root wheel 21 Feb 7 12:45 loader.conf
-r–r–r– 1 root wheel 174 Feb 7 09:47 loader.rc
-rw-r–r– 1 root wheel 35234122 Feb 7 12:04 netboot.img.bz2
-r–r–r– 1 root wheel 337920 Feb 7 08:34 pxeboot

/tftpboot/boot # ls -l /tftpboot/boot/kernel
total 2564
-rwxr-xr-x 1 root wheel 2565979 Feb 7 15:36 kernel.bz2

echo Loading kernel
load /boot/kernel/kernel
echo Loading md_image …
load -t md_image /boot/netboot.img
set vfs.root.mountfrom=”ufs:/dev/md0″
set console=”comconsole”

Here dear readers, is where I leave you.  If you’ve done all the right things … you should end up booting up into your newly built MFS root.  If you haven’t done all the right things or I’ve omitted something, then the Nazgul will find you.


#FreeBSD Vim port updated … ewwwww

I just ran across a system with ports/vim-lite updated to version 8. I now know why a coworker of mine, in the past, said using vim was like “running your fingers down a chalkboard, but for your eyes.”

In order to make vim usable for what I do, I asked the friendly folks in #bsdports on EFNet for suggestions. They suggested the following for my use, but your mileage may vary:

set bg=dark
set nohlsearch
set mouse=v

This seemed to turn off some of the new search features that people like in Vim 8 that caught me off guard. Hope this helps folks if you’re like me and were caught off guard by the whole thing.


Sometimes less is more. Checkout individual #FreeBSD ports for testing.

I needed to test a single FreeBSD port today for a specific release.  The FreeBSD project maintains a series of reference build machines for this type of work, but normally I maintain an entire ports tree checkout for this type of work.

The FreeBSD ports tree has 20k+ individual software packages maintained in it, and I was not in the mood to checkout all of that onto a host for the purpose of validating a build of QEMU on FreeBSD 10.

Turns out, if you pay attention, you can indeed do some individual checkouts of things in a sparse fashion and get results that are useful.  Your mileage may vary here depending on the needed dependencies, but this should be a useful hint to getting started.

I will only address how to do this with Subversion, but I’m sure that the folks who use git will have no trouble seeing how to duplicate this scenario.

  • svn co –depth files svn+ssh:// fbsd_ports
  • cd fbsd_ports
  • svn co svn+ssh://
  • svn co svn+ssh://
  • svn co –depth files svn+ssh://
  • cd lang
  • svn co svn+ssh://
  • svn co svn+ssh://
  • cd ..
  • svn co –depth files svn+ssh://
  • cd emulators
  • svn co svn+ssh://
  • svn co svn+ssh://

My goal, was to build qemu-user-static for FreeBSD 10.  There’s a bunch of dependencies here, so I’ll try and unwind them a bit.

In order to build any port, you need the files contained in the top of tree.  The support Makefiles in Mk/ and Templates/ are required to do a lot of things and may or may not be used as short cuts/helpers in the Makefiles of individual ports.

The Makefile of any directory that you traverse to get to your port will need to exist (emulators/Makefile for example)

The lang/perl and lang/python ports are needed here as there is some pollution in the Makefiles that requires something to be parsed during builds.  This probably shouldn’t be there, but is beyond the scope of what I wanted to get done this morning.

Checking out the individual ports into their proper locations and setting PORTSDIR in your environment to your new directory structure (/home/sbruno/fbsd_ports) will now allow you to use your new checkout in some fashion to build.


The EEPROM you have reached is no longer in service, #FreeBSD recovery of em(4) devices

Ran into a very strange error with legacy em(4) devices on FreeBSD this week.  Something I was doing while working on the EM_MULTIQUEUE kernel config option for em(4) seems to have blown up the PXE configuration in the EEPROM of one of my lem(4) devices.  Else, cosmic rays did it:

em3: <Intel(R) PRO/1000 Legacy Network Connection 1.1.0> port 0xe880-0xe8bf mem 0xfeb80000-0xfeb9ffff,0xfeb60000-0xfeb7ffff irq 19 at device 6.0 on pci5
em3: The EEPROM Checksum Is Not Valid
device_attach: em3 attach returned 5

It took me a while, but it seems when this happens, you can sometimes recover the device with Intel’s Ethernet Connections Boot Utility.

There is a DOS utility in this self-extracting CAB file called “bootutil” that can be used to reset your device to defaults.  After prepping a DOS usb stick, I booted my host from it and ran the tool against all devices as I couldn’t quite tell which device was actually having the issues:

bootutil -all -defcfg

This cleared the error and now I get a successful attach.  I literally have no idea what the problem was in the first place, but solved.

Props to the IPXE folks for having a great HOWTO on generically manipulating the firmware flash that gave me the clue on what to do.


The Short List #8: fetchmailrc/gmail/ssl … grrr #FreeBSD

Didn’t realize that a fetchmail implementation I was using was actually *not* using SSL for a month.  I had installed security/ca_root_nss but FreeBSD doesn’t assume that you want to use the certificates in this package.  I don’t understand it, but whatever.

So, add this to your fetchmailrc to actually use the certificate authorities in there and really do SSL to your gmail account:

sslcertfile /usr/local/share/certs/ca-root-nss.crt


The Short List #7: net-im/finch and password protected IRC channels on #FreeBSD

I discovered recently that net-im/finch can indeed join password protected IRC channels even though the channel add dialogue box doesn’t support it.

Add an IRC chat room that requires a password to your buddy list.

Exit out of finch and edit ~/.purple/blist.xml

Find the chat room you just added:

<chat proto=’prpl-irc’ account=’’>
<component name=’channel’>#supersekritircchannel</component>
<setting name=’gnt-autojoin’ type=’bool’>1</setting>

Simply add the password line after the channel name component:

<component name=’password’>supersekritpass</component>

Restart finch and you will now be able to auto-join the IRC channel.

Note that the password for this IRC channel is now in PLAIN TEXT in a file in your home directory.  So, ensure that you are doing this somewhere trustworthy.


Using the xdev target with qemu-user-static on #FreeBSD

I’ve been playing with building ports for ARM on an AMD64 machine via a bunch of tools.  The duct tape and bailing wire is a bit thick with this method, but if you keep at it, this should work.

1. build armv6 chroot:
make buildworld TARGET=arm TARGET_ARCH=armv6
make installworld TARGET=arm TARGET_ARCH=armv6 DESTDIR=/armv6
make distribution TARGET=arm TARGET_ARCH=armv6 DESTDIR=/armv6

2. build xdev
make xdev TARGET=arm TARGET_ARCH=armv6 NOSHARED=y

3. move xdev into chroot
mv /usr/armv6-freebsd /armv6/usr/

4. add toolchain to make.conf:
5. Install qemu-static-user from ports and copy into jail:
pkg instlal qemu-static-user
mkdir -p /armv6/usr/local/bin
cp /usr/local/bin/qemu-arm /armv6/usr/local/bin/

6. setup binmiscctl to handle armv6 translations:
binmiscctl add armv6 –interpreter “/usr/local/bin/qemu-arm” –magic “x7fx45x4cx46x01x01x01x00x00x00x00x00x00x00x00x00x02x00x28x00” –mask “xffxffxffxffxffxffxffx00xffxffxffxffxffxffxffxffxfexffxffxff” –size 20 –set-enabled

7. mount devfs and ports if needed
mount -t devfs devfs /armv6/dev
mount -t nullfs /usr/ports /armv6/usr/ports

8. chroot
chroot /armv6

Embedded FreeBSD

Using qemu-user to chroot and bootstrap other architectures on #FreeBSD

My last post spawned enough feedback that I thought I would dump some notes here for those interested in building a chroot on FreeBSD that allows you to test and prototype architectures, e.g. ARMv6 on AMD64.

The FreeBSD buildsys has many targets used for many things, the two we care about here are buildworld and distribution.  We will also be changing the output architecture through the use of TARGET and TARGET_ARCH command line variables.  I’ll assume csh is your shell here, just for simplicity.  You’ll need 10stable or 11current to do this, as it requires the binary activator via binmiscctl(8) which has not appeared in a release version of FreeBSD yet.

Checkout the FreeBSD source tree somewhere, your home directory will be fine and start a buildworld.  This will take a while, so get a cup of tea and relax.

make -s -j <number of cpus on your machine> buildworld TARGET=mips TARGET_ARCH=mips64 MAKEOBJDIRPREFIX=/var/tmp

Some valid combinations of TARGET/TARGET_ARCH are:









Once this is done, you have an installable tree in /var/tmp.  You need to be root for the next few steps, su now and execute these steps:

make -s installworld TARGET=mips TARGET_ARCH=mips64 MAKEOBJDIRPREFIX=/var/tmp DESTDIR=/opt/test

DESTDIR is where you intend on placing the installed FreeBSD system.  I chose /opt/test here only because I wanted to be FAR away from anything in my running system.  Just to be clear here, this will crush and destroy your host computer without DESTDIR set.

Next, there are some tweaks that have to be done by the buildsys, so run this command as root:

make -s distribution TARGET=mips TARGET_ARCH=mips64 MAKEOBJDIRPREFIX=/var/tmp DESTDIR=/opt/test

Now we need to install the emulator tools (QEMU) to allow us to use the chroot on our system.  I suggest using emulators/qemu-user-static for this as Juergen Lock has set it up for exactly this purpose.  It will install only the tools you need here.

Once that is installed, via pkg or ports, setup your binary activator module for the architecture of your chroot.  Use the listed options on the QEMU user mode wiki page for the architecture you want.  I know the arguments are not straight forward, but there should be examples for the target that you are looking for.

For this mips/mips64 example:

binmiscctl add mips64elf –interpreter “/usr/local/bin/qemu-mips64-static”
–magic “x7fx45x4cx46x02x02x01x00x00x00x00x00x00x00x00x00x00x02x00x08”
–mask “xffxffxffxffxffxffxffx00xffxffxffxffxffxffxffxffxffxfexffxff”
–size 20 –set-enabled

Copy the binary qemu that you setup in this step *into* the chroot environment:

mkdir -p /opt/tmp/usr/local/bin

cp /usr/local/bin/qemu-mips64-static /opt/tmp/usr/local/bin/

Mount devfs into the chroot:

mount -t devfs devfs /opt/tmp/dev

Want to try building ports in your chroot?  Mount the ports tree in via nullfs:

mkdir /opt/tmp/usr/ports

mount -t nullfs /usr/ports /opt/tmp/usr/ports

And now, through the QEMU and FreeBSD, you can simply chroot into the environment:

chroot /opt/tmp

Hopefully, you can now “do” things as though you were running on a MIPS64 or whatever architecture machine you have as a target.

arm:armv6, mips:mips, mips:mips64 are working at about %80-90 functionality.  powerpc:powerpc64 and powerpc:powerpc are still a work in progress and need more work.  sparc64:sparc64 immediately aborts and probably needs someone with an eye familiar with the architecture to give QEMU a look.  If you are interested in further development of the qemu-user targets, please see my github repo and clone away.

If you are looking to see what needs to be done, Stacey Son has kept an excellent log of open item on the FreeBSD Wiki


Cross building ports with qemu-user and poudriere-devel on #FreeBSD

I’ve spent the last few months banging though the bits and pieces of the work that Stacey Son implemented for QEMU to allow us to more or less chroot into a foreign architecture as though it were a normal chroot.  This has opened up a lot of opportunities to bootstrap the non-x86 architectures on FreeBSD.

Before I get started, I’d like to thank Stacey Son, Ed Maste, Juergen Lock, Peter Wemm, Justin Hibbits, Alexander Kabaev, Baptiste Daroussin and Bryan Drewery for the group effort in getting us the point of working ARMv6, MIPS32 and MIPS64 builds.  This has been a group effort for sure.

This will require a 10stable or 11current machine, as this uses Stacey’s binary activator patch to redirect execution of binaries through QEMU depending on the ELF header of the file.  See binmiscctl(8) for more details.

Mechanically, this is a pretty easy setup.  You’ll need to install ports-mgmt/poudriere-devel with the qemu-user option selected.  This will pull in the qemu-user code to emulate the environment we need to get things going.

I’ll pretend that you want an ARMv6 environment here.  This is suitable to build packages for the Rasberry PI and Beagle Bone Black.  Run this as root:

binmiscctl add armv6 –interpreter “/usr/local/bin/qemu-arm” –magic
x00x28x00″ –mask “xffxffxffxffxffxffxffx00xffxffxffxff
xffxffxffxffxfexffxffxff” –size 20 –set-enabled

This magic will load the imgact_binmisc.ko kernel module.  The rest of the command line instructs the kernel to redirect execution though /usr/local/bin/qemu-arm if the ELF header of the file matches an ARMv6 signature.

Build your poudriere jail (remember to install poudriere-devel for now as it has not been moved to stable at the time of this writing) with the following command:

poudriere jail -c -j 11armv632 -m svn -a armv6 -v head

Once this is done, you will be able to start a package build via poudriere bulk as you normally would:

poudriere bulk -j 11armv632 -a


poudriere bulk -j 11armv632 -f <my_file_of_ports_to_build>

Currently, we are running the first builds in the FreeBSD project to determine what needs the most attention first.  Hopefully, soon we’ll have something that looks like a coherent package set for non-x86 architectures.

For more information, work in progress things and possible bugs in qemu-user code, see Stacey’s list of things at:


Playing nice with others. git(1) and patches on #FreeBSD

I’ve been spending a lot of time massaging a branch of patches and other assorted bits and pieces for QEMU user mode on github

This led me down the path of being a good git user and contributor, so I’ll leave these notes for myself and others in the event you come into a situation where you need FreeBSD to play nice with people who are very git(1) centric.

After an update by to the devel/git port, you can now install git(1) and have it work out of the box.  The most frustrating thing, after using git for like 5 minutes, is to figure out how to extract a patch out of it and send it all pretty-like to the mailing list(s) that would be consuming the patch.

In its simplest incarnation, you can simply reference a commit hash and us it to generate a patch via git format-patch, but this will give you the entire commit diff between the referenced version and HEAD.  This, in my case generated approximately 3000 patch files.

e.g. git format-patch –output-directory ~/patches –to=”” c60a1f1b2823a4937535ecb97ddf21d06cfd3d3b

What I want, is a diff of one revision, which requires a start and ending hash:

format-patch –output-directory ~/patches –to=”” c60a1f1b2823a4937535ecb97ddf21d06cfd3d3b…c6ad44bb288c1fe85d4695b6a48d89823823552b

Now I send this to the mailing lists via my client.  Here is where I kind of head-desked a bit.  If you are like me and run a mail server yourself and you use SSL with self-signed certs, then this little bit if for you.  I lost about an hour trying to figure this little bit out.

The way to dump patches from your patch director (~/patches) is to use:

git send-email patches/*

This will use the following variables in your git environment:

Notice the empty “sendemail.smtpcertpath” variable.  Without that set to EMPTY, git would repeatedly fail on the self-signed cert that I use.  So, I’m pretty sure something still isn’t setup correctly.  However, it must be set to EMPTY and not undefined.  Else, you will repeatedly fail with certificate validation errors.