The Short List #8: Using #lldb with a core file on #FreeBSD

Debugging qemu this evening and it took me a minute or two to figure out the syntax for debugging a core file with lldb.

lldb mips-bsd-user/qemu-mips -c /mipsbuild/qemu-mips.core

Make sure you have permissions to access both the binary and the core, else you get a super unhelpful error of:

error: Unable to find process plug-in for core file ‘/mipsbuild/qemu-mips.core’

But, after that, you can start poking around:

Core file ‘/mipsbuild/qemu-mips.core’ (x86_64) was loaded.

Process 0 stopped

* thread #1: tid = 0, 0x00000000601816fa qemu-mips`_kill + 10, name = ‘qemu-mips’, stop reason = signal SIGILL

frame #0: 0x00000000601816fa qemu-mips`_kill + 10

qemu-mips`_kill + 10:

-> 0x601816fa: jb 0x60182f5c ; .cerror

0x60181700: ret

0x60181701: nop

0x60181702: nop

(lldb) bt

* thread #1: tid = 0, 0x00000000601816fa qemu-mips`_kill + 10, name = ‘qemu-mips’, stop reason = signal SIGILL

* frame #0: 0x00000000601816fa qemu-mips`_kill + 10

frame #1: 0x000000006003753b qemu-mips`force_sig(target_sig=<unavailable>) + 283 at signal.c:352

frame #2: 0x00000000600376dc qemu-mips`queue_signal(env=<unavailable>, sig=4, info=0x00007ffffffe8878) + 380 at signal.c:395

frame #3: 0x0000000060035566 qemu-mips`cpu_loop [inlined] target_cpu_loop(env=<unavailable>) + 1266 at target_arch_cpu.h:239

frame #4: 0x0000000060035074 qemu-mips`cpu_loop(env=<unavailable>) + 20 at main.c:201

frame #5: 0x00000000600362ae qemu-mips`main(argc=1623883776, argv=0x00007fffffffd898) + 2542 at main.c:588

frame #6: 0x000000006000030f qemu-mips`_start + 367

Edit:  The permission error on the core file is now more meaningful in later versions of llvm:

Sometimes you have to sit down and write #FreeBSD documentation

When working on new projects or hacks, sometimes you just have to stop, think and start writing down your processes and discoveries. While working on bootstrapping the DLink DIR-825C1, I realized that I had accumulated a lot of new (to me) knowledge from the FreeBSD Community (namely, Adrian Chadd and Warner Losh).

There is a less than clear way of constructing images for these embedded devices that has an analogue in the Linux community under the OpenWRT project. Many of the processes are the same, but enough are different that I thought it wise to write down some of the processes into the beginning of a hacker’s guide to doing stuff and/or things in this space.

The first document I came up with was based on the idea that we can netboot these little devices without touching the on-board flash device. This is what you should use to get the machine bootstrapped and figure out where all the calibration data for the wireless adapters exist. This is crucial to not destroying your device. The wireless calibration data (ART) is unique to each device, destroying it will mean you have to RMA this device.

The second document I’ve created is a description of how to construct the flash device hints entries in the kernel hints file for FreeBSD. I found the kernel hints file to be cumbersome in comparison to the linux kernel way of using device specific C files for unique characteristics.

Its interesting stuff if you have the hankering to dig a bit deeper into systems that aren’t PC class machines.

The Short List #6: Make the CD drive do something useful on #FreeBSD

Noted today that while grip could access the CD drive on my machine, clemetine-player and xfburn could not.

Figure out which device node your CD drive is with camcontrol:

camcontrol devlist | grep cd
at scbus4 target 0 lun 0 (cd0,pass2)

Simply add the following to /etc/devfs.conf and restart devfs to get access to the CD device:

perm /dev/cd0 0666
perm /dev/xpt0 0666
perm /dev/pass2 0666

Now bear in mind, that this means any user of your machine has access to the device now. Hopefully, on a desktop computer, you know all the users of your machine.

Burning all the bridges. Cleaning up jails with ezjail-admin on #FreeBSD

I noted that my updates on my jail host didn’t actually do a delete-old/delete-old-libs during the basejail process:

ezjail-admin update -i

I tend to update my jails with my base host svn updates to -current, so there’s a bit of churn and burn with regards to old files and such. This came to a head today as my src.conf on the base host declares WITHOUT_NIS to conserve my limited space.

The python port checks for the existence of the yp binaries to determine whether or not to build NIS support. So, if the old binaries are lying around and support for NIS is removed from your system, python’s build will abort with something like the following:

Install them as needed.
====> Compressing man pages (compress-man)
===> Installing for python27-2.7.6_2
===> Checking if lang/python27 already installed
===> Registering installation for python27-2.7.6_2 as automatic
pkg-static: lstat(/var/ports/basejail/usr/ports/lang/python27/work/stage/usr/local/lib/python2.7/lib-dynload/ No such file or directory
*** Error code 74

I realized that even though my host system was fairly clean (I do port rebuilds after each upgrade and delete-old delete-old-libs following that), the basejail was still filled with obsoleted files.

A super dangerous and super effective way to clean that up is the following:
yes | make delete-old DESTDIR=/usr/jails/basejail
yes | make delete-old-libs DESTDIR=/usr/jails/basejail

Dangerous, because you have to realize that your deleting binaries and libraries that might still be in use if you haven’t recompiled your ports packages. Effective, because it will cleanup and purge a lot of things if you haven’t done it in a while.

This also led me to understand that the /etc/src.conf tuneables WITHOUT_* don’t *stop* the buildsystem from creating the binaries and libraries. It doesn’t seem to shorten your build time. It *will* allow you to purge them from your system at install time with the delete-old make targets.

httperf tuning for #FreeBSD testing

Was playing around with httperf to excercise Apache / stunnel SSl benchmarks on FreeBSD this week and ran into the code that nerfs simultaneous connections down from the environment ulimit of maxfiles to the limit FD_SETSIZE as defined in <select.h>.

One can override this at compile time and push the system harder by passing in some ./configure foo:

env CC=”cc -DFD_SETSIZE=4096″

However, you will then be able to max out the number of ports in use very quickly if you try to use stunnel and apache in this configuration.  I noted that on our systems we raise the low port number and reduce the high port number for connections:



I set first down to 2000 and last up to 65534 for my testing.  This gives me quite a bit more ports to use in testing.  At this point I can run stunnel on 443 forwarding to apache on localhost:80 and get more than 8k simultaneous connections when using SSL accelerators on FreeBSD 10


The short list #5: coredumping with sudo on #FreeBSD

Things I learned from a misbehaving pam module managing our sudo context at work.  sudo, for security, will not dump core files if it hits a segfault.  You need to tell the kernel to allow set uid root binaries to core dump *and* you have to let sudo know that its ok via a sudo.conf entry.


kern.sugid_coredump: 1

/etc/sudo.conf –> Set disable_coredump true

ref –>


You need to construct additional pylons. Building wine for #FreeBSD

I’ve been playing Blizzards’s Starcraft 2 on Linux via wine emulation lately and thought I’d see if I can get the same thing working on FreeBSD via the emulators/i386-wine-devel port.  After talking with the fine folks in #bsdports on EFNet, I finally found a recipe that is poudriere friendly and seems to spit out something that sort of works.

David Naylor ( has a working method for constructing wine on FreeBSD and this should work in most cases for using current.  The method is really designed for building a binary package for releases, most folks wouldn’t want to go down this route.

In order to begin, get poudriere configured and ready to go.  You’ll need to construct an i386 jail for the first part of this process.  Something like I show in my poudiere blog post

poudriere jail -c -j 11i386 -v head -a i386 -m svn

This will give you a build environment to get the 32bit binaries for wine built and packaged up for step 2.

poudriere builk -j 11i386 emulators/i386-wine-devel

If all goes well, you now have an i386 package of wine that will be consumed as a distfile for the amd64 package build.  I redefine PORTSDIR=/usr/local/poudriere/ports/default in /etc/make.conf.

If you are like me and use poudriere for everything, copy it to /usr/local/poudriere/ports/defaults/distfiles/freebsd:11:x86:64/

Now you’ll need to edit the emulators/i386-wine-devel distfile with the appropriate information generated from a sha256 and ls -l of your packagefile in your local i386 repo:

sha256 i386-wine-devel-1.7.7,1.txz

SHA256 (i386-wine-devel-1.7.7,1.txz) = 8d0073d1c10be9afbe7c3c9874a31ac110c1f96cf6ddcda74ca16d31bad55d1b

Modify this with the following to make it compatible with your system:

SHA256 (freebsd:11:x86:64/i386-wine-devel-1.7.7,1.txz) = 8d0073d1c10be9afbe7c3c9874a31ac110c1f96cf6ddcda74ca16d31bad55d1b

Modify the to exclude checks for the OS version:

—    (revision 335346)
+++    (working copy)
@@ -41,10 +41,10 @@

.include <>

-.if !(${OSVERSION} >= 803000 && ${OSVERSION} < 900000) && !(${OSVERSION} >= 901000 && ${OSVERSION} < 1000000)
-IGNORE=        binaries compiled for FreeBSD 8.3+ and 9.1+ only
+#.if !(${OSVERSION} >= 803000 && ${OSVERSION} < 900000) && !(${OSVERSION} >= 901000 && ${OSVERSION} < 1000000)
+#IGNORE=        binaries compiled for FreeBSD 8.3+ and 9.1+ only

RUN_DEPENDS+=   ${DATADIR}/gecko/wine_gecko-2.24-x86.msi:${PORTSDIR}/emulators/wine-gecko-devel

And now, you can try building the package in your *AMD64* poudriere build with:

poudriere bulk -j 11amd64 emulators/i386-wine-devel

If my instructions have succeeded, you now have a package suitable for installation on your amd64 machine that will now let you do wine things.

Now, I need to figure out what the Blizzard Network Installer is trying to do as it runs, self-updates and hangs.

Moving forward by going slightly backwards and to the right, UEFI booting on #FreeBSD

The FreeBSD Foundation has been working towards the future of booting in x86 and catching up to our friends in Linux-land by sponsoring work on a UEFI enabled boot loader.  This work was taken on by Benno Rice ( and Ed Maste (

So far, it appears that one can indeed boot FreeBSD as I will demonstrate on my Thinkpad T520.

Starting with the UEFI project branch, one must build a 64bit version of libstand in tree.

cd uefi/lib/libstand && make

Modify the makefile in sys/boot/amd64

Index: amd64/efi/Makefile
— amd64/efi/Makefile    (revision 258775)
+++ amd64/efi/Makefile    (working copy)
@@ -77,8 +77,8 @@
LIBEFI=        ${.OBJDIR}/../../efi/libefi/libefi.a
CFLAGS+=    -I${.CURDIR}/../../common

+DPADD=        ${LIBFICL} ${LIBEFI} ../../../../lib/libstand/libstand.a
+LDADD=        ${LIBFICL} ${LIBEFI} ../../../../lib/libstand/libstand.a

.include <>

Now you can build loader.efi and get it to link against the 64bit version of libstand:

cd sys/boot && make

UEFI will look for a FAT formatted partition with the “efi” signature on it.  FreeBSD’s gpart can create this partition for you, so do the following foo:

gpart create -s gpt da0

gpart add -t efi da0

gpart add -t freebsd-ufs da0

$ gpart show da0
=>     34  2013117  da0  GPT  (983M)
34   131072    1  efi  (64M)
131106  1882045    2  freebsd-ufs  (919M)

newfs -t msdosfs /dev/da0p1

newfs /dev/da0p2

Mount the fat formatted partition, create the EFI directory structure(this is mandatory) and copy the loader.efi binary into place as bootx64.efi

mount -t msdosfs /dev/da0p1 /mnt

mkdir -p /mnt/efi/boot

cp uefi/sys/boot/amd64

Because the kernel currently needs to be aware of the new style UEFI memory map, you can’t run stock -current in this configuration.  You’ll need to use a kernel from the projects/uefi branch when constructing your bootable device.  I used a 1G usb thumbdrive for this test, so mount the UFS partition and use it as a DESTDIR for your installworld/installkernel:

make -s buildworld

make -s buildkernel

mount /dev/da0p2 /mnt

DESTDIR=/mnt make -s installworld

DESTDIR=/mnt make -s installkernel

DESTDIR=/mnt make -s distribution

Setup an /etc/fstab on this stick:

/dev/da0p2             /               ufs     rw,              1       1

At this point, your USB disk is ready for its first booting attempt.



I have to toggle UEFI/Legacy BIOS mode in my laptop.  For your entertainment, here it is.  This has the convient side effect of not booting from my other disk devices in my laptop as they do not have the “proper” fat formatted EFI partition on them.  This actually yeilds a pretty quick boot to the loader.

Amazing!  It did!  Sort of.

Now we have the entertainment of trying to figure out how to get here to multiuser.





With a “show” we find out that the loader has selected the EFI partition “part6” as the boot device.  “lsdev” shows us all the partitions that we could boot from, but I have chosen well in this example and can easily see that the one I really want is tagged with a “(removable)”.

In this case executing a “set currdev=part7” sets up the loader to boot and executing “boot” will get this system into multiuser.

Many thanks to the folks at the FreeBSD Foundation for these initial steps into UEFI.  The project branch in subversion is publicly available and I highly encourage folks to engage the community to get this closer to production grade.

The Unusual Suspects, #FreeBSD Vendor Summit 2013

I was fortunate enough this year to be able to help the FreeBSD Foundation host the 2013 Fall Vendor Summit at my workplace, Yahoo.  Our facilities in Sunnyvale are very first class and I like to help out with my non-technical resources whenever possible (because, frankly, if you’ve seen my code, you would prefer it that way).

George Neville-Neil of the FreeBSD Project and FreeBSD Foundation had asked if Yahoo could host again this year and we agreed to a one day presentation and get together at the main campus.

Lots of folks who don’t normally go for conferences showed up to this invitation only event, and for once it felt like we had a strong showing.  I had booked a conference room for 55 people and we had close to 70 show up.  It was really close to bordering on overflow into the hallway at one point.

I think my biggest takeaways this year was the fact that “FreeBSD Doesn’t Have Visualization” is now just a myth and doesn’t really match reality.  The Bhyve project has taken a good direction and now can spin up other o/s instances, like Linux, via the ACPI framework implemented during the Google Summer of Code projects.  It was also very good to see VMWare and Google Compute folks showing up and asking for “what we need to help you folks support FreeBSD in our cloud things.”

Instead of the hallway track at normal conferences, we had the “back of the room on the floor” track this year where there was much debating over the validity of git as a FreeBSD source management tool.  The thing is, the project already exports FreeBSD SVN src to a self hosted git repo ( and a github instance (   The debate swirls around the archaic “email patches to mailing lists” mentality instead of the “send pull request” things that the git world now has.

Interesting point from this discussion, perhaps we should now take the time to assign people who are more involved to important sections of kernel and source code.  The FreeBSD ports system has direct maintainers and a system to timeout maintainers who are AFK.  The FreeBSD base system has a more liberal approach as any committer can and does commit to any aspect of the tree.  Its common practice to not do this without review, but its no a true formal review process.  This leads to some cases where patches go to mailing lists and never get picked up and reviewed.

Otherwise, a fine time was had and I certainly look forward to the next conference, AsiaBSDCon 2014.