LXD Containers and FIDO Security Keys

With the rise of WebAuthn, I've had to figure out how expose my various FIDO security keys (YubiKey, Nitrokey, OnlyKey, SoloKeys, etc) to the LXD containers I use for web browsers.

The core of the solution is to expose the HIDRAW device that the security key is using to the LXD container — and to configure the device in the container to be owned by the user account who will use it. If you only have one such key plugged in, it's most likely using the /dev/hidraw0 device; and usually it's user 1000 who needs to use it. An LXD profile entry like the following allows such access:

config: {}
description: exposes FIDO devices
devices:
  hidraw0:
    required: false
    source: /dev/hidraw0
    type: unix-char
    uid: "1000"
name: fido
used_by: []

A profile like this can be created, configured, and applied to a container with the following commands:

$ lxc profile create fido
Profile fido created
$ lxc profile device add fido hidraw0 unix-char required=false source=/dev/hidraw0 uid=1000
$ lxc profile add mycontainer fido
Profile fido added to mycontainer

However, the exact HIDRAW device number that a particular security key uses is not stable, and may vary as you plug and unplug various keys (or other USB or Bluetooth devices). How do you tell which HIDRAW device is being used by a particular physical device? The simplest way is to print out the content of the uevent pseudo file in the sysfs filesystem corresponding to each HIDRAW device until you find the one you want. For example, this is what the entry for one of my SoloKeys looks like, at hidraw11:

$ cat /sys/class/hidraw/hidraw11/device/uevent
DRIVER=hid-generic
HID_ID=0003:00001209:0000BEEE
HID_NAME=SoloKeys Solo 2 Security Key
HID_PHYS=usb-0000:00:14.0-4/input1
HID_UNIQ=1234567890ABCDEF1234567890ABCDEF
MODALIAS=hid:b0003g0001v00001209p0000BEEE

You can also get similar information — without the specific device name, but with the general type of device, like FIDO_TOKEN — from the udevadm command:

$ udevadm info /dev/hidraw11
P: /devices/pci0000:00/0000:00:24.0/usb1/2-4/2-4:1.4/0003:1209:BEEE.0022/hidraw/hidraw11
N: hidraw11
L: 0
E: DEVPATH=/devices/pci0000:00/0000:00:24.0/usb1/2-4/2-4:1.4/0003:1209:BEEE.0022/hidraw/hidraw11
E: DEVNAME=/dev/hidraw11
E: MAJOR=232
E: MINOR=12
E: SUBSYSTEM=hidraw
E: USEC_INITIALIZED=123456789010
E: ID_FIDO_TOKEN=1
E: ID_SECURITY_TOKEN=1
E: ID_PATH=pci-0000:00:24.0-usb-0:4:1.4
E: ID_PATH_TAG=pci-0000_00_24_0-usb-0_4_1_4
E: ID_FOR_SEAT=hidraw-pci-0000_00_24_0-usb-0_4_1_4
E: TAGS=:uaccess:seat:snap_firefox_geckodriver:security-device:snap_firefox_firefox:
E: CURRENT_TAGS=:uaccess:seat:snap_firefox_geckodriver:security-device:snap_firefox_firefox:

Using the udevadm info and lxc profile device list and commands, you can write a simple script that checks each /dev/hidraw* device on your host system against the HIDRAW devices registered for a particular LXD profile, and add or remove HIDRAW devices dynamically to that profile to match the current FIDO devices you have plugged in. Here's such a script:

#!/bin/sh -eu
profile=${1:-fido}
existing=$(lxc profile device list $profile)

for dev_path in /dev/hidraw*; do
    dev_name=$(basename $dev_path)
    if udevadm info $dev_path | grep FIDO >/dev/null; then
        if ! echo "$existing" | egrep '^'$dev_name'$' >/dev/null; then
            lxc profile device add $profile $dev_name \
                unix-char required=false source=$dev_path uid=1000
        fi
    else
        if echo "$existing" | egrep '^'$dev_name'$' >/dev/null; then
            lxc profile device remove $profile $dev_name
        fi
    fi
done

echo done

You can run the script manually every time you plug in a new security key, to make sure the security key is registered at the right HIDRAW slot in your LXD profile — or you can add a custom udev rule file to run it automatically.

If you save the above script as /usr/local/bin/add-fido-hidraw-devices-to-lxc-profile.sh, you can then add the below file as /etc/udev/rules.d/75-fido.rules (replacing justin with the username of your daily user) to automatically run the script for several different brands of FIDO security keys:

# Nitrokey 3
SUBSYSTEM=="hidraw", KERNEL=="hidraw*", ATTRS{idVendor}=="20a0", ATTRS{idProduct}=="42b2", RUN+="/bin/su justin -c /usr/local/bin/add-fido-hidraw-devices-to-lxc-profile.sh"
# OnlyKey
SUBSYSTEM=="hidraw", KERNEL=="hidraw*", ATTRS{idVendor}=="1d50", ATTRS{idProduct}=="60fc", RUN+="/bin/su justin -c /usr/local/bin/add-fido-hidraw-devices-to-lxc-profile.sh"
# SoloKeys
SUBSYSTEM=="hidraw", KERNEL=="hidraw*", ATTRS{idVendor}=="1209", ATTRS{idProduct}=="5070|50b0|beee", RUN+="/bin/su justin -c /usr/local/bin/add-fido-hidraw-devices-to-lxc-profile.sh"
# Yubico YubiKey
SUBSYSTEM=="hidraw", KERNEL=="hidraw*", ATTRS{idVendor}=="1050", ATTRS{idProduct}=="0113|0114|0115|0116|0120|0121|0200|0402|0403|0406|0407|0410", RUN+="/bin/su justin -c /usr/local/bin/add-fido-hidraw-devices-to-lxc-profile.sh"

Run the sudo udevadm control --reload-rules and sudo udevadm trigger commands to reload your udev rule files and trigger them for your currently plugged-in devices. If you use a different brand of security key, you can probably find its vendor and product IDs in the libfido2 udev rules file (or you can figure it out from the output of the udevadm info command).

LXD Containers for Wayland GUI Apps

Having upgraded my home computers to Ubuntu 22.04, which features the latest version of LXD (5.5) via Snap, and using Wayland (via the Sway window manager), I spent some time working out how to run Wayland-native GUI apps in an LXD container. With the help of a few posts (Running X11 Software in LXD Containers, GUI Application via Wayland From Ubuntu LXD Container on Arch Linux Host, and Howto Use the Host's Wayland and XWayland Servers Inside Containers), I was able to get this working quite nicely.

Basic Profile

Most apps I tried, like LibreOffice or Eye of Gnome, worked with this basic LXD container profile (for Ubuntu 22.04 container images):

config: boot.autostart: false user.user-data: | #cloud-config write_files: - path: /usr/local/bin/mystartup.sh permissions: 0755 content: | #!/bin/sh uid=$(id -u) run_dir=/run/user/$uid mkdir -p $run_dir && chmod 700 $run_dir && chown $uid:$uid $run_dir ln -sf /mnt/wayland-socket $run_dir/wayland-0 - path: /usr/local/etc/mystartup.service content: | [Unit] After=local-fs.target [Service] Type=oneshot ExecStart=/usr/local/bin/mystartup.sh [Install] WantedBy=default.target runcmd: - mkdir -p /home/ubuntu/.config/systemd/user/default.target.wants - ln -s /usr/local/etc/mystartup.service /home/ubuntu/.config/systemd/user/default.target.wants/mystartup.service - ln -s /usr/local/etc/mystartup.service /home/ubuntu/.config/systemd/user/mystartup.service - chown -R ubuntu:ubuntu /home/ubuntu - echo 'export WAYLAND_DISPLAY=wayland-0' >> /home/ubuntu/.profile description: Basic Wayland Jammy devices: eth0: name: eth0 network: lxdbr0 type: nic root: path: / pool: default type: disk wayland-socket: bind: container connect: unix:/run/user/1000/wayland-1 listen: unix:/mnt/wayland-socket uid: 1000 gid: 1000 type: proxy

It binds the host's Wayland socket (/run/user/1000/wayland-1) to the container at /mnt/wayland-socket, via the wayland-socket device config. Via its cloud config user data, it sets up a systemd service in the container that will run when the ubuntu user logs in, and link the Wayland socket to its usual location in the container (/run/user/1000/wayland-0). This cloud config also adds the WAYLAND_DISPLAY variable to the ubuntu user's .profile, ensuring that Wayland-capable apps will try to access the Wayland socket at that location.

(Note that you may be using a different user ID or Wayland socket number on your own host; run ls /run/user/*/wayland-? to check. If so, change the connect: unix:/run/user/1000/wayland-1 line above to match the actual location of your Wayland socket.)

To set up a profile like this, save it as a file like wayland-basic.yml on the host. Create a new profile with the following command:

$ lxc profile create wayland-basic

And then update the profile with the file's content:

$ cat wayland-basic.yml | lxc profile edit wayland-basic

You can continue to edit the profile and update it with the same lxc profile edit command; LXD will apply your changes to existing containers which use the profile. You can view the latest version of the profile with the following command:

$ lxc profile show wayland-basic

With this profile set up, you can launch a new Ubuntu 22.04 container from it using the following command (the last argument, mycontainer, is the name to use for the new container):

$ lxc launch ubuntu:22.04 --profile wayland-basic mycontainer

Once launched, you can log into an interactive terminal session on the container as the ubuntu user with the following command:

$ lxc exec mycontainer -- sudo -u ubuntu -i

Once logged in, you can install apps into the container, like to install LibreOffice Writer (the LibreOffice alternative to Microsoft Word):

ubuntu@mycontainer:~$ sudo apt update ubuntu@mycontainer:~$ sudo apt install libreoffice-gtk3 libreoffice-writer

Then you can run the app, which should open up in a native Wayland window:

ubuntu@mycontainer:~$ libreoffice

Sharing Folders

This basic profile doesn't have access to the host's filesystem, however. To allow the container to access a specific directory on the host, run the following command on the host:

$ lxc config device add mycontainer mymount disk source=/home/me/Documents/myshare path=/home/ubuntu/mydir

This will mount the source directory from the host (/home/me/Documents/myshare) at the specified path in the container (/home/ubuntu/mydir). LXD's name for the device within the container will be mymount — you can use the device's name in combination with the container's own name to edit or remove the device; and you can mount additional directories if you give each mount device a unique name within the container.

Our basic profile allows only read access to the mounted directory within the container, however, as the directory will be mounted with the nobody user as its owner. To change the owner to the ubuntu user (so you can write to the directory from within the container), shut down the container, change the user ID mapping for its mounts, and then start the container back up again:

$ lxc stop mycontainer $ lxc config set mycontainer raw.idmap='both 1000 1000' $ lxc start mycontainer $ lxc exec mycontainer -- sudo -u ubuntu -i ubuntu@mycontainer:~$ ls -l mydir

The mounted mydir directory and its contents will now be owned by the ubuntu user, with full read and write access. (If you need to map a host user or group with an ID other than 1000 to the container's ubuntu user, you can do so with the uid and gid directives instead of the both directive; see the LXD idmap documentation for details.)

If you want to use these same settings for all containers that use the same profile, you can add these settings directly to the profile's config:

config: boot.autostart: false raw.idmap: both 1000 1000 user.user-data: | #cloud-config write_files: - path: /usr/local/bin/mystartup.sh permissions: 0755 content: | #!/bin/sh uid=$(id -u) run_dir=/run/user/$uid mkdir -p $run_dir && chmod 700 $run_dir && chown $uid:$uid $run_dir ln -sf /mnt/wayland-socket $run_dir/wayland-0 - path: /usr/local/etc/mystartup.service content: | [Unit] After=local-fs.target [Service] Type=oneshot ExecStart=/usr/local/bin/mystartup.sh [Install] WantedBy=default.target runcmd: - mkdir -p /home/ubuntu/.config/systemd/user/default.target.wants - ln -s /usr/local/etc/mystartup.service /home/ubuntu/.config/systemd/user/default.target.wants/mystartup.service - ln -s /usr/local/etc/mystartup.service /home/ubuntu/.config/systemd/user/mystartup.service - chown -R ubuntu:ubuntu /home/ubuntu - echo 'export WAYLAND_DISPLAY=wayland-0' >> /home/ubuntu/.profile description: Myshare Wayland Jammy devices: eth0: name: eth0 network: lxdbr0 type: nic mymount: source: /home/me/Documents/myshare path: /home/ubuntu/mydir type: disk root: path: / pool: default type: disk wayland-socket: bind: container connect: unix:/run/user/1000/wayland-1 listen: unix:/mnt/wayland-socket uid: 1000 gid: 1000 type: proxy

Launcher Script

When an LXD container is running, you don't have to log into it via a terminal session to launch an application in it — you can launch the application directly from the host. The following command will launch LibreOffice directly from the host:

$ lxc exec mycontainer -- sudo -u ubuntu -i libreoffice

So save the following as a shell script on the host (eg mycontainer-libreoffice.sh) and make it executable (eg chmod +x mycontainer-libreoffice.sh), and then you can simply run the script any time you want to launch libreoffice in mycontainer:

#/bin/sh lxc info mycontainer 2>/dev/null | grep RUNNING >/dev/null || (lxc start mycontainer; sleep 2) lxc exec mycontainer -- sudo -u ubuntu -i libreoffice

(Note that if you did not add the WAYLAND_DISPLAY variable to the user's .profile file, or if you added it to the user's .bashrc file instead of .profile, you'll need to include this variable in the launch command like this: lxc exec mycontainer -- sudo WAYLAND_DISPLAY=wayland-0 -u ubuntu -i libreoffice .)

AppArmor Issues

Some Wayland-capable GUI apps may fail to run inside an LXD container due to issues with the app's AppArmor profile; but you may be able to work-around it by adjusting the profile. One such app I've encountered is Evince.

A good way to check for AppArmor issues is by tailing the syslog, and filtering on its audit identifier, like with the following command:

$ journalctl -t audit -f

Access denied by AppArmor will look like this:

Aug 25 19:30:07 jp audit[99194]: AVC apparmor="DENIED" operation="connect" namespace="root//lxd-mycontainer_<var-snap-lxd-common-lxd>" profile="/usr/bin/evince" name="/mnt/wayland-socket" pid=99194 comm="evince" requested_mask="wr" denied_mask="wr" fsuid=1001000 ouid=1001000

In the case of Evince, I found I could work-around it by adjusting the container's own AppArmor profile for Evince. Run the following commands in the container to grant Evince read/write access to the Wayland socket:

ubuntu@mycontainer:~$ echo '/mnt/wayland-socket wr,' | sudo tee -a /etc/apparmor.d/local/usr.bin.evince ubuntu@mycontainer:~$ sudo apparmor_parser -r /etc/apparmor.d/usr.bin.evince

The first command adds a line to the user-managed additions of the Evince AppArmor policy (which is usually empty); the second command reloads the packaged version of the policy (a different file), which references the user-managed additions via an include statement.

Browser Quirks

Unfortunately, Firefox and Chromium don't work with the LXD-proxied Wayland socket (at least the Snap-packaged Ubuntu versions of Firefox and Chromium don't). But fortunately, they do work (mostly) when the Wayland socket is shared with them via disk mount.

If you create a new profile like the following, with a disk mount used to share the Wayland socket instead of a network proxy, you can keep the startup script the same as before:

config: boot.autostart: false raw.idmap: both 1000 1000 user.user-data: | #cloud-config write_files: - path: /usr/local/bin/mystartup.sh permissions: 0755 content: | #!/bin/sh uid=$(id -u) run_dir=/run/user/$uid mkdir -p $run_dir && chmod 700 $run_dir && chown $uid:$uid $run_dir ln -sf /mnt/wayland-socket $run_dir/wayland-0 - path: /usr/local/etc/mystartup.service content: | [Unit] After=local-fs.target [Service] Type=oneshot ExecStart=/usr/local/bin/mystartup.sh [Install] WantedBy=default.target runcmd: - mkdir -p /home/ubuntu/.config/systemd/user/default.target.wants - ln -s /usr/local/etc/mystartup.service /home/ubuntu/.config/systemd/user/default.target.wants/mystartup.service - ln -s /usr/local/etc/mystartup.service /home/ubuntu/.config/systemd/user/mystartup.service - chown -R ubuntu:ubuntu /home/ubuntu - echo 'export WAYLAND_DISPLAY=wayland-0' >> /home/ubuntu/.profile description: Browser Wayland Jammy devices: eth0: name: eth0 network: lxdbr0 type: nic root: path: / pool: default type: disk wayland-socket: source: /run/user/1000/wayland-1 path: /mnt/wayland-socket type: disk

Save this profile as a file like wayland-browser.yml. Create a new profile for it, and update the profile from the file's content:

$ lxc profile create wayland-browser $ cat wayland-browser.yml | lxc profile edit wayland-browser

Launch an Ubuntu 22.04 container with it, log into it, and install a browser:

$ lxc ubuntu:22.04 --profile wayland-basic myfirefox $ lxc exec myfirefox -- sudo -u ubuntu -i ubuntu@myfirefox:~$ sudo snap install firefox

Once installed, you should be able to start up the browser and have it open in a new Wayland window:

ubuntu@myfirefox:~$ firefox

Using a disk mount instead of a network proxy to share the Wayland socket seems much more flaky, however. I find that I'm not always able to start Firefox back up after quitting from it if I leave its LXD container running (especially if I put the computer to sleep in between quitting and starting again). Also, Firefox's "crash reporter" window, when it appears, seems to trigger a new crash, resulting in a continuous loop of crashes.

So now I always stop and restart the browser's LXD container before starting a new browser session (and I disable the crash reporter). This is what I use for my Firefox launcher script:

#/bin/sh lxc info myfirefox 2>/dev/null | grep STOPPED >/dev/null || lxc stop myfirefox lxc start myfirefox sleep 3 lxc exec myfirefox -- sudo MOZ_CRASHREPORTER_DISABLE=1 -u ubuntu -i firefox

And this for my Chromium launcher:

#/bin/sh lxc info mychromium 2>/dev/null | grep STOPPED >/dev/null || lxc stop mychromium lxc start mychromium sleep 3 lxc exec mychromium -- sudo -u ubuntu -i chromium --ozone-platform=wayland

Also, there are a few facets of the browsers that still don't work under this regime — in particular, open/save file dialogs don't appear when you try to download/upload files.

PulseAudio Output

To output audio from an LXD container, bind the host's PulseAudio socket (/run/user/1000/pulse/native) to the container at /mnt/pulse-socket, similar to the original Wayland socket:

config: boot.autostart: false raw.idmap: both 1000 1000 user.user-data: | #cloud-config write_files: - path: /usr/local/bin/mystartup.sh permissions: 0755 content: | #!/bin/sh uid=$(id -u) run_dir=/run/user/$uid mkdir -p $run_dir && chmod 700 $run_dir && chown $uid:$uid $run_dir ln -sf /mnt/wayland-socket $run_dir/wayland-0 mkdir -p $run_dir/pulse && chmod 700 $run_dir/pulse && chown $uid:$uid $run_dir/pulse ln -sf /mnt/pulse-socket $run_dir/pulse/native - path: /usr/local/etc/mystartup.service content: | [Unit] After=local-fs.target [Service] Type=oneshot ExecStart=/usr/local/bin/mystartup.sh [Install] WantedBy=default.target runcmd: - mkdir -p /home/ubuntu/.config/systemd/user/default.target.wants - ln -s /usr/local/etc/mystartup.service /home/ubuntu/.config/systemd/user/default.target.wants/mystartup.service - ln -s /usr/local/etc/mystartup.service /home/ubuntu/.config/systemd/user/mystartup.service - chown -R ubuntu:ubuntu /home/ubuntu - echo 'export WAYLAND_DISPLAY=wayland-0' >> /home/ubuntu/.profile description: Pulse Wayland Jammy devices: eth0: name: eth0 network: lxdbr0 type: nic root: path: / pool: default type: disk pulse-socket: bind: container connect: unix:/run/user/1000/pulse/native listen: unix:/mnt/pulse-socket uid: 1000 gid: 1000 type: proxy wayland-socket: source: /run/user/1000/wayland-1 path: /mnt/wayland-socket type: disk

Update the startup script to link the PulseAudio socket to its usual location in the container (/run/user/1000/pulse/native) when the ubuntu user logs in, just like we did for the Wayland socket. (Note that the mystartup.sh script's content from the cloud config of this profile is applied only when the container is first created, so you have to manually edit it in any containers that you've already created if you want to update them, too.)

Useful Commands

If you are just getting started with LXD containers, here are a few more useful commands that are good to know:

• lxc ls: Lists all LXD containers.
• lxc snapshot mycontainer mysnapshot: Creates a snapshot of mycontainer named mysnapshot.
• lxc restore mycontainer mysnapshot: Restores mycontainer to the mysnapshot snapshot.
• lxc delete mycontainer: Deletes mycontainer.
• lxc storage info default: Shows the space used and available in the default storage pool.
• lxc config show mycontainer: Shows the container-customized config settings for mycontainer.
• lxc config show mycontainer -e: Shows all config settings for mycontainer (including those inherited from its profiles).

How To Test OpenRC Services with Docker-Compose

Similar to how I abused Docker conceptually to test systemd services with docker-compose, I spent some time recently trying to do the same thing with OpenRC for Alpine Linux.

It basically requires the same steps as systemd. With the base 3.12 Alpine image, it's a matter of:

1. Install OpenRC
2. Optionally map /sys/fs/cgroup
3. Start up with /sbin/init
4. Run tests via docker exec

1. Install OpenRC

The base Alpine images don't include OpenRC, so you have to install it with apk. I do this in my Dockerfile:

FROM alpine:3.12 RUN apk add openrc CMD ["/sbin/init"]

2. Optionally map /sys/fs/cgroup

Unlike with systemd, I didn't have to set up any tmpfs mounts to get OpenRC services running. I also didn't have to map the /sys/fs/cgroup directory -- but if I didn't, I would get a bunch of cgroup-related error messages when starting and stopping services (although the services themselves still seemed to work fine). So I just went ahead and mapped the dir in my docker-compose.yml to avoid those error messages:

version: '3' services: my_test_container: build: . image: my_test_image volumes: - /sys/fs/cgroup:/sys/fs/cgroup:ro

3. Start up with /sbin/init

With the Alpine openrc package, the traditional /sbin/init startup command works to start OpenRC. I added CMD ["/sbin/init"] to my Dockerfile to start up with it, but you could instead add command: /sbin/init to the service in your docker-compose.yml file.

4. Run tests via docker exec

The above docker-compose.yml and Dockerfile will allow you to start up OpenRC in my_test_container with one command:

docker-compose up -d my_test_container

With OpenRC up and running, you can use a second command to execute a shell on the very same container to test it out:

docker-compose exec my_test_container /bin/sh

Or use exec to run other commands to test the services managed by OpenRC:

docker-compose exec my_test_container rc-status --servicelist

Cleaning up

The clean up steps with OpenRC are also basically the same as with systemd:

1. Stop the running container: docker-compose stop my_test_container
2. Remove the saved container state: docker-compose rm my_test_container
3. Remove the built image: docker image rm my_test_image

Testing Systemd Services on Arch, Fedora, and Friends

Following up on a previous post about how to test systemd services with docker-compose on Ubuntu, I spent some time recently trying to do the same thing with a few other Linux distributions. I was able to get the same tricks to work on these other distributions:

• Amazon Linux
• Arch
• CentOS
• Debian
• Fedora
• openSUSE
• RHEL

A few of those distros required an additional tweak, however.

One more tmpfs directory for Arch and Fedora

For Arch and Fedora, I had to do one more thing: add /tmp as a tmpfs mount.

So the docker-compose.yml file for those distros should look like this:

version: '3' services: my_test_container: build: . image: my_test_image tmpfs: - /run - /run/lock - /tmp volumes: - /sys/fs/cgroup:/sys/fs/cgroup:ro

Or when running as a regular docker command, start the container like this:

docker run \ --tmpfs /run --tmpfs /run/lock --tmpfs /tmp \ --volume /sys/fs/cgroup:/sys/fs/cgroup:ro \ --detach --rm \ --name my_test_container my_test_image

Different init script location for openSUSE

For openSUSE, the systemd init script is located at /usr/lib/systemd/systemd instead of just /lib/systemd/systemd. So the Dockerfile I used for it looks like this:

FROM opensuse/leap:15 RUN zypper install -y systemd CMD ["/usr/lib/systemd/systemd"]

Java Thread Dumps for Daemons With jstack

jstack is a really helpful utility that comes standard with most linux JDK versions. It allows you to generate java thread dumps in situations where kill -3 won't work. kill -3 (aka kill -QUIT) will dump a java process's threads to stderr — but this, of course, works only when you still have access to stderr. If you're running a java process as a daemon (like a jetty or tomcat or jboss etc server), stderr usually is inaccessible.

Fortunately, jstack allows you to generate thread dumps without needing to read stderr from the original java process. jstack will dump the threads to jstack's own stdout, which you can pipe to a file, or through a pager, or just send directly to your terminial. There a few tricks to using it, however, which don't seem to be documented anywhere:

1. Use the right jstack executable

jstack usually will work only if it came with the same exact JDK version as the target JVM process is running. Since a lot of servers end up having several different JVM versions installed on them, it's important to make sure that the version of jstack you're trying to use is the right one — the jstack executable at /usr/bin/jstack won't necessarily be correct. And since jstack doesn't accept a -version flag, it's pretty hard to tell which version /usr/bin/jstack actually is.

So the most reliable way to run jstack is from the bin directory of the JDK which you're using to run the target JVM process. On ubuntu, this usually will be a subdirectory of one of the JDKs in the /usr/lib/jvm directory (like /usr/lib/jvm/java-6-openjdk-amd64 for the 64-bit version of the java 6 JDK). In that case, you might run jstack like this (when the target JVM's process ID is 12345):

/usr/lib/jvm/java-6-openjdk-amd64/bin/jstack 12345

2. Run jstack as the same user as the target JVM

You need to run jstack as the same user as which the target JVM is running. For example, if you're running jetty as a user named jetty, (and the jetty process ID is 12345) use sudo to execute jstack as the jetty user:

sudo -u jetty jstack 12345

(I learned this trick from Michael Moser's jstack - the missing manual blog post — apparently jstack uses a named pipe to communicate with the target JVM process, and that pipe's permissions allow only the user who created the target JVM process to read or write the pipe.)

3. Try, try again

Sometimes, even if you do those first two things, jstack will still tell you to go get bent (or some other inscrutable error message of similar intent). I've found that if I just try running it again a couple of times, jstack magically will work on the second or third try.

4. Don't use -F

Even though jstack sometimes itself will suggest that you try -F (particularly if you've got a version mismatch between jstack and the target JVM), resist the temptation to "force" it. When you use jstack with the -F option, jstack will actually stop the target process (ie kill -STOP). Only use the -F option if your app is already good and hung (because it certainly will be once you use -F).

Fastest Ext4 Options

When I got a new external hard drive for storing mp3s, recorded TV shows, etc, I wanted to know how to set it up most optimally (using the ext4 filesystem). I ended up following Luca Spiller's Ext4 Options for a Media Drive for the most part, but with a few tweaks:

Formatting the Drive

After plugging in the hard drive and running sudo fdisk -l to check what name the OS had assigned it (/dev/sdb1), I formatted the drive with the following options: -m 0 to create no extra room for root (I don't intend to use it as a boot drive); and -L bb to assign it a label of bb (so I can reference it by label in my /etc/fstab):

sudo mkfs.ext4 -m 0 -L bb /dev/sdb1

Configuring the Drive Options

Then I updated my /etc/fstab configuration with an entry for the new drive. Beyond the filesystem permission options of user, rw, exec, and suid (the order of which is significant), and the noauto option (to ignore the drive when booting), I added some options that make writing data less safe — but faster. Use man mount to get a brief description of these and all other possible options:

LABEL=bb /mnt/bb ext4 user,rw,exec,suid,noauto,noatime,nobh,nobarrier,commit=60,data=writeback,journal_async_commit 0 0

If this was an internal hard drive, or one that I intended to have connected at all times, I would have skipped the permissions and noauto options (so as to use the default permissions and allow it to auto-mount at boot time), and would have just specified the performance options:

LABEL=bb /mnt/bb ext4 noatime,nobh,nobarrier,commit=60,data=writeback,journal_async_commit 0 2

Creating the Drive Mount Point

In /etc/fstab I had configured the drive's mount point as /mnt/bb, so I created it and set its owner to myself (so I could mount the drive as a regular user, since I had included the user option in the /etc/fstab config):

sudo mkdir /mnt/bb && sudo chown justin:justin /mnt/bb

Mounting the Drive

Now whenever I plug in the drive, I can mount it as a regular user; either via its label:

mount -L bb

Or via its mount point:

mount /mnt/bb

Lucid Windows with Xmonad

I just got a new "Bonobo" laptop from System 76 (which comes with Ubuntu 10.04 Lucid Lynx installed); and I set it up with xmonad, a nice clean, tiling window-manager. But instead of messing around with .xsession, I'm launching it via the "hybrid" method detailed on the xmonad haskell wiki. With this mechanism, I can still login to x with the default gnome desktop — but now I can also log in with xmonad; plus I can keep the gnome-panel etc that comes with the default ubuntu install. Also, this is pretty much the simplest way to set up xmonad on Lucid.

1. Install Xmonad

So the first thing is to install xmonad:

$ sudo apt-get xmonad

2. Create an Xmonad.start Script

Now crate a shell script to execute when you login with an xmonad session. It will simply set the WINDOW_MANAGER environment variable to xmonad, and then start up gnome in the regular fashion:

$ sudo echo '#!/bin/sh export WINDOW_MANAGER=xmonad gnome-session' > /usr/local/bin/xmonad.start $ sudo chmod +x /usr/local/bin/xmonad.start

3. Update Xmonad.desktop

Lucid has already created an xmonad.desktop entry in /usr/share/applications and /usr/share/xssessions for you; you just need to update them to run your xmonad.start script (instead of running xmonad directly):

$ sudo perl -pi -e 's/^Exec=xmonad$/Exec=xmonad.start/' /usr/share/{applications,xsessions}/xmonad.desktop

4. Set up Xmonad.hs

If you want some sort of dock displayed (like gnome-panel, dzen, xmobar, etc.) you need to set up a custom .xmonad/xmonad.hs configuration file; and make sure you include manageDocks in the manageHook, and avoidStruts in the layoutHook (and you need to import XMonad.Hooks.ManageDocks for this).

You can try out my xmonad.hs, or this simplified xmonad.hs:

import XMonad import XMonad.Hooks.ManageDocks import XMonad.Layout.Grid import XMonad.Layout.Master -- automatically manage docks (gnome-panel/dzen/xmobar/etc) myManageHook = manageDocks -- each layout is separated by ||| -- layout 1: grid w/ master (expand/contract by 3/100; master takes up 1/2 of screen) -- layout 2: standard full (master window fullscreen, hides dock) myLayoutHook = (avoidStruts $ mastered (3/100) (1/2) $ Grid) ||| Full main = xmonad $ defaultConfig { manageHook = myManageHook <+> manageHook defaultConfig, layoutHook = myLayoutHook }

After editing xmonad.hs, validate it with the following command:

$ xmonad --recompile

If you're already running xmonad, you can reload it with the new config by pressing <mod>-q.

5. Try it Out

You're all ready to try it out! Log out of your current x session (ie the Log Out... menu item in the gnome-panel), and on the login screen, select XMonad from the Sessions menu on the login screen's bottom panel:

If you're new to xmonad, follow the rest of the xmonad guided tour. If you screwed up something and xmonad won't start or appears to hang, hit <ctrl>-<shift>-<f1> to get to a terminal, login, fix xmonad.sh (with xmonad --recompile to check that it compiles), and then restart x with sudo service gdm restart.

Here's the finished product on my laptop:

On Donner and Blitzen

The first company I worked for was pretty much a Windows-only shop. They used Exchange for email, VSS for version control, and Visual Studio as an IDE. This was the first time I had ever used Windows, and I disliked it (although not as badly as I thought I would) — but I didn't really have a choice. Since 2004, though, I've been fortunate enough to be able to use Linux on my primary dev box at work. My current company (and all the ones I've ever worked for) still uses Exchange, so I've struggled with how to deal with this over the years.

Now Dasher! Now Dancer!

When I first started using Windows, I didn't know the difference between Outlook and Outlook Express (apparently the former is an Exchange client and the other is a generic SMTP/POP client), and so I used Outlook Express (because I wanted my mail faster, of course). After a few months I finally figured out why I couldn't "accept" meeting requests: I needed to use Outlook for that. So I switched to Outlook proper; and even after I moved my primary dev box to Linux, I still kept a Windows machine on the local Windows domain to keep using Outlook (and also to test IE, of course).

Now Prancer and Vixen!

But switching back and forth between my Windows and Linux boxes got old pretty quick. First I tried Eudora as a SMTP/POP client (which I had used on the old Mac OS before OS X). That was okay, but POP is pretty primitive and usually doesn't work well (it's hard to keep the local read/deleted states in sync with the server, so you usually just end up downloading everything locally and deleting it on the server — which means you better have a good local backup story).

On Comet! On Cupid!

Then I tried Evolution as an IMAP client, which works (at least with Exchange) much better than POP. You can keep mail in sync on the server, and even use Exchange's server-side folder structure. The two big things missing still, though, are contacts and meetings. Evolution is supposed to be able to integrate more tightly with Exchange (possibly including these two features) via Exchange's web interface, but I've never been able to get that to work. So I just used Exchange's web interface whenever I wanted to lookup a person in Exchange, or to schedule/respond-to a meeting.

On Donner

Eventually I got restless, and decided to try Thunderbird. I can't say that it's really any better or worse than Evolution, just somewhat different. It's what I use now, but it still doesn't have contacts and meetings (although for a while I got it little bit of contact integration by syncing it with the corporate LDAP server). But then one day I found Lightning (and you're welcome for taking so long to get to the point of this post).

And Blitzen! (aka Finally, the Point of this Post)

Lightning, an extension for Thunderbird, allows just enough integration with Exchange's meeting functionality to be really really useful to me. Lightning is mainly a calendaring extension, but it's also able to parse Exchange's meeting-request emails — it displays a button right in the email to add the meeting-request to your Lightning calendar. So now I finally have a way to organize my meeting schedule (other than to highlight the meeting-request emails in my inbox — plus in the past Exchange unhelpfully moved these emails to a separate Calendar folder if I used the Exchange web interface to accept the meeting).

Here are some pics of the Lightning "Today Pane" (displayed on the right-side of my normal email listings), and the full Lightning "Calendar" interface (which is pretty much exactly what you'd expect and need from a calendering UI). My apologies for the super-gay Thunderbird persona (I can't seem to login to the Personas website in Thunderbird, so for now I'm stuck with picking one of the "popular" personas).

Lightning "Today Pane"

Lightning "Calendar"

On 64-Bit Linux

The one gotcha with this (and kind of my motivation for writing this post) is that the version of Lightning from addons.mozilla.org doesn't work on 64-bit Linux. Fortunately, Ubuntu has this documented and taken care of on their ThunderbirdLightning Community Documentation page. To make it work, I just had to download the 64-bit lightning.xpi, and install it manually (via the Install... button in the Extensions tab of the Add-ons dialog, which you can get at via the Tools > Add-ons menu item). Now dash away, dash away, dash away all!

IE VPCs on Ubuntu

I love the fact that Microsoft makes these Internet Explorer Application Compatibility VPC Images available, but I hate the fact that they're so darn hard to setup on linux. Plus the last few times they've rev'd the images, they've packaged them in a way where the licensing breaks if you don't run them in Microsoft's VPC software (obviously available for Windows only). And of course, each of the different VPCs (for ie6, ie7, and ie8) use the same disk uuid, which doesn't play nicely with VirtualBox. So this is what I end up doing:

Initial Setup

1) Download the latest VPCs.

2) Install unrar to extract the images:

sudo apt-get install unrar

3) Extract each image:

unrar e -o+ IE6-XP-SP3.exe

(I also move and rename them to something convenient, like ie6.vhd.)

4) Install VirtualBox:

sudo apt-get install virtualbox-ose virtualbox-guest-additions

Rinse and Repeat

After 3 days or reboots of the image, these VPC images will stop working in VirtualBox, so if you're not careful you'll have to repeat these steps fairly often (fortunately, VirtualBox's snapshotting capabilities can help here). Keep massaging the scalp until you've produced a thick, luxurious lather:

5) Clone the vhds, to give them universally unique uuids (apparently Microsoft doesn't quite get the meaning of uuid). Also, this'll give you a pristine image when you need to repeat the process:

VBoxManage clonehd ie6.vhd ie6cloned.vhd

6) Fire up the VirtualBox ui:

virtualbox

(I actually usually do this via the desktop menubar: Applications > Accessories > VirtualBox OSE.)

7) Add each of the vhds (you can also do this via VBoxManage commands, but I'm too lazy to figure them out):

1. Click the New toolbar item. Click Next.
2. Enter a name (like ie6) and click Next.
3. Select the Base Memory Size (256 MB for ie6/7; 512 MB for ie8), click Next.
4. Select Use existing hard disk, click the little Folder icon.
5. Click the Add toolbar item.
6. Browse to the (cloned) .vhd file, select it, and click Open.
7. Select the vhd you just added, and click Select.
8. Click Next, then click Finish.

8) Run each image (select the image, click the Start toolbar item).

9) After XP boots, click No to the Windows Product Activation prompt, then ignore all the other whiny messages (resist the temptation to freaking close them already!), and select the Devices > Install Guest Additions... menu item from the image's menubar (the linux window's menubar, not XP's — to get out of XP, press the right-control key). Click the Continue Anyway when Windows tells you that installing this software will usher in the End Times. Don't reboot when it finishes — fix a few other things first.

10) Because I use the Dvorak keyboard layout, the first thing I do in XP is change the keyboard layout, via the Start > Control Panel > Regional and Language Options control panel; flip to the Languages tab, click Details...; then click Add..., select Dvorak, click OK, make it the Default, OK, OK, OK already!

11) Next I get rid of XP's screensaver, via the Start > Control Panel > Display Properties control panel. Flip to the Screen Saver tab, and select (None) from the Screen saver drop down (and click OK when you're done with the display settings). Also choose a larger screen size (Settings tab) and try to find a desktop background that doesn't make you want to puke.

12) Still ignoring XP's whimperings, open a command prompt (Start > All Programs > Accessories > Command Prompt), and enter:

D:\VBoxWindowsAdditions-x86.exe /extract /D=C:\Drivers

13) Now find one of those Hardware Update Wizard dialogs that XP keeps popping up. If you couldn't take it and closed them all, open Start > Administrative Tools > Computer Management, then click Device Manager, then double-click Ethernet Controller, and finally click the Reinstall Driver... button. Follow these steps:

1. Select the No, not this time radio-button, click Next.
2. Select the Install from a list or specific location (Advanced) radio buton, click Next.
3. Select the Search for the best driver in these locations radio-button, unselect the Search removable media check-box, select the Include this location in the search check-box, and type in C:\Drivers\x86\Network\AMD (or browse to it, if you're visually inclined).
4. Click Next, click Finish.

Now we're finished with non-ie stuff! Ignore any other Hardware Wizards or other Forces of Darkness that XP sends your way.

14) For ie7, run the Install the IE Explorer 7 Readiness Toolkit item on your desktop.

15) Finally, run ie! You should see some crappy Microsoft home page — this means you've got the networking stuff hooked up right. Select Tools > Internet Options, and click the Use blank button so you never have to see this again. Then flip to the Advanced tab, and unselect the Disable script debugging (Internet Explorer) check-box, so the debugger will come up whenever you run across a javascript error.

OK, now reboot Windows. You've only got one more boot, so don't ever restart again! Once you boot up, save a snapshot with VirtualBox so you can always go back to this happy place.

See Also

Lots of other people have already gone down this road; especially helpful are:

• Primeval Soup: IE8 in Ubuntu with VirtualBox
• Bites in the Margin: HOWTO: Use VirtualBox to setup an Internet Explorer testing machine
• Shape Shed: Testing with IE6, IE7 and IE8 on VirtualBox
• Ian Ma: Ubuntu Jaunty, VirtualBox, and IE6/7/8