On the next day, I eventually realised that an hard disk was logically missing on the system and the ZFS mirror it was belonging to was working in a degraded mode. This disk holding a swap partition, the panic quite makes sense: some data was stored there and could not be paged-out anymore.

# zpool list
NAME   SIZE  ALLOC   FREE    CAP  DEDUP  HEALTH  ALTROOT
data   294G  68,2G   226G    23%  1.25x  DEGRADED  -
tank  1,81T   300G  1,52T    16%  1.06x  ONLINE  -

# zpool status data
  pool: data
 state: DEGRADED
status: One or more devices could not be opened.  Sufficient replicas exist for
        the pool to continue functioning in a degraded state.
action: Attach the missing device and online it using 'zpool online'.
   see: http://www.sun.com/msg/ZFS-8000-2Q
  scan: none requested
config:

NAME                                            STATE     READ WRITE CKSUM
data                                            DEGRADED     0     0     0
  mirror-0                                      DEGRADED     0     0     0
    gptid/36711e52-a69e-11de-8adf-0018f38af467  ONLINE       0     0     0
    15152536002702365387                        UNAVAIL      0     0     0  was /dev/gptid/602da1ae-c474-11de-960d-0008a14dbca1

errors: No known data errors

This disk already had problems in the past and I even had to improve sysutils/smartmontools periodic script to take into account SMART attributes that have been failing at some point in the past but have recovered since that time on that disk. This time, the disk is really dead, so no other choice than changing it. Hopefully, I have a brunch of spare heard disks on the shelve, so I took two 500 GB disks to replace the two 320 GB of the degraded ZFS pool.

I first replaced the broken disk with a new one, identified it using geom(8) and partitioned using basically the same settings I used when installing FreeBSD on full-ZFS:

# geom disk list ada1
Geom name: ada1
Providers:
1. Name: ada1
   Mediasize: 500107862016 (465G)
   Sectorsize: 512
   Mode: r2w2e3
   descr: ST3500418AS
   ident: (null)
   fwsectors: 63
   fwheads: 16

# geom part create -s GPT ada1
ada1 created
# geom part add -s 128 -t freebsd-boot ada1
ada1p1 added
# geom part add -s 4G -t freebsd-swap ada1
ada1p2 added
# geom part add -t freebsd-zfs ada1
ada1p3 added
# geom part show ada1
=>       34  976773101  ada1  GPT  (465G)
         34        128     1  freebsd-boot  (64k)
        162    8388608     2  freebsd-swap  (4.0G)
    8388770  968384365     3  freebsd-zfs  (461G)

# geom part bootcode -b /boot/pmbr -p /boot/gptzfsboot -i 1 ada1
bootcode written to ada1

I then replaced the unavailable ZFS partition with the new one:

# zpool replace data 15152536002702365387 ada1p3
Make sure to wait until resilver is done before rebooting.

If you boot from pool 'data', you may need to update
boot code on newly attached disk 'ada1p3'.

Assuming you use GPT partitioning and 'da0' is your new boot disk
you may use the following command:

gpart bootcode -b /boot/pmbr -p /boot/gptzfsboot -i 1 da0

… waited a few moments for resilvering to finish:

% zpool status data
  pool: data
 state: DEGRADED
status: One or more devices is currently being resilvered.  The pool will
	continue to function, possibly in a degraded state.
action: Wait for the resilver to complete.
  scan: resilver in progress since Mon Jan  2 19:26:44 2012
    330M scanned out of 68,2G at 4,07M/s, 4h44m to go
    329M resilvered, 0,47% done
config:

	NAME                                            STATE     READ WRITE CKSUM
	data                                            DEGRADED     0     0     0
	  mirror-0                                      DEGRADED     0     0     0
	    gptid/36711e52-a69e-11de-8adf-0018f38af467  ONLINE       0     0     0
	    replacing-1                                 UNAVAIL      0     0     0
	      15152536002702365387                      UNAVAIL      0     0     0  was /dev/gptid/602da1ae-c474-11de-960d-0008a14dbca1
	      ada1p3                                    ONLINE       0     0     0  (resilvering)

errors: No known data errors

… then shut the system down, replaced the working 320 GB disk by a new 500 GB one and booted into FreeBSD again:

% zpool status data
  pool: data
 state: DEGRADED
status: One or more devices could not be opened.  Sufficient replicas exist for
	the pool to continue functioning in a degraded state.
action: Attach the missing device and online it using 'zpool online'.
   see: http://www.sun.com/msg/ZFS-8000-2Q
  scan: resilvered 68,2G in 1h58m with 0 errors on Mon Jan  2 21:25:42 2012
config:

	NAME                     STATE     READ WRITE CKSUM
	data                     DEGRADED     0     0     0
	  mirror-0               DEGRADED     0     0     0
	  3635039039460500206    UNAVAIL      0     0     0  was /dev/gptid/36711e52-a69e-11de-8adf-0018f38af467
	    ada1p3               ONLINE       0     0     0

errors: No known data errors

Same story with the other disk:

geom part create -s GPT ada0
ada0 created
# geom part add -s 128 -t freebsd-boot ada0
ada0p1 added
# geom part add -s 4G -t freebsd-swap ada0
ada0p2 added
# geom part add -t freebsd-zfs ada0
ada0p3 added
# geom part show ada0
=>       34  976773101  ada0  GPT  (465G)
         34        128     1  freebsd-boot  (64k)
        162    8388608     2  freebsd-swap  (4.0G)
    8388770  968384365     3  freebsd-zfs  (461G)

# geom part bootcode -b /boot/pmbr -p /boot/gptzfsboot -i 1 ada0
bootcode written to ada0
# zpool replace data 3635039039460500206 ada0p3
Make sure to wait until resilver is done before rebooting.

If you boot from pool 'data', you may need to update
boot code on newly attached disk 'ada0p3'.

Assuming you use GPT partitioning and 'da0' is your new boot disk
you may use the following command:

	gpart bootcode -b /boot/pmbr -p /boot/gptzfsboot -i 1 da0

When done, I saw that the available space on the data zpool was still the same:

# zpool list
NAME   SIZE  ALLOC   FREE    CAP  DEDUP  HEALTH  ALTROOT
data   294G  65,3G   229G    22%  1.19x  ONLINE  -
tank  1,81T   300G  1,52T    16%  1.08x  ONLINE  -

This is due to the autoexpand property set to off by default and that should be set to on before replacing disks if this feature is desired.

# zpool set autoexpand=on data

Hopefully, it is possible to use the zpool(8)'s online command to make ZFS take into account the extra space available for the pools:

# zpool online -e data ada0p3

# zpool list
NAME   SIZE  ALLOC   FREE    CAP  DEDUP  HEALTH  ALTROOT
data   460G  65,3G   395G    14%  1.20x  ONLINE  -
tank  1,81T   300G  1,52T    16%  1.06x  ONLINE  -

Location of Feature Keys on the ST2030.

I finally decided to configure these keys with proper numbers instead of browsing my directory each time I want to make a call. But the phone only ships with a single paper sheet to hand-write the names corresponding to the configured numbers and my hand-writing is somewhat like what would produce a drunk cow with Creutzfeldt–Jakob disease. I browsed the Internet for a sheet template to fill-in, print and cut, but was unable to find some.

So, I took Inkscape and did one myself. For your convenience, I make it available as the original SVG file (so that you can fill-it before printing) and a PDF file for those who like hand-writing:

• Thomson ST2030 Feature Key Sheet Template SVG;
• Thomson ST2030 Feature Key Sheet Template PDF.

If you don't have a precise idea of which numbers you call often, you can extract this information from Asterisk's log file. In my case, my calls come from the internal context so I just did this to get the top-20 of the numbers I call:

% cat /var/log/asterisk/cdr-csv/Master.csv | grep 'internal' | cut -d, -f3 | \
    sed -e 's|+33|0|' | sort | uniq -c | sort -nr | head -n 20

The first line of the script was:

#!/bin/bash -x

Those who know me might imagine how my eyes were bleeding (the other may want to read my previous entry about writing portable shebang for shell scripts), but this shows another problem one may encounter when porting a shell script with an invalid shebang.

Let's assume the script has bashisms and we want to use bash to run it. According to my previous post, you may want to write this:

#!/usr/bin/env bash -x

Unfortunately, this won't work: env(1) will look for a binary names bash -x in the $PATH and fail:

env: bash -x: No such file or directory

There is however a simple solution: the set builtin(1). The right syntax is:

#!/usr/bin/env bash
set -x

Basically, you may not have gpg-agent(1) running. The man page gives a few pointer on how to run it, here is my way to do so, in ${HOME}/.zshenv:

if ! pgrep -U `id -u` gpg-agent >/dev/null 2>&1; then
    eval `gpg-agent --daemon --write-env-file "${HOME}/.gpg-agent-info"`
else
    eval `cat "${HOME}/.gpg-agent-info"`
    export GPG_AGENT_INFO
fi

Hope this will help someone :-)

After updating to the latest development version of GNOME thanks' to MarcusCom ports I wanted to log out and back in but X refused to restart because of the in-heavy-development Nouveau video driver. I run in this problem every once a while and a full reboot solve this problem. However, the system did not boot:

ZFS: unsupported ZFS version 14 (should be 13)
ZFS: unsupported ZFS version 14 (should be 13)
NO ZFS pools located? can't boot

Okay, I recall running the following to update my full-ZFS system from ZFS 13 to 14 a while ago:

zpool upgrade -a

I completly forgot about updating the GPT bootloader accordingly and thus it was unable to use the version of ZFS on my disks.

The FreeBSD 8.0-STABLE livefs

The FreeBSD projects provides snapshots of -STABLE version of the Operating System (I am running FreeBSD 8.0-STABLE) and the livefs disk contains the fixit environement which would allow me to solve my problems.

I so used my laptop to look for such an ISO image and burn it, unfortunately, the bootp and dns servers are run by the machine that was down. Hopefully, I had a web browser oppenned on this system and it had the address of google's servers in it's cache. I could search for my ISP DNS and add them to /etc/resolv.conf. I was then able to download the image and burn it.

Fixing

By default, the livefs will not have ZFS support unless you ask for it before starting. In other word, at the boot menu, Escape to loader prompt and type:

OK load zfs
OK boot

When the sysinstall(8) menu appears, select Fixit / CDROM/DVD. At the Fixit# prompt, you will have to search for your zfs pools, mount your filesystems, locate the new version of the bootloader and install it. This is basically a transcript of what I did (I previously described my setup in this blog):

Fixit# zpool import -f data # at this point I had to leave and return in the Fixit environement
                            # because by ZFS root was hiding the livefs root.
Fixit# zfs set mountpoint=/mnt data
Fixit# gpart bootcode -p /mnt/boot/gptzfsboot -i 1 ad10
Fixit# gpart bootcode -p /mnt/boot/gptzfsboot -i 1 ad12
Fixit# zfs set mountpoint=legacy data
Fixit# zfs umount -a
Fixit# logout

Note to myself: in the future do not forget to update the bootloader when updating my ZFS filesystems.

Since the console is usually available on the serial port, I looked for some cable to connect the OpenRD to my computer. Unfortunately, I didn't have the required cable. I then spent about two hour with various serial cables, cables looking like serial cables but which where not serial cables, parallel SUB-D25 connectors, a serial SUB-D25 cross-over cable and a soldering iron to build a null-modem SUB-D9 serial cable. Unfortunately, when I connected my computer to the OpenRD, I was still not able to access the console using cu(1).

After reading a bit of documentation, I realised I missed the fact that the console was available on USB and not via the serial port. However, FreeBSD did not recognised the chip the OpenRD uses to provide serial over USB support.

Using usbconfig(8), I could get the vendor ID and product ID the uftdi(4) driver was lacking. I patches the source code and a few minutes later could enjoy a set of new devices when I plug the OpenRD on my computer:

% ls /dev/cuaU*
/dev/cuaU0
/dev/cuaU0.init
/dev/cuaU0.lock
/dev/cuaU1
/dev/cuaU1.init
/dev/cuaU1.lock

I can now access the OpenRD console using cu(1):

% cu -l /dev/cuaU1 -s 115200

\o/ I can boot a FreeBSD 8.0-STABLE arm kernel!

I filled-in the usb/140951 problem report in the FreeBSD GNATS with the patch. If you own an OpenRD and want to have access to the console, the patch is just waiting for you.

When merging port trees, portshaker(8) first clones the upstream FreeBSD ports tree to the target location. For this purpose, portshaker(8) used to rely on rsync(1) because the target ports tree was supposed to be quite near to the source ports tree. With an UFS file-system, it took my computer circa 10 minutes to merge the 390 Mib of files.

I recently switched to full ZFS, and this gave the clone process a boost: less than four minutes where required to perform the clone action for the first time, and thanks to caching, a single minute was enought to clone a second time.

But ZFS provides cloning capabilities, thus, I added support for the ZFS file-system into portshaker(8). Enabling this feature is as simple as adding the following to /usr/local/etc/portshaker.conf:

use_zfs="yes"

There is however a prerequisite for this to work: the target ports tree's parent directory and the source directory have to be ZFS filesystems. For example, on my computer, the FreeBSD upstream ports are stored in /var/cache/portshaker/freebsd and I merge my ports to the usual /usr/ports directory. All these directories are ZFS filesystems:

romain@marvin ~ % zfs list /usr/ports /var/cache/portshaker /var/cache/portshaker/freebsd
NAME                                USED  AVAIL  REFER  MOUNTPOINT
data/usr/ports                     57,1M  80,9G   402M  /usr/ports
data/var/cache/portshaker           514M  80,9G  48,6M  /var/cache/portshaker
data/var/cache/portshaker/freebsd   427M  80,9G   390M  /var/cache/portshaker/freebsd

When merging, portshaker(8) snapshots the FreeBSD ports and clone them to the target ports tree. This requires only a few seconds to achieve.

Update to the latest FreeBSD version, switch to 64 bit, and switch to full ZFS. In short, use the best I can have (or what is supposed to be the best I can have) on my computer.

Install an amd64 system

While cross compiling an amd64 system from i386 should be possible, I wanted to get sure that the nouveau video driver actually worked on amd64. I have so decided to install a fresh FreeBSD 8 on my system and perform a few tests before updating.

At that time, FreeBSD 8-BETA2 was just available, so I downloaded and installed it on a fresh partition created in place of the mysterious free space at the end of my hard disk:

# mkdir goinfre/FreeBSD
# cd goinfre/FreeBSD
# mkdir base kernels manpages
# ftp -a ftp.freebsd.org
ftp> cd /pub/FreeBSD/releases/amd64/8.0-BETA2/base
ftp> lcd base
ftp> mget *
ftp> cd ../kernels
ftp> lcd ../kernels
ftp> mget *
ftp> cd ../manpages
ftp> lcd ../manpages
ftp> mget *
ftp> bye
# ( cd base && DESTDIR=/goinfre sh install.sh )
# ( cd kernels && DESTDIR=/goinfre sh install.sh generic )
# mv /goinfre/boot/GENERIC /goinfre/boot/kernel
# ( cd manpages && DESTDIR=/goinfre sh install.sh )
# bsdlabel -w -B -b /goinfre/boot/boot /dev/ad10s3

Since my hardware BIOS sucks when it is supposed to give up control of USB to the kernel, USB is disabled in my BIOS and I have no keyboard to choose from which slice to boot when the system starts. I have so to instruct the boot manager to boot on the third slice on next boot:

# boot0cfg -s 3 ad10

I then rebooted and setup FreeBSD 8-BETA2 networking, patched the kernel sources, checked that the nouveau video driver worked as expected, installed packages, and so on.

Setting up the disks

I basically followed instructions from the blog post Setting up a zfs-only system by Ulf Lilleengen (nanolulf), adjusting settings according to my system (4 GB swap, all available space for the ZFS partition):

# gpart delete -i 1 ad12
# gpart destroy ad12
# gpart create -s GPT ad12
# gpart add -b 34 -s 128 -t freebsd-boot ad12
# echo "4 * 1024*1024*1024 / 512" | bc
8388608
# gpart add -b 162 -s 8388608 -t freebsd-swap ad12
# gpart show ad12
=>       34  625142381  ad12  GPT  (298G)
         34        128     1  freebsd-boot  (64K)
        162    8388608     2  freebsd-swap  (4.0G)
    8388770  616753645        - free -  (294G)
# gpart add -b 8388770 -s 616753645 -t freebsd-zfs ad12
# gpart bootcode -b /boot/pmbr ad12
# gpart bootcode -p /boot/gptzfsboot -i 1 ad12

Then I created the various ZFS filesystems and extracted FreeBSD 8 stable (well, still named 8.0-RC1 actually)

# zpool create data /dev/ad12p3
# zfs create data/tmp
# zfs create data/usr
# zfs create data/var

Installing FreeBSD 8 from sources

Since I had the right branch of the FreeBSD subversion repository checked in into the temporary FreeBSD 8 system — with local patches — I copied it to the ZFS filesystem and updated it.

# cp -R /usr/src /data/usr
# cd /data/usr/src
# svn up
# make buildworld buildkernel
# make DESTDIR=/data installworld
# make DESTDIR=/data installkernel
# make DESTDIR=/data distribution

I then customezed /boot/loader.conf:

cat << EOT > /data/boot/loader.conf
nouveau_load=yes
zfs_load=yes
vfs.root.mountfrom="zfs:data"
EOT

Configured the swap device:

cat << EOT > /data/etc/fstab
# Device                Mountpoint      FStype  Options         Dump    Pass#
/dev/ad12p2             none            swap    sw              0       0
EOT

Customised new /etc/make.conf:

cat << EOT > /data/etc/make.conf
LOADER_ZFS_SUPPORT=YES
EOT

Setup the network in /etc/rc.conf:

cat << EOT > /data/etc/rc.conf
# This file now contains just the overrides from /etc/defaults/rc.conf.
# Please make all changes to this file, not to /etc/defaults/rc.conf.

# Enable network daemons for user convenience.
ifconfig_rl0="inet 192.168.63.42  netmask 255.255.255.0"
defaultrouter="192.168.63.1"
hostname="marvin.blogreen.org"

named_enable="YES"
# asterisk_enable="YES"
# dbus_enable="YES"
# hald_enable="YES"

# gdm_enable="YES"
# gdm_lang="fr_FR.UTF-8"
EOT

I finally finished the ZFS setup:

# zfs export data && zfs import data
# cp /boot/zfs/zpool.cache /data/boot/zfs
# zfs set mountpoint=legacy data
# zfs set mountpoint=/usr data/usr
# zfs set mountpoint=/var data/var
# zpool set bootfs=data data

I then booted the fresh install from ZFS and finished the system configuration:

# passwd
# tzsetup
# adduser

Mirroring data

Since I have two identical disks on my system and wanted to have some kind of redundancy for a while, I set-up the second disk as a mirror of the first one.

First, I prepared the old disk so it has the same structure as the new with the ZFS filesystem:

# gpart show ad10
=>       63  625142385  ad10  MBR  (298G)
         63  104856192     1  freebsd  (50G)
  104856255  419425020     2  freebsd  (200G)
  524281275  100856070     3  freebsd  (48G)
  625137345       5103        - free -  (2.5M)
# gpart delete -i 1 ad10
# gpart delete -i 2 ad10
# gpart delete -i 3 ad10
# gpart destroy ad10
# gpart create -s GPT ad10
# gpart add -b 34 -s 128 -t freebsd-boot ad10
# gpart add -b 162 -s 8388608 -t freebsd-swap ad10
# gpart show ad10
=>       34  625142381  ad10  GPT  (298G)
         34        128     1  freebsd-boot  (64K)
        162    8388608     2  freebsd-swap  (4.0G)
    8388770  616753645        - free -  (294G)
# gpart add -b 8388770 -s 616753645 -t freebsd-zfs ad10
# gpart bootcode -b /boot/pmbr ad10
# gpart bootcode -p /boot/gptzfsboot -i 1 ad10

I added my new 4 Gib swap to /etc/fstab and enabled it:

# echo "/dev/ad10p2             none            swap    sw              0       0" >> /etc/fstab
# swapon /dev/ad10p2

I could then tell ZFS to mirror ad10p3 and ad12p3:

# zpool attach data ad12p3 ad10p3
# zpool status
  pool: data
 state: ONLINE
status: One or more devices is currently being resilvered.  The pool will
	continue to function, possibly in a degraded state.
action: Wait for the resilver to complete.
 scrub: resilver in progress for 0h2m, 0.35% done, 13h33m to go
config:

	NAME        STATE     READ WRITE CKSUM
	data        ONLINE       0     0     0
	  mirror    ONLINE       0     0     0
	    ad12p3  ONLINE       0     0     0  44.6M resilvered
	    ad10p3  ONLINE       0     0     0  754M resilvered

errors: No known data errors

The nice part is of course that you can continue to use the system while you are doing all this!

While the targeted audience is definitively not the *BSD kernel hacker, and despite the fact that each issue (so far) is spotted on a BSD flavor; the diversity of the articles featured in the magazine will make any Free Software enthusiast find useful information.

I never wrote a review about the magazine before, mainly because a lack of time, and now might be a bit too late. At the time of writing, BSD Magazine is likely to be discontinued because of lower benefits than expected :-(.

However, BSD Magazine is not dead yet and can still be saved! How? Many ways:

• Buy one copy of BSD Magazine. If you can't find it, visit the BSD Magazine website and tell them. You can also order the latest issue from FreeBSD Mall with a discount;
• Subscribe to the print or electronic subscription;
• Spread the word. Speak about it on your website, blog, etc. Microblog about it (using the #savebsdmag hashtag).
• Advertise in BSD Magazine if you have something related to *BSD or FreeSoftware to sell.

What you used to do

1. plug your USB device in. The device automagically appears on the desktop and by default a file manager window shows its contents;
2. copy your files;
3. left-click the device and select Unmount Volume;
4. click Ok when a message dialog tells you that the volume cannot be unmounted :-s;
5. Choose one:
   • If you are a violent person,
     1. open a terminal;
     2. run killall -9 gam_server && umount /media/KINGSTON
     3. unplug your USB device.
   • If you are not a violent person,
     1. save your work;
     2. close your session;
     3. log-in into a console;
     4. run gnome-unmount --pseudonym KINGSTON
     5. unplug your USB device;
     6. log-out from the console and log-in to X.

The new way of doing it

1. plug your USB device in. The device automagically appears on the desktop and by default a file manager window shows its contents;
2. copy your files;
3. left-click the device and select Unmount Volume;
4. unplug your USB device :-D.

I'm not sure that FreeBSD is the platform of choice for beginners, but this problem was really irritating. Having it fixed may help people discovering FreeBSD not running away screaming I guess.