Romain Tartière's blog
ZFS: unsupported ZFS version 14 (should be 13)
How I got there?
After updating to the latest development version of GNOME thanks' to MarcusCom ports I wanted to log out and back in but X refused to restart because of the in-heavy-development Nouveau video driver. I run in this problem every once a while and a full reboot solve this problem. However, the system did not boot:
ZFS: unsupported ZFS version 14 (should be 13) ZFS: unsupported ZFS version 14 (should be 13) NO ZFS pools located? can't boot
Okay, I recall running the following to update my full-ZFS system from ZFS 13 to 14 a while ago:
zpool upgrade -a
I completly forgot about updating the GPT bootloader accordingly and thus it was unable to use the version of ZFS on my disks.
The state of Mono 2.6 on FreeBSD
Long time no post... Well, I an writing a serie of articles about NFC and it takes me a lot of time, not mentioning I still do many other things in parallel. If you are interested in this area, stay tuned ;-)
So apart NFC, what's new with Mono on FreeBSD? First, you wight have seen that mono 2.6 has been released for a while and is still not available in FreeBSD ports. The main reasons are:
Unbreaking Mono on FreeBSD 6.4
A few days ago ports/140916: lang/mono (2.4.2.3) installation fails was opened. The reported problem had already been reported a few time but with insufficient feedback so far and I was not able to diagnose the problem, not mentioning providing a fix. But this time, the reporter could spot that the problem was happening for him only with FreeBSD-6.4. I did the test on a fresh FreeBSD-6.4 install and could trigger the error:
arthur# portsnap fetch [...] arthur# portsnap extract [...] arthur# cd /usr/ports/lang/mono arthur# make -V PKGNAME mono-2.4.2.3_1 arthur# make [...] if test -w ../mcs; then :; else chmod -R +w ../mcs; fi cd ../mcs && gmake NO_DIR_CHECK=1 PROFILES='net_1_1 net_2_0 net_3_5 net_2_1' CC='cc' all-profiles gmake[3]: Entering directory `/usr/ports/lang/mono/work/mono-2.4.2.3/mcs' gmake profile-do--net_1_1--all profile-do--net_2_0--all profile-do--net_3_5--all profile-do--net_2_1--all gmake[4]: Entering directory `/usr/ports/lang/mono/work/mono-2.4.2.3/mcs' gmake PROFILE=basic all gmake[5]: Entering directory `/usr/ports/lang/mono/work/mono-2.4.2.3/mcs' gmake[6]: *** [build/deps/basic-profile-check.exe] Error 1 gmake[6]: Entering directory `/usr/ports/lang/mono/work/mono-2.4.2.3/mcs' *** The compiler 'false' doesn't appear to be usable. *** Trying the 'monolite' directory. gmake[7]: Entering directory `/usr/ports/lang/mono/work/mono-2.4.2.3/mcs' gmake[8]: *** [build/deps/basic-profile-check.exe] Error 138 gmake[8]: Entering directory `/usr/ports/lang/mono/work/mono-2.4.2.3/mcs' *** The contents of your 'monolite' directory may be out-of-date *** You may want to try 'make get-monolite-latest' gmake[8]: *** [do-profile-check-monolite] Error 1 gmake[8]: Leaving directory `/usr/ports/lang/mono/work/mono-2.4.2.3/mcs' gmake[7]: *** [do-profile-check] Error 2 gmake[7]: Leaving directory `/usr/ports/lang/mono/work/mono-2.4.2.3/mcs' gmake[6]: *** [do-profile-check-monolite] Error 2 gmake[6]: Leaving directory `/usr/ports/lang/mono/work/mono-2.4.2.3/mcs' gmake[5]: *** [do-profile-check] Error 2 gmake[5]: Leaving directory `/usr/ports/lang/mono/work/mono-2.4.2.3/mcs' gmake[4]: *** [profile-do--basic--all] Error 2 gmake[4]: Leaving directory `/usr/ports/lang/mono/work/mono-2.4.2.3/mcs' gmake[3]: *** [profiles-do--all] Error 2 gmake[3]: Leaving directory `/usr/ports/lang/mono/work/mono-2.4.2.3/mcs' gmake[2]: *** [all-local] Error 2 gmake[2]: Leaving directory `/usr/ports/lang/mono/work/mono-2.4.2.3/runtime' gmake[1]: *** [all-recursive] Error 1 gmake[1]: Leaving directory `/usr/ports/lang/mono/work/mono-2.4.2.3' gmake: *** [all] Error 2 *** Error code 1 Stop in /usr/ports/lang/mono. *** Error code 1 Stop in /usr/ports/lang/mono.
OpenRD console access on FreeBSD.
This morning, I received the Open-RD I bought a few days ago and started playing with my new device. One of the first thing I wanted to access was (of course) the system console. While the default setup provides a SSH daemon and even GDM and a full desktop, my goal is to have FreeBSD on this device and move the services I run at home from my personal computer to this low-consumption computer (and I don't intend to switch to GNU/Linux).
Since the console is usually available on the serial port, I looked for some cable to connect the OpenRD to my computer. Unfortunately, I didn't have the required cable. I then spent about two hour with various serial cables, cables looking like serial cables but which where not serial cables, parallel SUB-D25 connectors, a serial SUB-D25 cross-over cable and a soldering iron to build a null-modem SUB-D9 serial cable. Unfortunately, when I connected my computer to the OpenRD, I was still not able to access the console using cu(1).
After reading a bit of documentation, I realised I missed the fact that the console was available on USB and not via the serial port. However, FreeBSD did not recognised the chip the OpenRD uses to provide serial over USB support.
Using usbconfig(8), I could get the vendor ID and product ID the uftdi(4) driver was lacking. I patches the source code and a few minutes later could enjoy a set of new devices when I plug the OpenRD on my computer:
% ls /dev/cuaU* /dev/cuaU0 /dev/cuaU0.init /dev/cuaU0.lock /dev/cuaU1 /dev/cuaU1.init /dev/cuaU1.lock
I can now access the OpenRD console using cu(1):
% cu -l /dev/cuaU1 -s 115200
\o/ I can boot a FreeBSD 8.0-STABLE arm kernel!
I filled-in the usb/140951 problem report in the FreeBSD GNATS with the patch. If you own an OpenRD and want to have access to the console, the patch is just waiting for you.
Introducing ZFS support in portshaker(8)
portshaker(8) is a tool designed for merging partial ports trees into the FreeBSD ports tree. In other words, it implements some kind of overlay for the FreeBSD ports.
When merging port trees, portshaker(8) first clones the upstream FreeBSD ports tree to the target location. For this purpose, portshaker(8) used to rely on rsync(1) because the target ports tree was supposed to be quite near to the source ports tree. With an UFS file-system, it took my computer circa 10 minutes to merge the 390 Mib of files.
I recently switched to full ZFS, and this gave the clone process a boost: less than four minutes where required to perform the clone action for the first time, and thanks to caching, a single minute was enought to clone a second time.
But ZFS provides cloning capabilities, thus, I added support for the ZFS file-system into portshaker(8). Enabling this feature is as simple as adding the following to /usr/local/etc/portshaker.conf:
use_zfs="yes"
There is however a prerequisite for this to work: the target ports tree's parent directory and the source directory have to be ZFS filesystems. For example, on my computer, the FreeBSD upstream ports are stored in /var/cache/portshaker/freebsd and I merge my ports to the usual /usr/ports directory. All these directories are ZFS filesystems:
romain@marvin ~ % zfs list /usr/ports /var/cache/portshaker /var/cache/portshaker/freebsd NAME USED AVAIL REFER MOUNTPOINT data/usr/ports 57,1M 80,9G 402M /usr/ports data/var/cache/portshaker 514M 80,9G 48,6M /var/cache/portshaker data/var/cache/portshaker/freebsd 427M 80,9G 390M /var/cache/portshaker/freebsd
When merging, portshaker(8) snapshots the FreeBSD ports and clone them to the target ports tree. This requires only a few seconds to achieve.
Banshee: Like Frankenstein! It rips!
A while ago (I mean when it landed in the FreeBSD ports tree), Banshee reacted when a music CD-ROM was in a CD-Drive: it crashed. Well, not so cool in fact, you had to double-check it was empty before launching the player...
More recently, I was happy to find out that it was not the case anymore: Banshee was simply unable to detect the CD-ROM anymore. Still not so cool: you were still unable to listen to an audio CD, not mentioning ripping it.
For some reason I cannot really figure out (I have spent quite some time in fixing Banshee for FreeBSD and giving it some love), I took a look at this problem.
Obviously, the first rusty part of the chain of tools involved with audio CD was MusicBrainz and more precisely musicbrainz-sharp (a client library for .NET / Mono). FreeBSD having it's own ioctls for managing CD drives, and musicbrainz-sharp having 3 concrete classes for managing such devices called DiscLinux, DiscWin32 and DiscWin32NT, a piece was definitively missing. With a great deal of inspiration, it was called DiscFreeBSD and provides bindings for almost everything referenced in sys/cdio.h. When instanced, this class reads the CD in the drive and computes an ID used to fetch information about the audio CD from the Internet.
Unfortunately, I could not test the library on a system one which it is supposed to work (read GNU/Linux) so I asked a friend of mine, Baptiste (Yes! It works!) [edit: 2009-11-10:12:04 he just told me he has a a-bit-better website where he is playing now...], to tell me how it goes and to help me make the library behave the same on GNU/Linux and FreeBSD. We quickly had some good results:
Yay! Tracks from Sopor Aeternus' album Les fleurs du Mal are being listed!
Unfortunately, it was not possible to listen to the CD nor rip it... Not so cool once more.
And here takes place the magic of GStreamer!
cd /usr/ports/audio/gstreamer-plugins-cdparanoia && make install
Now, Banshee can play and RIP audio CDs with a single mouse clic!
Playing Cinema Strange's The Astonished Eyes Of Evening...
And ripping it (progress in the bottom left corner)...
All this will be soon in your ports tree!
For those who can't wait, patches for musicbrainz-shap are in the project's trac: Add support for FreeBSD in MusicBrainzSharp.
Packing structures with Mono or how to fix alignment problems for P/Invoke
This is basically a[nother] note for myself.
I have just spent a while wondering why the size of some data structures in C# was wrong while writing bindings to the CD-ROM control features of the FreeBSD libc.
Considering the following code snippet:
struct msf {
public byte unused;
public byte minute;
public byte second;
public byte frame;
};
[StructLayout (LayoutKind.Explicit)]
struct msf_lba {
[FieldOffset (0)] public msf msf;
[FieldOffset (0)] public int lba;
[MarshalAs (UnmanagedType.ByValArray, SizeConst = 4)]
[FieldOffset (0)] public byte [] addr;
};
struct cd_toc_entry {
byte unused1;
public byte control_addr_type;
public byte track;
byte unused2;
public msf_lba addr;
};
struct cd_sub_channel_media_catalog {
public byte data_format;
public int mc_valid;
[MarshalAs (UnmanagedType.ByValArray, SizeConst = 15)]
public byte [] mc_number;
};
We can compute the size of each structure (MSDN page about types size):
- msf
- A msf structure contains 4 members each of 1 byte long so the whole structure is 4 bytes;
- msf_lba
- This is not more than the union of a msf structure (4 bytes), an int (4 bytes) and an array of 4 byte (4 bytes), so it's 4 bytes long too;
- cd_toc_entry
- A cd_toc_entry structure is composed of an unused byte (1 byte), 2 data byte (1 byte + 1 byte), another unused byte (1 byte) and a msf_lba structure (4 bytes) so it is 1 + 1 + 1 + 1 + 4 = 8 bytes long;
- cd_sub_channel_media_catalog
- A cd_sub_channel_media_catalog structure starts with 1 byte (1 byte), followed by an int (4 bytes), and an array of 15 byte (15 bytes). The whole structure is therefore 1 + 4 + 15 = 20 bytes long.
These results can be checked using Marshal.SizeOf:
Marshal.SizeOf (typeof (msf)) = 4 Marshal.SizeOf (typeof (msf_lba)) = 4 Marshal.SizeOf (typeof (cd_toc_entry)) = 8 Marshal.SizeOf (typeof (cd_sub_channel_media_catalog)) = 24
So the cd_sub_channel_media_catalog structure it 4 bytes long more that expected. This really sounds like alignment problems in C. A quick search on the Interop with Native Libraries page on Novell's wiki does not give any info about such a problem. Switching the two first fields order fix the size (but of course breaks the workability) so it is definitively an alignment problem that takes place here.
After some search, I could find the StructLayoutAttribute.Pack Field (actually I could not find this piece of information using the MSDN search engine, I had to search on the web similar problems and find this one with a reference to « Pack » to find this page in the Microsoft .NET documentation).
Finally, the solution is to write this:
[StructLayout (LayoutKind.Sequential, Pack = 1)]
struct cd_sub_channel_media_catalog {
public byte data_format;
public int mc_valid;
[MarshalAs (UnmanagedType.ByValArray, SizeConst = 15)]
public byte [] mc_number;
};
The MusicBrainzSharp port to FreeBSD can go on!
Updating FreeBSD 7 (i386) to 8 (amd64)
Goal
Update to the latest FreeBSD version, switch to 64 bit, and switch to full ZFS. In short, use the best I can have (or what is supposed to be the best I can have) on my computer.
Install an amd64 system
While cross compiling an amd64 system from i386 should be possible, I wanted to get sure that the nouveau video driver actually worked on amd64. I have so decided to install a fresh FreeBSD 8 on my system and perform a few tests before updating.
At that time, FreeBSD 8-BETA2 was just available, so I downloaded and installed it on a fresh partition created in place of the mysterious free space at the end of my hard disk:
# mkdir goinfre/FreeBSD # cd goinfre/FreeBSD # mkdir base kernels manpages # ftp -a ftp.freebsd.org ftp> cd /pub/FreeBSD/releases/amd64/8.0-BETA2/base ftp> lcd base ftp> mget * ftp> cd ../kernels ftp> lcd ../kernels ftp> mget * ftp> cd ../manpages ftp> lcd ../manpages ftp> mget * ftp> bye # ( cd base && DESTDIR=/goinfre sh install.sh ) # ( cd kernels && DESTDIR=/goinfre sh install.sh generic ) # mv /goinfre/boot/GENERIC /goinfre/boot/kernel # ( cd manpages && DESTDIR=/goinfre sh install.sh ) # bsdlabel -w -B -b /goinfre/boot/boot /dev/ad10s3
Since my hardware BIOS sucks when it is supposed to give up control of USB to the kernel, USB is disabled in my BIOS and I have no keyboard to choose from which slice to boot when the system starts. I have so to instruct the boot manager to boot on the third slice on next boot:
# boot0cfg -s 3 ad10
I then rebooted and setup FreeBSD 8-BETA2 networking, patched the kernel sources, checked that the nouveau video driver worked as expected, installed packages, and so on.
Setting up the disks
I basically followed instructions from the blog post Setting up a zfs-only system by Ulf Lilleengen (nanolulf), adjusting settings according to my system (4 GB swap, all available space for the ZFS partition):
# gpart delete -i 1 ad12
# gpart destroy ad12
# gpart create -s GPT ad12
# gpart add -b 34 -s 128 -t freebsd-boot ad12
# echo "4 * 1024*1024*1024 / 512" | bc
8388608
# gpart add -b 162 -s 8388608 -t freebsd-swap ad12
# gpart show ad12
=> 34 625142381 ad12 GPT (298G)
34 128 1 freebsd-boot (64K)
162 8388608 2 freebsd-swap (4.0G)
8388770 616753645 - free - (294G)
# gpart add -b 8388770 -s 616753645 -t freebsd-zfs ad12
# gpart bootcode -b /boot/pmbr ad12
# gpart bootcode -p /boot/gptzfsboot -i 1 ad12
Then I created the various ZFS filesystems and extracted FreeBSD 8 stable (well, still named 8.0-RC1 actually)
# zpool create data /dev/ad12p3 # zfs create data/tmp # zfs create data/usr # zfs create data/var
Installing FreeBSD 8 from sources
Since I had the right branch of the FreeBSD subversion repository checked in into the temporary FreeBSD 8 system — with local patches — I copied it to the ZFS filesystem and updated it.
# cp -R /usr/src /data/usr # cd /data/usr/src # svn up # make buildworld buildkernel # make DESTDIR=/data installworld # make DESTDIR=/data installkernel # make DESTDIR=/data distribution
I then customezed /boot/loader.conf:
cat << EOT > /data/boot/loader.conf nouveau_load=yes zfs_load=yes vfs.root.mountfrom="zfs:data" EOT
Configured the swap device:
cat << EOT > /data/etc/fstab # Device Mountpoint FStype Options Dump Pass# /dev/ad12p2 none swap sw 0 0 EOT
Customised new /etc/make.conf:
cat << EOT > /data/etc/make.conf LOADER_ZFS_SUPPORT=YES EOT
Setup the network in /etc/rc.conf:
cat << EOT > /data/etc/rc.conf # This file now contains just the overrides from /etc/defaults/rc.conf. # Please make all changes to this file, not to /etc/defaults/rc.conf. # Enable network daemons for user convenience. ifconfig_rl0="inet 192.168.63.42 netmask 255.255.255.0" defaultrouter="192.168.63.1" hostname="marvin.blogreen.org" named_enable="YES" # asterisk_enable="YES" # dbus_enable="YES" # hald_enable="YES" # gdm_enable="YES" # gdm_lang="fr_FR.UTF-8" EOT
I finally finished the ZFS setup:
# zfs export data && zfs import data # cp /boot/zfs/zpool.cache /data/boot/zfs # zfs set mountpoint=legacy data # zfs set mountpoint=/usr data/usr # zfs set mountpoint=/var data/var # zpool set bootfs=data data
I then booted the fresh install from ZFS and finished the system configuration:
# passwd # tzsetup # adduser
Mirroring data
Since I have two identical disks on my system and wanted to have some kind of redundancy for a while, I set-up the second disk as a mirror of the first one.
First, I prepared the old disk so it has the same structure as the new with the ZFS filesystem:
# gpart show ad10
=> 63 625142385 ad10 MBR (298G)
63 104856192 1 freebsd (50G)
104856255 419425020 2 freebsd (200G)
524281275 100856070 3 freebsd (48G)
625137345 5103 - free - (2.5M)
# gpart delete -i 1 ad10
# gpart delete -i 2 ad10
# gpart delete -i 3 ad10
# gpart destroy ad10
# gpart create -s GPT ad10
# gpart add -b 34 -s 128 -t freebsd-boot ad10
# gpart add -b 162 -s 8388608 -t freebsd-swap ad10
# gpart show ad10
=> 34 625142381 ad10 GPT (298G)
34 128 1 freebsd-boot (64K)
162 8388608 2 freebsd-swap (4.0G)
8388770 616753645 - free - (294G)
# gpart add -b 8388770 -s 616753645 -t freebsd-zfs ad10
# gpart bootcode -b /boot/pmbr ad10
# gpart bootcode -p /boot/gptzfsboot -i 1 ad10
I added my new 4 Gib swap to /etc/fstab and enabled it:
# echo "/dev/ad10p2 none swap sw 0 0" >> /etc/fstab # swapon /dev/ad10p2
I could then tell ZFS to mirror ad10p3 and ad12p3:
# zpool attach data ad12p3 ad10p3 # zpool status pool: data state: ONLINE status: One or more devices is currently being resilvered. The pool will continue to function, possibly in a degraded state. action: Wait for the resilver to complete. scrub: resilver in progress for 0h2m, 0.35% done, 13h33m to go config: NAME STATE READ WRITE CKSUM data ONLINE 0 0 0 mirror ONLINE 0 0 0 ad12p3 ONLINE 0 0 0 44.6M resilvered ad10p3 ONLINE 0 0 0 754M resilvered errors: No known data errors
The nice part is of course that you can continue to use the system while you are doing all this!
LINQ, PostgreSQL and Mono
LINQ to DB is planned for mono-2.6, however I wanted to access data from a PostgreSQL database, without spending much time playing with Npgsql, and using the stable version of mono: 2.4.2.3.
Here comes DbLinq, which aims to provide linq to SQL
for many DataBase systems. While this project is still considered experimental to some extends by its developers, I wanted to give it a try. Since it way quite a PITA, here are the details for building using DbLinq with mono.
Get DbLinq from the Subversion repository
The latest stable release is quite old, and the project page warns we recommend using the latest SVN HEAD, so checkout the code from the google code repository:
svn checkout http://dblinq2007.googlecode.com/svn/trunk/ dblinq2007
Open the solution in MonoDevelop (or your favourite editor) and edit each project options so that assemblies are signed with the DbLinq.snk key.
You can then compile all needed assemblies and the DbMetal utility.
Get the System.Data.Linq.dll assembly
This assembly is not yet in the stable mono releases. You can compile it from the mono trunk:
svn checkout http://anonsvn.mono-project.com/source/trunk/mcs cd mcs && ./configure && gmake
The assembly for the 2.0 profile is then available as class/lib/net_2_0/System.Data.Linq.dll.
Start you project
Create a new project as usual and add any required references. For example, my project use PostgreSQL so I added the following assemblies:
- DbLinq.dll
- DbLinq.PostgreSQL.dll
- Npgsql.dll
- System.Data.dll
- System.Data.Linq.dll
Generate classes to access your DataBase
The DbMetal utility generate classes from a DataBase schema. Before using it, ensure that DbMetal.exe.config exists in the DbMetal.exe directory and if not, copy it from DbMetal/App.config. You can then use the utility to generate your code:
mono DbMetal.exe /conn:"server=localhost;user id=user;\ password=password; database=db" /provider=PostgreSQL \ /code:db.cs /pluralize
Add the generated db.cs to your project and start coding:
using System;
using Npgsql;
class MainClass
{
public static void Main(string[] args)
{
string connStr = "server=localhost;user id=user; password=password; database=db";
NpgsqlConnection conn = new NpgsqlConnection(connStr);
using (Db db = new Db(conn))
{
foreach (Employees e in db.Employees)
Console.WriteLine(e.Name);
}
}
}
1 year, 1 month, 1 week, 1 day
Oh No, Not Again 