Shell

Nov

Introducing ZFS support in portshaker(8)

portshaker(8) is a tool designed for merging partial ports trees into the FreeBSD ports tree. In other words, it implements some kind of overlay for the FreeBSD ports.

When merging port trees, portshaker(8) first clones the upstream FreeBSD ports tree to the target location. For this purpose, portshaker(8) used to rely on rsync(1) because the target ports tree was supposed to be quite near to the source ports tree. With an UFS file-system, it took my computer circa 10 minutes to merge the 390 Mib of files.

I recently switched to full ZFS, and this gave the clone process a boost: less than four minutes where required to perform the clone action for the first time, and thanks to caching, a single minute was enought to clone a second time.

But ZFS provides cloning capabilities, thus, I added support for the ZFS file-system into portshaker(8). Enabling this feature is as simple as adding the following to /usr/local/etc/portshaker.conf:

use_zfs="yes"

There is however a prerequisite for this to work: the target ports tree's parent directory and the source directory have to be ZFS filesystems. For example, on my computer, the FreeBSD upstream ports are stored in /var/cache/portshaker/freebsd and I merge my ports to the usual /usr/ports directory. All these directories are ZFS filesystems:

romain@marvin ~ % zfs list /usr/ports /var/cache/portshaker /var/cache/portshaker/freebsd
NAME                                USED  AVAIL  REFER  MOUNTPOINT
data/usr/ports                     57,1M  80,9G   402M  /usr/ports
data/var/cache/portshaker           514M  80,9G  48,6M  /var/cache/portshaker
data/var/cache/portshaker/freebsd   427M  80,9G   390M  /var/cache/portshaker/freebsd

When merging, portshaker(8) snapshots the FreeBSD ports and clone them to the target ports tree. This requires only a few seconds to achieve.

Apr

You, bash users, are driving me crazy!

This is a visceral message with a lot of emotion (I mean I am not thinking before writing) so I will exceptionally use my big voice:

WILL YOU EVER BASH SCRIPTS WRITERS STOP THIS INSANE MESS WITH THIS SHELL?

Details bellow... with some words in capital letters as proposed by [RFC2119].

Jul

Incremental backup from shell scripts

In the past days, I have been working on an incremental backup solution for my company dedicated servers. These servers hosts gigabytes of data spread between websites, documents repositories and tools.

For each of our machine, a complete backup consists in two actions:

a local copy with appropriate privileges of everything that cannot be safely hot-copied to a remote location (such as database clusters that have to be dumped to SQL files and system files that need root privileges to be read);
directories to save on the host are copied to a backup server.

The first part is performed by Backupninja, a very flexible tool that rely on many other programs to do its job. I particularly like it since it goes in the UNIX philosophy: do one thing and do it well. Basically, this first part was trivial to set up.