April 12, 2008

Raid-1 using gmirror(8)

Categories: FreeBSD, Sysadmin.

When configuring Astase development server, I set up software mirroring using FreeBSD gmirror. This is much more a memo than a technical note, but I decided to blog about it anyway.

In this post, the disk where FreeBSD was installed is ad4 and will be referred as the first disk, while the disk that we will configure for mirroring is ad6 and will be referred as the second disk. FreeBSD is installed on the second slice of the disk using the default partition scheme.

Both disks have been sliced the same way when installing using sysinstall(8). It would of course have been possible to use fdisk(8) to do this. If both disks are identical, the quick way of doing this is to copy the slice table:

# fdisk -p ad4 > /tmp/slices
# fdisk -f /tmp/slices ad6

Disks labels for slices ad4s2 and ad6s2 are now like this:

dev# bsdlabel /dev/ad4s2
# /dev/ad4s2:
8 partitions:
#        size   offset    fstype   [fsize bsize bps/cpg]
  a:  1048576        0    4.2BSD     2048 16384     8 
  b:  4123712  1048576      swap                    
  c: 426847050        0    unused        0     0         # "raw" part, don't edit
  d:  4157440  5172288    4.2BSD     2048 16384 28528 
  e:  1048576  9329728    4.2BSD     2048 16384     8 
  f: 416468746 10378304    4.2BSD     2048 16384 28528 
dev# bsdlabel /dev/ad6s2
# /dev/ad6s2:
8 partitions:
#        size   offset    fstype   [fsize bsize bps/cpg]
  c: 426847050        0    unused        0     0         # "raw" part, don't edit

Copy the disk labels

First, we have to duplicate the partition scheme from the first disk's slice to the second disk's slice. If both slices have the same size, we can avoid some arithmetic and blindly copy the labels from on slice to the other:

# bsdlabel /dev/ad4s2 > /tmp/partitions
# bsdlabel -R /dev/ad6s2 /tmp/partitions

If the two slices have different sizes, labels will have to be tweaked a bit.


As soon as we have labelled the second disk's slice, new devices appeared in /dev/ and can be used.

Edit /etc/fstab and add the freshly created swap device:

/dev/ad6s2b             none            swap    sw              0       0

Use swapon(8) to enable the it. swapinfo(8) should report more available swap:

# swapon /dev/ad6s2b
# swapinfo
Device          1K-blocks     Used    Avail Capacity
/dev/ad4s2b       2061856        0  2061856     0%
/dev/ad6s2b       2061856        0  2061856     0%
Total             4123712        0  4123712     0%

Enable gmirror

gmirror(8) is not included in the GENERIC kernel but rather provided as a kernel module, geom_mirror.ko.

# geom mirror load
# kldstat
Id Refs Address            Size     Name
 1    2 0xffffffff80100000 ac6c08   kernel
 2    1 0xffffffffa2b8d000 cb41     geom_mirror.ko

Since we want to have the root device on a raid-1 mirror, the geom_mirror module has to be loaded before the system boots up. We thus edit the file /boot/loader.conf and ask the loader to load the geom_mirror kernel module before giving control to the kernel:


Setup mirroring

Since we are mirroring partitions that contains data, mirroring a partition is a two step process:

  1. label the existing partition to mark it as part of a RAID array;
  2. insert additional devices to the RAID array.

The only drawback is that it is not possible to label an active partition. You will therefore have to boot on a livefs if you want to have the root filesystem mirrored.

Label a partition

I will detail here how to switch from a single /var/ partition on /dev/ad4s2d to a mirrored partition on /dev/ad4s2d and /dev/ad6s2d.

Booting in single user mode and issuing the following command is enough to configure the existing partition that holds data as the first component of a RAID array:

# gmirror label var ad4s2d

A new device is then available: /dev/mirror/var. You should not mount /var from /dev/ad4s2d anymore but rather from /dev/mirror/var so edit /etc/fstab according to this:

/dev/mirror/var         /var            ufs     rw              2       2

Insert devices in the array

Once a RAID array is configured, adding devices is as easy as:

# gmirror insert var ad6s2d

The system will then automagically synchronise the added device with the RAID array data. The synchronisation process can be tracked using gmirror:

# gmirror status
      Name    Status  Components
mirror/var  DEGRADED  ad4s2d
                      ad6s2d (5%)

For more information:

# gmirror list
Geom name: var
Components: 2
Balance: split
Slice: 4096
Flags: NONE
GenID: 0
SyncID: 1
ID: 4064985514
1. Name: mirror/var
   Mediasize: 536870400 (512M)
   Sectorsize: 512
   Mode: r2w1e1
1. Name: ad4s2d
   Mediasize: 536870912 (512M)
   Sectorsize: 512
   Mode: r1w1e1
   State: ACTIVE
   Priority: 0
   Flags: NONE
   GenID: 0
   SyncID: 1
   ID: 19539080
2. Name: ad6s2d
   Mediasize: 536870912 (512M)
   Sectorsize: 512
   Mode: r1w1e1
   Priority: 0
   GenID: 0
   SyncID: 1
   Synchronized: 83%
   ID: 916017489

Mirroring the root partition

Setting up the root partition is a bit trickier since the partition has to be unmounted before labelling. You may however do this operation from another FreeBSD install or using the livefs provided on FreeBSD ftp site (and mirrors). The only thing to think about is to load geom_mirror.ko in the kernel and update /etc/fstab before rebooting the system.

Make additional disks bootable

At this stage, the system boots and runs well. However, the second disk is not bootable. As a result, if the first disk fails, the system cannot boot anymore.

As explained in boot(8), we have to:

  1. configure the MBR so that it allows us to choose a slice to start from (PC partition);
  2. configure the slice so that it find and loads the loader(8) which will in turn find and load the kernel.
# boot0cfg -B ad6
# bsdlabel -B ad6s2

Monitoring gmirror(8)

The system can be tuned to include gmirror(8) status in daily mails:

# echo 'daily_status_gmirror_enable="YES"' >> /etc/periodic.conf

A new section then appears in the daily run output describing the RAID array status like so:

Checking status of gmirror(8) devices:
       Name    Status  Components
mirror/root  COMPLETE  ad4s2a
 mirror/var  COMPLETE  ad4s2d
 mirror/tmp  COMPLETE  ad4s2e
 mirror/usr  COMPLETE  ad4s2f

Overview of the GEOM setup

The global GEOM setup is shown bellow:

Is is clearly visible on this graph that the whole system is mirrored. A disk failure will not result in a data loss.


On December 2, 2009, eFeS wrote:

Could it be possible not to use all of the slices for RAID1?
I mean, is there any possiblity to put the slice containing /tmp into a RAID0 array, and the remaining slices into RAID1?

Regards: eFeS

On December 2, 2009, Romain Tartière wrote:

Hi eFeS!

GEOM does not care about the device you tell it to consume to setup providers. You can setup RAID1 on a RAID0 volume and a remote GELI encrypter volume on another system if you want. Unlike LVM, you have no limitation nor constraints in the order you apply transformations.

GEOM is magic!

Comments RSS feed | Leave a Reply…