Raid-1 using gmirror(8)
Categories: FreeBSD, Sysadmin.
When configuring Astase development server, I set up software mirroring using FreeBSD gmirror. This is much more a memo than a technical note, but I decided to blog about it anyway.
In this post, the disk where FreeBSD was installed is ad4 and will be referred as the first disk, while the disk that we will configure for mirroring is ad6 and will be referred as the second disk. FreeBSD is installed on the second slice of the disk using the default partition scheme.
Both disks have been sliced the same way when installing using sysinstall(8). It would of course have been possible to use fdisk(8) to do this. If both disks are identical, the quick way of doing this is to copy the slice table:
# fdisk -p ad4 > /tmp/slices # fdisk -f /tmp/slices ad6
Disks labels for slices ad4s2 and ad6s2 are now like this:
dev# bsdlabel /dev/ad4s2 # /dev/ad4s2: 8 partitions: # size offset fstype [fsize bsize bps/cpg] a: 1048576 0 4.2BSD 2048 16384 8 b: 4123712 1048576 swap c: 426847050 0 unused 0 0 # "raw" part, don't edit d: 4157440 5172288 4.2BSD 2048 16384 28528 e: 1048576 9329728 4.2BSD 2048 16384 8 f: 416468746 10378304 4.2BSD 2048 16384 28528 dev# bsdlabel /dev/ad6s2 # /dev/ad6s2: 8 partitions: # size offset fstype [fsize bsize bps/cpg] c: 426847050 0 unused 0 0 # "raw" part, don't edit
Copy the disk labels
First, we have to duplicate the partition scheme from the first disk's slice to the second disk's slice. If both slices have the same size, we can avoid some arithmetic and blindly copy the labels from on slice to the other:
# bsdlabel /dev/ad4s2 > /tmp/partitions # bsdlabel -R /dev/ad6s2 /tmp/partitions
If the two slices have different sizes, labels will have to be tweaked a bit.
Add SWAP
As soon as we have labelled the second disk's slice, new devices appeared in /dev/ and can be used.
Edit /etc/fstab and add the freshly created swap device:
/dev/ad6s2b none swap sw 0 0
Use swapon(8) to enable the it. swapinfo(8) should report more available swap:
# swapon /dev/ad6s2b # swapinfo Device 1K-blocks Used Avail Capacity /dev/ad4s2b 2061856 0 2061856 0% /dev/ad6s2b 2061856 0 2061856 0% Total 4123712 0 4123712 0%
Enable gmirror
gmirror(8) is not included in the GENERIC kernel but rather provided as a kernel module, geom_mirror.ko.
# geom mirror load # kldstat Id Refs Address Size Name 1 2 0xffffffff80100000 ac6c08 kernel 2 1 0xffffffffa2b8d000 cb41 geom_mirror.ko
Since we want to have the root device on a raid-1 mirror, the geom_mirror module has to be loaded before the system boots up. We thus edit the file /boot/loader.conf and ask the loader to load the geom_mirror kernel module before giving control to the kernel:
geom_mirror_load="YES"
Setup mirroring
Since we are mirroring partitions that contains data, mirroring a partition is a two step process:
- label the existing partition to mark it as part of a RAID array;
- insert additional devices to the RAID array.
The only drawback is that it is not possible to label an active partition. You will therefore have to boot on a livefs if you want to have the root filesystem mirrored.
Label a partition
I will detail here how to switch from a single /var/ partition on /dev/ad4s2d to a mirrored partition on /dev/ad4s2d and /dev/ad6s2d.
Booting in single user mode and issuing the following command is enough to configure the existing partition that holds data as the first component of a RAID array:
# gmirror label var ad4s2d
A new device is then available: /dev/mirror/var. You should not mount /var from /dev/ad4s2d anymore but rather from /dev/mirror/var so edit /etc/fstab according to this:
/dev/mirror/var /var ufs rw 2 2
Insert devices in the array
Once a RAID array is configured, adding devices is as easy as:
# gmirror insert var ad6s2d
The system will then automagically synchronise the added device with the RAID array data. The synchronisation process can be tracked using gmirror:
# gmirror status
Name Status Components
mirror/var DEGRADED ad4s2d
ad6s2d (5%)
For more information:
# gmirror list Geom name: var State: DEGRADED Components: 2 Balance: split Slice: 4096 Flags: NONE GenID: 0 SyncID: 1 ID: 4064985514 Providers: 1. Name: mirror/var Mediasize: 536870400 (512M) Sectorsize: 512 Mode: r2w1e1 Consumers: 1. Name: ad4s2d Mediasize: 536870912 (512M) Sectorsize: 512 Mode: r1w1e1 State: ACTIVE Priority: 0 Flags: NONE GenID: 0 SyncID: 1 ID: 19539080 2. Name: ad6s2d Mediasize: 536870912 (512M) Sectorsize: 512 Mode: r1w1e1 State: SYNCHRONIZING Priority: 0 Flags: DIRTY, SYNCHRONIZING GenID: 0 SyncID: 1 Synchronized: 83% ID: 916017489
Mirroring the root partition
Setting up the root partition is a bit trickier since the partition has to be unmounted before labelling. You may however do this operation from another FreeBSD install or using the livefs provided on FreeBSD ftp site (and mirrors). The only thing to think about is to load geom_mirror.ko in the kernel and update /etc/fstab before rebooting the system.
Make additional disks bootable
At this stage, the system boots and runs well. However, the second disk is not bootable. As a result, if the first disk fails, the system cannot boot anymore.
As explained in boot(8), we have to:
- configure the MBR so that it allows us to choose a slice to start from (PC partition);
- configure the slice so that it find and loads the loader(8) which will in turn find and load the kernel.
# boot0cfg -B ad6 # bsdlabel -B ad6s2
Monitoring gmirror(8)
The system can be tuned to include gmirror(8) status in daily mails:
# echo 'daily_status_gmirror_enable="YES"' >> /etc/periodic.conf
A new section then appears in the daily run output describing the RAID array status like so:
Checking status of gmirror(8) devices:
Name Status Components
mirror/root COMPLETE ad4s2a
ad6s2a
mirror/var COMPLETE ad4s2d
ad6s2d
mirror/tmp COMPLETE ad4s2e
ad6s2e
mirror/usr COMPLETE ad4s2f
ad6s2f
Overview of the GEOM setup
The global GEOM setup is shown bellow:
Is is clearly visible on this graph that the whole system is mirrored. A disk failure will not result in a data loss.
Comments
On December 2, 2009, eFeS wrote:
Could it be possible not to use all of the slices for RAID1?
I mean, is there any possiblity to put the slice containing /tmp into a RAID0 array, and the remaining slices into RAID1?Regards: eFeS
On December 2, 2009, Romain Tartière wrote:
Hi eFeS!
GEOM does not care about the device you tell it to consume to setup providers. You can setup RAID1 on a RAID0 volume and a remote GELI encrypter volume on another system if you want. Unlike LVM, you have no limitation nor constraints in the order you apply transformations.
GEOM is magic!
![[RSS]](/images/rss.png){kind=small}
