ZFS on FreeBSD Notes

Disk Setup

Before you start using disks with ZFS you want to prep them. For the purpose of these examples we assume your disks are /dev/ada0, /dev/ada1, /dev/ada2, ..., so on.

# avoid dd or gpart Operation not permitted errors
sysctl kern.geom.debugflags=0x10

Identifying Disks

There are several ways to identify the devices on your system. The crudest of all is to go through the output of dmesg and see what was recognized on system boot. But you can do better:

# get the list of drives (including flash drives)
camcontrol devlist

# info on a specific device
camcontrol identify ada1

# another option is to use smartctl (/usr/ports/sysutils/smartmontools)
# get the list of drives
smartctl --scan

# info on a specific device
smartctl -a /dev/ada0

# another option to get device info
diskinfo -v /dev/ada6 # you can also use: ada6
diskinfo -v /dev/gpt/disk6 # you can also use: gpt/disk6

Note that the above tools can also give you a disk's serial number, which is helpful when trying to identify which physical disk is mapped to a given device. For example, if ZFS ever reports that ada0 failed, you can use

camcontrol identify ada0 | grep "serial number"

to get the failed device's serial number so you can go through the physical disks and identify the one that you have to replace.

If you hand off GPT partitions (as explained later) to ZFS, it may report a failed drive as gpt/disk6. In this case you can either go through the output of gpart list to see which physical device the disk6 label belongs to, and run the above camcontrol command, or you can directly find it with:

# ignore the last two characters (slice number)
diskinfo -v gpt/disk6 | grep ident

MBR formatted disks

If your disk was previously MBR formatted you want to do:

dd if=/dev/zero of=/dev/ada0 bs=512 count=34

# Now remove the secondary or backup table at the end of the disk
diskinfo -v /dev/ada0 | perl -ne 'if(/(\d+)\s*#\s*mediasize in sectors/){print $1-34,"\n";}'
# 488397134 = mediasize minus 34, the result from above
dd if=/dev/zero of=/dev/ada0 bs=512 seek=488397134

GPT formatted disks

For GPT formatted disks you can simply do:

# note: for gpart ada0 is the same as /dev/ada0
gpart destroy -F ada0

Now restore kern.geom.debugflags and reboot.

# restore setting
sysctl kern.geom.debugflags=0x0
reboot

Partition the Drive

We could now hand off the entire disk to ZFS, but there is a little problem. Even when all your drives are from the same manufacturer and of the same model and size, they may not have the exact same size. If you ever need to replace a bad drive and your replacement is one sector short, you won't be able to use it. There is a solution: partition the drive with gpart to a slightly smaller size than its total size, and hand off the partition to ZFS. Thus we can do:

gpart create -s gpt ada0

# -b : start after 1M (which is a 4K boundary, better performance on some disks)
# -s : subtract 100k sectors, or 50M
# -a : 4k alignment for Advanced Format (4KB sectors) drives
gpart add -b 2048 -s 488297168 -t freebsd-zfs -l disk0_P4GT9LTC -a 4k ada0

Notice how we put the disk serial number in the disk label (after the underscore); this is helpful when a drive fails and we have to identify which physical drive to replace.

Create a ZFS Pool

Now you can use the disk partition to create a pool with:

# note: for zpool gpt/disk0_P4GT9LTC is the same as /dev/gpt/disk0_P4GT9LTC
zpool create ztest gpt/disk0_P4GT9LTC

# confirm 4k alignment, the command below should report ashift set to 12
zdb ztest | grep ashift