FreeBSD 7.2 with ZFS as an iSCSI target

Reason for writing this quick-start howto...

During my migration from ESXi 4.0 to XenServer 5.5, I ran into a bunch of problems with two iSCSI implementations, namely FreeNAS and OpenFiler. While XenServer failed to talk to FreeNAS altogether, OpenFiler should definitely not be used in a VM (I tried their XenServer appliance), because under a bit more load it will start throwing error messages at some point, until the consumers eventually block altogether, and only a restart of the OpenFiler and all consuming VMs would help.
So I had a look at FreeBSD, who "borrowed" the istgt iSCSI target implementation from NetBSD. It's not part of the base yet, but available as a port, and as such working very well. Here's how to turn your FreeBSD box into an iSCSI NAS.

(As far as I know, you can do the same with OpenSolaris, but I'm more familiar with FreeBSD and lacked time to try it with OpenSolaris... laters )

Pre-Requisites

Before you start with the setup, please keep a couple of things in mind:

• iSCSI needs a very reliable and fast network, hence GBit Ethernet is mandatory
• It's recommended to have all consumers and the NAS box on the same physical trusted network, hence separated from other traffic on a second NIC, of course
• both ZFS and istgt (iSCSI target) are only available on FreeBSD 7.x
• keep in mind that ZFS, the underlying filesystem, is still flagged experimental in FreeBSD 7.2
• you do need unused disk space on your box, which you can use for the ZFS filesystem

If you don't have a RAID controller with RAID-1, -5, or -10 in your box, it's highly recommended to have two free slices of the same size on two separate disks. That would enable you to use ZFS's built-in software RAID functionality.

Create a basic ZFS layout

ZFS can deal with labels, slices, or with entire devices. In this example, I will assume that ad0s3 is blank.

Firstly, we need to set up a pool, called san in this example (note that you can add devices on the fly later, when you run out of space):

$ zpool create san 

$ zfs list
NAME         USED  AVAIL  REFER  MOUNTPOINT
san         90.9G   354G  82.0G  /san

(You will see different values, depending on your pool, of course. I am just my existing setup to clarify things. Too lazy to re-do from scratch )

The next step is to create a data set, which we can later use for iSCSI. It is important to limit it's size with a quota. Otherwise you wouldn't be able to tell how big the iSCSI targets are, and more importantly, how much space you've got provisioned.
The second vital thing to do is to not assign a mountpoint for the data sets. The client using the iSCSI target is responsible for formatting space in the iSCSI target. You do not want to mess with it.

Here's what we do:

zfs create san/target
zfs set mountpoint=none san/target
zfs set quota=200G san/target

Now you've provisioned 200G for a future iSCSI target, referenced to as san/target.

So let's install the software...

Install iSCSI target

First, we need a current version of the ports tree, and compile istgt (iSCSI target):

$ portsnap fetch update
$ portinstall -c istgt-20090428

That's all the software we need. istgt is the userland daemon to provide ZFS data sets via iSCSI. The configuration files, or rather examples, can be found in /usr/local/etc/istgt. Copy the files and strip the sample extension:

cd /usr/local/etc/istgt
cp auth.conf.sample auth.conf
cp istgt.conf.sample istgt.conf
cp istgtcontrol.conf.sample istgtcontrol.conf

For now, we don't need to touch anything but istgt.conf. All you have to change to fit your needs is:

 Portal DA1 192.168.2.36:3260

Netmask 192.168.2.0/24

I strongly recommend – again – to have the iSCSI target machine and all consumers on the same physical network, and also in the same subnet!

The last thing to do is to apply your ZFS pool's configuration to istgt.conf:

REPLACE:
LUN0 Storage /tank/iscsi/istgt-disk1 10GB

WITH:
LUN0 Storage /san/target

You don't need to fiddle with the authentication right now. But if you want to (after running an initial test without), the last group in auth.conf is a good place to start, especially the "onlysingle" user with non-mutual CHAP authentication.
(I haven't found any other methods to work with either XenServer or ESXi.)

Adding ZFS & istgt to the startup process

Now, you're almost there. Only two things left:

Adding two lines to /etc/rc.conf:

istgt_enable="YES" 
zfs_enable="YES"

...and...

Starting the deamon:

/usr/local/etc/rc.d/istgt start

Final ZFS tweaks

Now you can connect XenCenter or ESX hosts to the iSCSI target. If everything worked out, please add these important tweaks to your /boot/loader.conf and reboot the machine afterwards:

zfs_load="YES"
vm.kmem_size_max=1536M
vm.kmem_size=1536M
vfs.zfs.zil_disable=1

They make sure that ZFS will find enough kernel memory for its massive caching. You should only reduce the value (not less than 0.5GB!), if you haven't got enough memory in your machine.

All right. That's it. Have fun

Shortcuts