Applied FreeBSD: Basic iSCSI

iSCSI is often touted as a low-cost replacement for fibre-channel (FC) Storage Area Networks (SANs). Instead of having to setup a separate fibre-channel network for the SAN, or invest in the infrastructure to run Fibre-Channel over Ethernet (FCoE), iSCSI runs on top of standard TCP/IP. This means that the same network equipment used for routing user data on a network could be utilized for the storage as well. In practice, to get high-levels of performance, it is advised that system designers consider iSCSI Host Bus Adaptors (HBAs) for each iSCSI participating team, and that the network at a minimum have a separate VLAN for iSCSI traffic–or more ideally, have separate physical network.

My disclaimer: this article does not cover any of the above performance enhancements! The systems in this article are setup and configured in a VMWare Workstation virtualized environment so that I don’t have to physically procure all of the hardware just to learn about iSCSI.

This article will cover a very basic setup where a FreeBSD server is configured as an iSCSI Target, and another FreeBSD server is configured as the iSCSI Initiator. The iSCSI Target will export a single disk drive, and the initiator will create a filesystem on this disk and mount it locally. Advanced topics, such as multipath, ZFS storage pools, failover controllers, etc. are not covered.  Please refer to the following documentation on iSCSI for more information:

Now to get started…

iSCSI Target Test Setup

The disk drive which should be shared on the network is /dev/ada0, a 5G SATA disk created in VMWare that I attached to the system before starting it up. With FeeBSD, iSCSI is controled by the ctld daemon, so this needs to be enabled on the system. While at it, why not go ahead and enable it at boot time too?

root@bsdtarget:~ # echo ‘ctld_enable=”YES”‘ >> /etc/rc.conf
root@bsdtarget:~ # service start ctld
Starting iscsid.

The real magic is the /etc/ctl.conf file, which contains all of the information necessary for ctld to share disk drives on the network. Check out the man page for /etc/ctl.conf for more details; below is the configuration file that I created for this test setup. Note that on a system that has never had iSCSI configured, there will be no existing configuration file, so go ahead and create it.

root@bsdtarget:/dev # less /etc/ctl.conf
auth-group test{
chap “iscsitest” “bsdforthewin”
}

portal-group pg0 {
discovery-auth-group no-authentication
listen 192.168.22.128
}

target iqn.2017-02.lab.testing:basictarget {
auth-group no-authentication
portal-group pg0
lun 0 {
path /dev/ada0
size 5G
}
lun 1 {
path /dev/ada1
size 5G
}
}

For this setup, LUN 0 will be used by a FreeBSD iSCSI Initiator. I have LUN 1 configured for experimenting with Windows Server at a later time. Before starting ctld, it is a good idea to make sure that the /etc/ctl.conf file is not readable by all users (ctld will complain). At a later point it might be necessary to add iSCSI authentication for the sessions, and it would not be wise to have all users able to look at the authentication secret password.

root@bsdtarget:~ # chmod 640 /etc/ctl.conf
root@bsdtarget:~ # service start ctld

If there are any syntax errors or warnings, ctld will complain about it on the console. The ctladm tool can be used to query the system for more information, for example:

root@bsdtarget:/dev # ctladm lunlist
(7:0:0/0): <FREEBSD CTLDISK 0001> Fixed Direct Access SPC-4 SCSI device
(7:0:1/1): <FREEBSD CTLDISK 0001> Fixed Direct Access SPC-4 SCSI device
root@bsdtarget:/dev # ctladm devlist
LUN Backend Size (Blocks) BS Serial Number Device ID
0 block 10485760 512 MYSERIAL 0 MYDEVID 0
1 block 10485760 512 MYSERIAL 1 MYDEVID 1

 

That’s really it for the iSCSI Target configuration. The real effort is in setting up the /etc/ctl.conf file. And for a real production system, there would be more configuration with the exported disks, such as using ZFS shares, RAID-1 mirroring, et cetera.

iSCSI Initiator Test Setup

In order for a FreeBSD host to become an iSCSI Initiator, the iscsd daemon needs to be started. It doesn’t hurt to go ahead and add the instruction to /etc/rc.conf so that iscsid is started when the system comes up.

root@bsdinitiator:~ # echo ‘iscsid_enable=”YES”‘ >> /etc/rc.conf
root@bsdinitiator:~ # service start iscsid
Starting iscsid.

Next, the iSCSI Initiator can manually connect to the iSCSI target using the iscsictl tool. While setting up a new iSCSI session, this is probably the best option. Once you are sure the configuration is correct, add the configuration to the /etc/iscsi.conf file (see man page for this file). For iscsictl, pass the IP address of the target as well as the iSCSI IQN for the session:

root@bsdinitiator:~ # iscsictl -A -p 192.168.22.128 -t iqn.2017-02.lab.testing:basictarget

The command returns silently, but a look at /var/message/logs shows that the remote disk was recognized and is now recognized by the Initiator as /dev/da1.

da1 at iscsi3 bus 0 scbus34 target 0 lun 0
da1: <FREEBSD CTLDISK 0001> Fixed Direct Access SPC-4 SCSI device
da1: Serial Number MYSERIAL 0
da1: 150.000MB/s transfers
da1: Command Queueing enabled
da1: 5120MB (10485760 512 byte sectors)

The iSCSI session connection status can also be verified with iscsictl:

root@bsdinitiator:~ # iscsictl -L
Target name                                                 Target portal          State
iqn.2017-02.lab.testing:basictarget    192.168.22.128       Connected: da1

Once the disk is recognized by the iSCSI Initiator system, it can be configured for use on the Initiator like a regular SCSI/SATA disk attached to the system physically. The commands below create a partition and UFS filesystem on /dev/da1.

root@bsdinitiator:~ # gpart create -s gpt /dev/da1
root@bsdinitiator:~ # gpart add -t freebsd-ufs -l 1m /dev/da1
root@bsdinitiator:~ # newfs -U /dev/da1p1
/dev/da1p1: 5120.0MB (10485688 sectors) block size 32768, fragment size 4096
using 9 cylinder groups of 626.09MB, 20035 blks, 80256 inodes.
with soft updates
super-block backups (for fsck_ffs -b #) at:
192, 1282432, 2564672, 3846912, 5129152, 6411392, 7693632, 8975872, 10258112
root@bsdinitiator:~ # mkdir /iscsi_share
root@bsdinitiator:~ # mount -t ufs -o rw /dev/da1p1 /iscsi_share

If there is already a filesystem resident on the device, it only needs to be mounted after the iSCSI session is connected. Back on the iSCSI Target machine, it is possible to see all of the iSCSI Initiators connected

root@bsdtarget:/dev # ctladm islist
ID   Portal                   Initiator name                                                 Target name
6    192.168.22.136   iqn.1994-09.org.freebsd:bsdinitiator      iqn.2017-02.lab.testing:basictarget

Finally, if for some reason it is necessary to disconnect the system, unmount the filesystem and use iscsictl to disconnect the iSCSI session.

root@bsdinitiator:~ # umount /iscsi_share
root@bsdinitiator:~ # iscsictl -R -t iqn.2017-02.lab.testing:basictarget

There is much more to explore with iSCSI, this is just the very beginning, but it serves a a model and a starting point for this work. More to come in the future!

Update: Windows iSCSI Initiator

I am not a very savvy Windows user, and I am very new to Windows Server. I have just started to learn some of the basics. As such, I thought I’d try setting up a Windows Server 2016 host as an iSCSI Initiator. I won’t go into much detail other than what is required for setting up the iSCSI parts. Go ahead and fire up Server Manager.

initiator1

From the “Tools” menu, select iSCSI Initiator. It is also possible to start this application from the Windows search tool by searching for “iSCSI Initiator”. As shown below, when running it for the first time, Microsoft’s iSCSI service may not be running. If not, start it up!initiator2

There are many options for configuring the iSCSI Initiator, but for demonstration purposes we’ll cover the basic case. In the Target box, enter the IP address of the iSCSI Target machine and click on the Quick Connect button.

initiator4

A screen should pop-up window finding the IQN for the iSCSI Target service, and it should also state somewhere that the Login was successful.

initiator6

After closing out the pop-up window, the target should now be in the Discovered targets area of the Targets tab.

initiator5

Next go to the Volumes and Devices tab. Unless you know the exact mount point for the iSCSI volume, the best bet is to click the Auto Configure button which will get the data from the iSCSI Target, as shown below.

initiator7

I bet you wouldn’t have memorized that!  Both LUN 0 and LUN 1 are recognized by Windows. Press OK to exit out of the iSCSI Initiator application. Next, open up the Disk Management application on the Windows Server.

initiator8

Notice that two new 5 GB disks are present. The tricky part here is that I am not sure which is LUN 0 and which is LUN 1. My best guess is that the disk that is recognized as a healthy primary partition is LUN 0 which contains a GPT label and is UFS formatted.  Thus the unrecognized disk must be LUN 1, which was not modified by the FreeBSD iSCSI Initiator. In reality I would deploy two iSCSI portal groups, one for the FreeBSD iSCSI Initiators and one for Windows iSCSI Initiators. I might have a third portal group for shared volumes.

For the unrecognized volume, create a MBR partition with the Disk Management tool, and then create a FAT32 partition on this disk as well. I decided to name the partition ISCSI_BSD. As shown below, on this Windows Server, the E: drive is now LUN 1, or /dev/ada2 back on my FreeBSD iSCSI Target machine.

initiator9

The iSCSI drive shows up as a regular drive in Windows Explorer, as shown below.

initiator10

Inside I created a special message for viewing from the FreeBSD side:

initiator11

Finally, it is possible to verify the Windows access using the FreeBSD iSCSI Initiator. Reload the iSCSI Target session data, and now /dev/da2 is available on the FreeBSD Initiator. Even nicer, the FAT32 partitions are recognized by FreeBSD–less work to do! On Windows Server an MBR partition was created, which shows up as /dev/da2p1 in FreeBSD, and the actual FAT32 data partition is /dev/da2p2.

root@bsdinitiator:/iscsi_win_edrive # iscsictl -R -t iqn.2017-02.lab.testing:basictarget
root@bsdinitiator:/iscsi_win_edrive # iscsictl -A -p 192.168.22.128 -t iqn.2017-02.lab.testing:basictarget

root@bsdinitiator:~ # ls -l /dev/da2*
crw-r—– 1 root operator 0x72 Mar 4 22:32 /dev/da2
crw-r—– 1 root operator 0x77 Mar 4 22:32 /dev/da2p1
crw-r—– 1 root operator 0x78 Mar 4 22:32 /dev/da2p2
root@bsdinitiator:~ # mount_msdosfs /dev/da2p2 /iscsi_win_edrive
root@bsdinitiator:~ # cd /iscsi_win_edrive/
root@bsdinitiator:/iscsi_win_edrive # ls
$RECYCLE.BIN hello.txt
System Volume Information
root@bsdinitiator:/iscsi_win_edrive # cat hello.txt
Hello FreeBSD! This is Windows Server!
I made your /dev/ada1 into a FAT32 partition.
I call it E: Drive. Thank you!

It works! I can see the message from Windows land.

Advertisements

FreeBSD Major/Minor Version Upgrades

Last year at some point I installed FreeBSD 10.2-RELEASE on an old i686 computer. In the time between now and then, there was a 10.3 minor release as well as an 11.0 major release. I am not in a hurry to get to FreeBSD 11.0-RELEASE yet, but I thought it was time to make the minor version upgrade to 10.3-RELEASE. Fortunately, upgrading FreeBSD is an easy process.

The current version of FreeBSD installed on the machine is shown below.

root@bsdbox:~ # uname -a
FreeBSD bsdbox 10.2-RELEASE FreeBSD 10.2-RELEASE #0 r286666: Wed Aug 12 19:31:38 UTC 2015 root@releng1.nyi.freebsd.org:/usr/obj/usr/src/sys/GENERIC i386

To upgrade to the next minor version, use the freebsd-update command with the upgrade option and the release version.

root@bsdbox:~ # freebsd-update -r 10.3-RELEASE upgrade

Depending on your system, this command could take a while to run while it inspects the system. On this i686 machine, I just let the command run and came back after an hour or so to check on its progress.

Looking up update.FreeBSD.org mirrors… 4 mirrors found.
Fetching metadata signature for 10.2-RELEASE from update6.freebsd.org… done.
Fetching metadata index… done.
Fetching 1 metadata files… done.
Inspecting system…

The following components of FreeBSD seem to be installed:
kernel/generic src/src world/base

The following components of FreeBSD do not seem to be installed:
world/doc world/games

Does this look reasonable (y/n)?

After inspecting the system, the upgrade process is just making sure I am happy with the parts that will be upgraded. Enter “y” to continue, and then if you have an i686 machine, walk away and give it some time to fetch the updates.

Fetching metadata signature for 10.3-RELEASE from update6.freebsd.org… done.
Fetching metadata index… done.
Fetching 1 metadata patches. done.
Applying metadata patches… done.
Fetching 1 metadata files… done.
Inspecting system… done.
Fetching files from 10.2-RELEASE for merging… done.
Preparing to download files…
…(output truncated for the sake of sanity)…
590….39600….39610….39620….39630….39640….39650….39660….39670….39680….39690….39700….39710….39720….39730….39740….39750….39760….39770….39780….39790….39800….39810….39820….39830….39840….39850….39860….39870….39880….39890….39900….39910….39920….39930….39940….39950….39960….39970….39980….39990….40000….40010….40020….40030 done.
Applying patches… done.
Fetching 873 files… done.
Attempting to automatically merge changes in files… done.

The following file could not be merged automatically: /etc/ssh/ssh_config
Press Enter to edit this file in vi and resolve the conflicts
manually…

I had made some modifications to /etc/ssh/ssh_config on my baseline system, so rather than just overwriting the file, FreeBSD is offering me the opportunity to look at a diff of my original and the new version that FreeBSD needs to install. Luckily for me, I had just added one line, and I no longer needed it, so I edit out the changes and let FreeBSD move onward. Once you save the changes and exit out of the editor, the upgrade process continues.

…(truncated)…
/var/db/etcupdate/current/root/.k5login
/var/db/etcupdate/current/root/.login
/var/db/etcupdate/current/root/.profile
/var/db/etcupdate/log
/var/db/mergemaster.mtree
/var/yp/Makefile.dist
To install the downloaded upgrades, run “/usr/sbin/freebsd-update install”.

Excellent! The updates have been downloaded, my system has been examined, and all of the necessary modifications have been merged into the software to install. To initiate the actual install:

root@bsdbox:~ # freebsd-update install
Installing updates…
Kernel updates have been installed. Please reboot and run
“/usr/sbin/freebsd-update install” again to finish installing updates.

At this point, the system is ready to be rolled over to the next version of FreeBSD. Reboot the system:

root@bsdbox:~ # shutdown -r +1

Once the system comes back up, check out the version:

root@bsdbox:~ # uname -a
FreeBSD bsdbox 10.3-RELEASE-p11 FreeBSD 10.3-RELEASE-p11 #0: Mon Oct 24 18:47:18 UTC 2016 root@amd64-builder.daemonology.net:/usr/obj/usr/src/sys/GENERIC i386

Nice! To finalize the install, run the freebsd-update command with the install option as directed in the output before the last system reboot.

root@bsdbox:~ # freebsd-update install

That is it, the system is now upgraded from FreeBSD 10.2 to 10.3! An upgrade to a major version is no different from the above, just specify the proper version. For example, to upgrade to FreeBSD 11.0:

root@bsdbox:~ # freebsd-update -r 11.0-RELEASE upgrade

Easy!

FreeBSD and Iomega ZIP Drives

Iomega ZIP drives and disks are dead technology, but I have an old box from 1999 that has a fully functioning 100MB ZIP drive. Out of the box, FreeBSD supports these devices through the SCSI subsystem. During the boot process, FreeBSD recognizes the drive as /dev/da0.

da0 at ata0 bus 0 scbus0 target 1 lun 0
da0: <IOMEGA ZIP 100 12.A> Removable Direct Access SCSI device
da0: 11.100MB/s transfers (PIO3, ATAPI 12bytes, PIO 65534bytes)
da0: Attempt to query device size failed: NOT READY, Medium not present

The last line is not a critical failure, it just means that there is no ZIP disk inserted in the drive. Rebooting the machine with a ZIP disk inserted results in the following output:

da0 at ata0 bus 0 scbus0 target 1 lun 0
da0: <IOMEGA ZIP 100 12.A> Removable Direct Access SCSI device
da0: 11.100MB/s transfers (PIO3, ATAPI 12bytes, PIO 65534bytes)
da0: 96MB (196608 512 byte sectors: 255H 255S/T 3C)

The last line shows that the ZIP disk media was recognized, and the number of 512-byte blocks is listed. A disk can be inserted at any time, and interestingly enough, when a disk is inserted into the driver, there are no messages in /var/log/messages indicating which partition the media is listed as. Running a listing on the devices in /dev can reveal some hints though.

root@bsdbox:/media # ls /dev
acpi consolectl kbd0 pci ttyv4
….
… da0 …
… da0s4 …

The drive represents the ZIP media on /dev/da0s4. According to older documentation from FreeBSD 6, slice 4 is where the device driver places the media partition.  To access the ZIP disk, first create a mount point on the filesystem, such as /media/zipdisk, and then mount the device using the mount(8). The zip disks I have are FAT formatted, so I need to use the “-t msdosfs” option.

root@bsdbox:/meda # mkdir /media/zipdisk
root@bsdbox:/meda # mount -t msdosfs /dev/da0s4 /media/zipdisk

This disk is now available on /media/zipdisk for reading and writing.  When finished with the disk, before physically ejecting it, be sure to unmount the filesystem first.

root@bsdbox:/meda # umount /media/zipdisk

Now it is safe to press the disk eject button on the drive. Of course for frequent use of a ZIP drive, entries should be added to /etc/fstab to make things easier. For infrequent use, however, the above methodology should work just fine!

FreeBSD and USB Thumbdrives

USB thumbdrives are a convenient way of moving relatively small volumes of data around machines. FreeBSD fully supports such devices. Insert the thumbdrive into an open USB port and check to make sure FreeBSD detects its using the command below.

# dmesg | tail

USB drives are handled by the SCSI subsystem so look for output that would resemble the following:

ugen0.3: <vendor 0x0d7d> at usbus0
umass0: <vendor 0x0d7d USB DISK 2.0, class 0/0, rev 2.00/0.50, addr 3> on usbus0
umass0:  SCSI over Bulk-Only; quirks = 0xc180
umass0:2:0:-1: Attached to scbus2
da1 at umass-sim0 bus 0 scbus2 target 0 lun 0
da1: < USB DISK 2.0 1.16> Removable Direct Access SCSI device
da1: Serial Number 073A0C251C0B
da1: 1.000MB/s transfers
da1: 124MB (253952 512 byte sectors: 64H 32S/T 124C)
da1: quirks=0x3<NO_SYNC_CACHE,NO_6_BYTE>

From the output above, we know the device is /dev/da1.  You can also query the SCSI system to see if the device is found. Note this output also shows “da1”.

root@bsdbox:~ # camcontrol devlist
<WDC WD400EB-00CPF0 06.04G06>      at scbus0 target 0 lun 0 (pass0,ada0)
<IOMEGA ZIP 100 12.A>              at scbus0 target 1 lun 0 (pass1,da0)
<LG CD-ROM CRD-8400B 1.03>         at scbus1 target 0 lun 0 (cd0,pass2)
<SONY CD-RW  CRX220E1 6YS1>        at scbus1 target 1 lun 0 (cd1,pass3)
< USB DISK 2.0 1.16>               at scbus2 target 0 lun 0 (da1,pass4)
Trying to mount just /dev/da1 is not going to succeed. We have to tell FreeBSD which partition to mount. Luckily, for most USB thumbdrives without any encryption (such as IronKeys), there is just one partition on the drive. As such, we can specify /dev/da1s1 to get slice #1, or the first partition. To mount the partition manually, we can use the mount(8) command while specifying the filesystem on the device. The USB thumbdrive I have is FAT formatted, so the “-t msdosfs” option to the mount command will tell FreeBSD to override the default format of UFS. Before running the mount command, however, make sure you have a place to mount the filesystem. I happened to choose /media/usb.
root@bsdbox:~ # mkdir /media/usb
root@bsdbox:~ # mount -t msdosfs /dev/da1s1 /media/usb
The mount command should complete silently, and you should be able to access /media/usb and copy files to and from the device. When you have finished with the device and want to remove it, use the umount command.
root@bsdbox:~ # umount /media/usb
It is safe to remove the device now without potentially damaging the filesystem on the device. After physically unplugging the device from the USB port, if you go back and look at the messages on the system (dmesg | tail), you will see something like the following:
ugen0.3: <vendor 0x0d7d> at usbus0 (disconnected)
umass0: at uhub0, port 2, addr 3 (disconnected)
da1 at umass-sim0 bus 0 scbus2 target 0 lun 0
da1: < USB DISK 2.0 1.16> s/n 073A0C251C0B detached
(da1:umass-sim0:0:0:0): Periph destroyed