Applied FreeBSD: Basic iSCSI

iSCSI is often touted as a low-cost replacement for fibre-channel (FC) Storage Area Networks (SANs). Instead of having to setup a separate fibre-channel network for the SAN, or invest in the infrastructure to run Fibre-Channel over Ethernet (FCoE), iSCSI runs on top of standard TCP/IP. This means that the same network equipment used for routing user data on a network could be utilized for the storage as well. In practice, to get high-levels of performance, it is advised that system designers consider iSCSI Host Bus Adaptors (HBAs) for each iSCSI participating team, and that the network at a minimum have a separate VLAN for iSCSI traffic–or more ideally, have separate physical network.

My disclaimer: this article does not cover any of the above performance enhancements! The systems in this article are setup and configured in a VMWare Workstation virtualized environment so that I don’t have to physically procure all of the hardware just to learn about iSCSI.

This article will cover a very basic setup where a FreeBSD server is configured as an iSCSI Target, and another FreeBSD server is configured as the iSCSI Initiator. The iSCSI Target will export a single disk drive, and the initiator will create a filesystem on this disk and mount it locally. Advanced topics, such as multipath, ZFS storage pools, failover controllers, etc. are not covered.  Please refer to the following documentation on iSCSI for more information:

Now to get started…

iSCSI Target Test Setup

The disk drive which should be shared on the network is /dev/ada0, a 5G SATA disk created in VMWare that I attached to the system before starting it up. With FeeBSD, iSCSI is controled by the ctld daemon, so this needs to be enabled on the system. While at it, why not go ahead and enable it at boot time too?

root@bsdtarget:~ # echo ‘ctld_enable=”YES”‘ >> /etc/rc.conf
root@bsdtarget:~ # service start ctld
Starting iscsid.

The real magic is the /etc/ctl.conf file, which contains all of the information necessary for ctld to share disk drives on the network. Check out the man page for /etc/ctl.conf for more details; below is the configuration file that I created for this test setup. Note that on a system that has never had iSCSI configured, there will be no existing configuration file, so go ahead and create it.

root@bsdtarget:/dev # less /etc/ctl.conf
auth-group test{
chap “iscsitest” “bsdforthewin”
}

portal-group pg0 {
discovery-auth-group no-authentication
listen 192.168.22.128
}

target iqn.2017-02.lab.testing:basictarget {
auth-group no-authentication
portal-group pg0
lun 0 {
path /dev/ada0
size 5G
}
lun 1 {
path /dev/ada1
size 5G
}
}

For this setup, LUN 0 will be used by a FreeBSD iSCSI Initiator. I have LUN 1 configured for experimenting with Windows Server at a later time. Before starting ctld, it is a good idea to make sure that the /etc/ctl.conf file is not readable by all users (ctld will complain). At a later point it might be necessary to add iSCSI authentication for the sessions, and it would not be wise to have all users able to look at the authentication secret password.

root@bsdtarget:~ # chmod 640 /etc/ctl.conf
root@bsdtarget:~ # service start ctld

If there are any syntax errors or warnings, ctld will complain about it on the console. The ctladm tool can be used to query the system for more information, for example:

root@bsdtarget:/dev # ctladm lunlist
(7:0:0/0): <FREEBSD CTLDISK 0001> Fixed Direct Access SPC-4 SCSI device
(7:0:1/1): <FREEBSD CTLDISK 0001> Fixed Direct Access SPC-4 SCSI device
root@bsdtarget:/dev # ctladm devlist
LUN Backend Size (Blocks) BS Serial Number Device ID
0 block 10485760 512 MYSERIAL 0 MYDEVID 0
1 block 10485760 512 MYSERIAL 1 MYDEVID 1

 

That’s really it for the iSCSI Target configuration. The real effort is in setting up the /etc/ctl.conf file. And for a real production system, there would be more configuration with the exported disks, such as using ZFS shares, RAID-1 mirroring, et cetera.

iSCSI Initiator Test Setup

In order for a FreeBSD host to become an iSCSI Initiator, the iscsd daemon needs to be started. It doesn’t hurt to go ahead and add the instruction to /etc/rc.conf so that iscsid is started when the system comes up.

root@bsdinitiator:~ # echo ‘iscsid_enable=”YES”‘ >> /etc/rc.conf
root@bsdinitiator:~ # service start iscsid
Starting iscsid.

Next, the iSCSI Initiator can manually connect to the iSCSI target using the iscsictl tool. While setting up a new iSCSI session, this is probably the best option. Once you are sure the configuration is correct, add the configuration to the /etc/iscsi.conf file (see man page for this file). For iscsictl, pass the IP address of the target as well as the iSCSI IQN for the session:

root@bsdinitiator:~ # iscsictl -A -p 192.168.22.128 -t iqn.2017-02.lab.testing:basictarget

The command returns silently, but a look at /var/message/logs shows that the remote disk was recognized and is now recognized by the Initiator as /dev/da1.

da1 at iscsi3 bus 0 scbus34 target 0 lun 0
da1: <FREEBSD CTLDISK 0001> Fixed Direct Access SPC-4 SCSI device
da1: Serial Number MYSERIAL 0
da1: 150.000MB/s transfers
da1: Command Queueing enabled
da1: 5120MB (10485760 512 byte sectors)

The iSCSI session connection status can also be verified with iscsictl:

root@bsdinitiator:~ # iscsictl -L
Target name                                                 Target portal          State
iqn.2017-02.lab.testing:basictarget    192.168.22.128       Connected: da1

Once the disk is recognized by the iSCSI Initiator system, it can be configured for use on the Initiator like a regular SCSI/SATA disk attached to the system physically. The commands below create a partition and UFS filesystem on /dev/da1.

root@bsdinitiator:~ # gpart create -s gpt /dev/da1
root@bsdinitiator:~ # gpart add -t freebsd-ufs -l 1m /dev/da1
root@bsdinitiator:~ # newfs -U /dev/da1p1
/dev/da1p1: 5120.0MB (10485688 sectors) block size 32768, fragment size 4096
using 9 cylinder groups of 626.09MB, 20035 blks, 80256 inodes.
with soft updates
super-block backups (for fsck_ffs -b #) at:
192, 1282432, 2564672, 3846912, 5129152, 6411392, 7693632, 8975872, 10258112
root@bsdinitiator:~ # mkdir /iscsi_share
root@bsdinitiator:~ # mount -t ufs -o rw /dev/da1p1 /iscsi_share

If there is already a filesystem resident on the device, it only needs to be mounted after the iSCSI session is connected. Back on the iSCSI Target machine, it is possible to see all of the iSCSI Initiators connected

root@bsdtarget:/dev # ctladm islist
ID   Portal                   Initiator name                                                 Target name
6    192.168.22.136   iqn.1994-09.org.freebsd:bsdinitiator      iqn.2017-02.lab.testing:basictarget

Finally, if for some reason it is necessary to disconnect the system, unmount the filesystem and use iscsictl to disconnect the iSCSI session.

root@bsdinitiator:~ # umount /iscsi_share
root@bsdinitiator:~ # iscsictl -R -t iqn.2017-02.lab.testing:basictarget

There is much more to explore with iSCSI, this is just the very beginning, but it serves a a model and a starting point for this work. More to come in the future!

Update: Windows iSCSI Initiator

I am not a very savvy Windows user, and I am very new to Windows Server. I have just started to learn some of the basics. As such, I thought I’d try setting up a Windows Server 2016 host as an iSCSI Initiator. I won’t go into much detail other than what is required for setting up the iSCSI parts. Go ahead and fire up Server Manager.

initiator1

From the “Tools” menu, select iSCSI Initiator. It is also possible to start this application from the Windows search tool by searching for “iSCSI Initiator”. As shown below, when running it for the first time, Microsoft’s iSCSI service may not be running. If not, start it up!initiator2

There are many options for configuring the iSCSI Initiator, but for demonstration purposes we’ll cover the basic case. In the Target box, enter the IP address of the iSCSI Target machine and click on the Quick Connect button.

initiator4

A screen should pop-up window finding the IQN for the iSCSI Target service, and it should also state somewhere that the Login was successful.

initiator6

After closing out the pop-up window, the target should now be in the Discovered targets area of the Targets tab.

initiator5

Next go to the Volumes and Devices tab. Unless you know the exact mount point for the iSCSI volume, the best bet is to click the Auto Configure button which will get the data from the iSCSI Target, as shown below.

initiator7

I bet you wouldn’t have memorized that!  Both LUN 0 and LUN 1 are recognized by Windows. Press OK to exit out of the iSCSI Initiator application. Next, open up the Disk Management application on the Windows Server.

initiator8

Notice that two new 5 GB disks are present. The tricky part here is that I am not sure which is LUN 0 and which is LUN 1. My best guess is that the disk that is recognized as a healthy primary partition is LUN 0 which contains a GPT label and is UFS formatted.  Thus the unrecognized disk must be LUN 1, which was not modified by the FreeBSD iSCSI Initiator. In reality I would deploy two iSCSI portal groups, one for the FreeBSD iSCSI Initiators and one for Windows iSCSI Initiators. I might have a third portal group for shared volumes.

For the unrecognized volume, create a MBR partition with the Disk Management tool, and then create a FAT32 partition on this disk as well. I decided to name the partition ISCSI_BSD. As shown below, on this Windows Server, the E: drive is now LUN 1, or /dev/ada2 back on my FreeBSD iSCSI Target machine.

initiator9

The iSCSI drive shows up as a regular drive in Windows Explorer, as shown below.

initiator10

Inside I created a special message for viewing from the FreeBSD side:

initiator11

Finally, it is possible to verify the Windows access using the FreeBSD iSCSI Initiator. Reload the iSCSI Target session data, and now /dev/da2 is available on the FreeBSD Initiator. Even nicer, the FAT32 partitions are recognized by FreeBSD–less work to do! On Windows Server an MBR partition was created, which shows up as /dev/da2p1 in FreeBSD, and the actual FAT32 data partition is /dev/da2p2.

root@bsdinitiator:/iscsi_win_edrive # iscsictl -R -t iqn.2017-02.lab.testing:basictarget
root@bsdinitiator:/iscsi_win_edrive # iscsictl -A -p 192.168.22.128 -t iqn.2017-02.lab.testing:basictarget

root@bsdinitiator:~ # ls -l /dev/da2*
crw-r—– 1 root operator 0x72 Mar 4 22:32 /dev/da2
crw-r—– 1 root operator 0x77 Mar 4 22:32 /dev/da2p1
crw-r—– 1 root operator 0x78 Mar 4 22:32 /dev/da2p2
root@bsdinitiator:~ # mount_msdosfs /dev/da2p2 /iscsi_win_edrive
root@bsdinitiator:~ # cd /iscsi_win_edrive/
root@bsdinitiator:/iscsi_win_edrive # ls
$RECYCLE.BIN hello.txt
System Volume Information
root@bsdinitiator:/iscsi_win_edrive # cat hello.txt
Hello FreeBSD! This is Windows Server!
I made your /dev/ada1 into a FAT32 partition.
I call it E: Drive. Thank you!

It works! I can see the message from Windows land.

Advertisements

FreeBSD Major/Minor Version Upgrades

Last year at some point I installed FreeBSD 10.2-RELEASE on an old i686 computer. In the time between now and then, there was a 10.3 minor release as well as an 11.0 major release. I am not in a hurry to get to FreeBSD 11.0-RELEASE yet, but I thought it was time to make the minor version upgrade to 10.3-RELEASE. Fortunately, upgrading FreeBSD is an easy process.

The current version of FreeBSD installed on the machine is shown below.

root@bsdbox:~ # uname -a
FreeBSD bsdbox 10.2-RELEASE FreeBSD 10.2-RELEASE #0 r286666: Wed Aug 12 19:31:38 UTC 2015 root@releng1.nyi.freebsd.org:/usr/obj/usr/src/sys/GENERIC i386

To upgrade to the next minor version, use the freebsd-update command with the upgrade option and the release version.

root@bsdbox:~ # freebsd-update -r 10.3-RELEASE upgrade

Depending on your system, this command could take a while to run while it inspects the system. On this i686 machine, I just let the command run and came back after an hour or so to check on its progress.

Looking up update.FreeBSD.org mirrors… 4 mirrors found.
Fetching metadata signature for 10.2-RELEASE from update6.freebsd.org… done.
Fetching metadata index… done.
Fetching 1 metadata files… done.
Inspecting system…

The following components of FreeBSD seem to be installed:
kernel/generic src/src world/base

The following components of FreeBSD do not seem to be installed:
world/doc world/games

Does this look reasonable (y/n)?

After inspecting the system, the upgrade process is just making sure I am happy with the parts that will be upgraded. Enter “y” to continue, and then if you have an i686 machine, walk away and give it some time to fetch the updates.

Fetching metadata signature for 10.3-RELEASE from update6.freebsd.org… done.
Fetching metadata index… done.
Fetching 1 metadata patches. done.
Applying metadata patches… done.
Fetching 1 metadata files… done.
Inspecting system… done.
Fetching files from 10.2-RELEASE for merging… done.
Preparing to download files…
…(output truncated for the sake of sanity)…
590….39600….39610….39620….39630….39640….39650….39660….39670….39680….39690….39700….39710….39720….39730….39740….39750….39760….39770….39780….39790….39800….39810….39820….39830….39840….39850….39860….39870….39880….39890….39900….39910….39920….39930….39940….39950….39960….39970….39980….39990….40000….40010….40020….40030 done.
Applying patches… done.
Fetching 873 files… done.
Attempting to automatically merge changes in files… done.

The following file could not be merged automatically: /etc/ssh/ssh_config
Press Enter to edit this file in vi and resolve the conflicts
manually…

I had made some modifications to /etc/ssh/ssh_config on my baseline system, so rather than just overwriting the file, FreeBSD is offering me the opportunity to look at a diff of my original and the new version that FreeBSD needs to install. Luckily for me, I had just added one line, and I no longer needed it, so I edit out the changes and let FreeBSD move onward. Once you save the changes and exit out of the editor, the upgrade process continues.

…(truncated)…
/var/db/etcupdate/current/root/.k5login
/var/db/etcupdate/current/root/.login
/var/db/etcupdate/current/root/.profile
/var/db/etcupdate/log
/var/db/mergemaster.mtree
/var/yp/Makefile.dist
To install the downloaded upgrades, run “/usr/sbin/freebsd-update install”.

Excellent! The updates have been downloaded, my system has been examined, and all of the necessary modifications have been merged into the software to install. To initiate the actual install:

root@bsdbox:~ # freebsd-update install
Installing updates…
Kernel updates have been installed. Please reboot and run
“/usr/sbin/freebsd-update install” again to finish installing updates.

At this point, the system is ready to be rolled over to the next version of FreeBSD. Reboot the system:

root@bsdbox:~ # shutdown -r +1

Once the system comes back up, check out the version:

root@bsdbox:~ # uname -a
FreeBSD bsdbox 10.3-RELEASE-p11 FreeBSD 10.3-RELEASE-p11 #0: Mon Oct 24 18:47:18 UTC 2016 root@amd64-builder.daemonology.net:/usr/obj/usr/src/sys/GENERIC i386

Nice! To finalize the install, run the freebsd-update command with the install option as directed in the output before the last system reboot.

root@bsdbox:~ # freebsd-update install

That is it, the system is now upgraded from FreeBSD 10.2 to 10.3! An upgrade to a major version is no different from the above, just specify the proper version. For example, to upgrade to FreeBSD 11.0:

root@bsdbox:~ # freebsd-update -r 11.0-RELEASE upgrade

Easy!

FreeBSD and Iomega ZIP Drives

Iomega ZIP drives and disks are dead technology, but I have an old box from 1999 that has a fully functioning 100MB ZIP drive. Out of the box, FreeBSD supports these devices through the SCSI subsystem. During the boot process, FreeBSD recognizes the drive as /dev/da0.

da0 at ata0 bus 0 scbus0 target 1 lun 0
da0: <IOMEGA ZIP 100 12.A> Removable Direct Access SCSI device
da0: 11.100MB/s transfers (PIO3, ATAPI 12bytes, PIO 65534bytes)
da0: Attempt to query device size failed: NOT READY, Medium not present

The last line is not a critical failure, it just means that there is no ZIP disk inserted in the drive. Rebooting the machine with a ZIP disk inserted results in the following output:

da0 at ata0 bus 0 scbus0 target 1 lun 0
da0: <IOMEGA ZIP 100 12.A> Removable Direct Access SCSI device
da0: 11.100MB/s transfers (PIO3, ATAPI 12bytes, PIO 65534bytes)
da0: 96MB (196608 512 byte sectors: 255H 255S/T 3C)

The last line shows that the ZIP disk media was recognized, and the number of 512-byte blocks is listed. A disk can be inserted at any time, and interestingly enough, when a disk is inserted into the driver, there are no messages in /var/log/messages indicating which partition the media is listed as. Running a listing on the devices in /dev can reveal some hints though.

root@bsdbox:/media # ls /dev
acpi consolectl kbd0 pci ttyv4
….
… da0 …
… da0s4 …

The drive represents the ZIP media on /dev/da0s4. According to older documentation from FreeBSD 6, slice 4 is where the device driver places the media partition.  To access the ZIP disk, first create a mount point on the filesystem, such as /media/zipdisk, and then mount the device using the mount(8). The zip disks I have are FAT formatted, so I need to use the “-t msdosfs” option.

root@bsdbox:/meda # mkdir /media/zipdisk
root@bsdbox:/meda # mount -t msdosfs /dev/da0s4 /media/zipdisk

This disk is now available on /media/zipdisk for reading and writing.  When finished with the disk, before physically ejecting it, be sure to unmount the filesystem first.

root@bsdbox:/meda # umount /media/zipdisk

Now it is safe to press the disk eject button on the drive. Of course for frequent use of a ZIP drive, entries should be added to /etc/fstab to make things easier. For infrequent use, however, the above methodology should work just fine!

Disable console bell in FreeBSD

I use Bash and frequently utilize tab completion. On my FreeBSD box, when there are multiple options during tab completion, the annoying console bell always obnoxiously makes itself heard. For C programmers, it is the ‘\a’ that you included in your printf() calls when first learning C to drive others crazy.

To disable it, as root user you can run:

root@bsdbox# sysctl hw.syscons.bell=0
hw.syscons.bel;: 1-> 0

Add the “hw.syscons.bell=0” command to /etc/sysctl.conf to permanently disable it.  Good riddance!

WindowMaker as the X11 window manager

My light-weight window manager of choice is WindowMaker.  It is simple, easy to modify, and stays out of the way.  The default window manager installed with X11 on FreeBSD, however, is twm.  I first experienced twm in the Virginia Tech Computer Science Dept. UNIX lab (all FreeBSD boxes), and tvm looks exactly the same today.  I cannot pick on it too much, however.  The guy in the office next door to me at work uses tvm on his Linux box.  I suppose I ought to call it an Emacs box, since he does everything out of emacs.  I digress…

Please read this excellent tutorial on xinit and .xinitrc files from the Fluxbox project first.  The tutorial explains very clearly how the xinit program works and how .xinitrc files are processed.  In short, however, once the end of .xinitrc is reached, X11 is shutdown.  Thus you need a blocking command in xinitrc scripts that prevents the script from exiting.  In the default xinitrc script,  /usr/X11R6/lib/X11/xinit/xinitrc, I noticed towards the bottom of the script:

twm &

It was followed by three xterm commands. The first two were also run in the background, but the final xterm was not run in the background, thus preventing X11 from terminating until that terminal is closed by the user.  No more twm on this box though!  I removed the xterm commands, and changed the twm line to the following:

wmaker

Now, WindowMaker is launched and until the user exits the WindowMaker session X11 will continue to run. All users on my FreeBSD will have WindowMaker when the execute startx.  I also created a .xinitrc in my user account home directory as follows:

exec wmaker

In reality, however, I have XDM to run as my “magic client” (see article linked above). Instead of managing a separate .xinitrc and .xsession file, I just symlinked the files together so that any changes are available in both files, in the cast that I ever disable XDM.

ln -s .xinitrc .xsession

After I log into the system via XDM, my .xsession file is executed and WindowMaker is launched.

Setting up XDM

The X Display Manager, or XDM, is a program that allows one to log into a system using a GUI program.  Rather than logging into a system and then manually starting X, X comes up after boot and is ready to go.  While XDM (or GDM for GNOME or KDM for KDE) is not necessary on server systems, any system where X Windows is used on a regular basis benefits from XDM.  And XDM is easy to set up too.  With FreeBSD, one can simply build and install from the ports collections:

cd /usr/ports/x11/xdm
make install clean

Now one just needs to modify /etc/ttys to enable XDM. The XDM entry is already in this file, it is just disabled on the default install. To enable XDM after installing it, change “off” to “on” between the xterm and secure parameters.

ttyv8 "/usr/local/bin/xdm -nodaemon" xterm on secure"

Unlike GDM or KDM, XDM does not allow one to select a window manager. The window manager should be executed from a .xsession script in your home directory. I setup my system so that .xsession is a symlink to a .xinitrc file (read by startx when manually starting X11) in my home directory. If you need XDM and startx to have different startup methods, then you must provide separate files with customization for each program.

Without a .xsession file, XDM and startx default to /usr/X11R6/lib/X11/xinit/xinitrc. With my default install of FreeBSD, at the bottom of the script, the window manager twm is started, along with three xterms which are opened by default.

I cannot stand twm, so next I will look at setting up a different light-weight window manager. As my experience with Ubuntu and GNOME on this machine proved, I don’t have the memory on this machine for desktop environments like GNOME or KDE.

nVidia graphics driver on FreeBSD 8.2

Update January 2016: Driver version is now 96.43.23…doesn’t build in FreeBSD 10.2. My initial guess is the clang system is catching stuff that gcc did not in the driver…

First of all, find the proper driver for your device.  I used nVidia’s driver download page to get the right version: 96.43.20.  My card is an nVidia GeForce4 MX 420, and it is not supported on higher version number drivers that support more modern video cards.

Download the driver tarball with your web brower or wget and unarchive it.

cd /root
tar -xzf NVIDIA-FreeBSD-x86-96.43.20
cd  /root/NVIDIA-FreeBSD-x86-96.43.20
vi doc/README

As you can see in the documentation, you just need to do a ‘make install’.  Before doing so, however, comment out lines 25 through 36 of src/nv-freebsd.h as this check will prevent you from being able to build the driver.  (I don’t know why this check was in here…perhaps on more modern drivers this is not necessary)

/*
#if __FreeBSD_version >= 800000
#error This driver does not support FreeBSD 8.x/-CURRENT!
#endif
#if __FreeBSD_version >= 700000 && __FreeBSD_version < 700055 #error This driver does not support FreeBSD 7.x/-CURRENT! #endif #if __FreeBSD_version >= 600000 && __FreeBSD_version < 600034
#error This driver does not support FreeBSD 6.x/-CURRENT!
#endif
#if __FreeBSD_version < 503000
#error This driver requires FreeBSD 5.3 or later!
#endif
*/

Build and install the driver:

make install

If the driver builds with no errors, the driver kernel module should be installed in /boot/modules.  To test the driver and make sure that everything works,

I used the nvidia-xconfig tool to create a basic xorg.conf file. The tool automatically detected my video card and setup the correct information in the Device section.

Section "Device"
    Identifier     "Device0"
    Driver         "nvidia"
    VendorName     "NVIDIA Corporation"
    BoardName      "NV17 [GeForce4 MX 420]"
    BusID          "PCI:1:0:0"
EndSection

The tool also added additional required modules by the driver.

Section "Module"
	Load  "dbe"
	Load  "dri"
	Load  "dri2"
	Load  "extmod"
	Load  "glx"
	Load  "record"
EndSection

I then had to customize the monitor settings. I have a wide-screen monitor that I like to run at 1920×1200. I tried explicitly setting this in the Modes subsection of the Screen section, but was unable to get the setting to work (though 1024×768 worked with no problems). Apparently, X is smart enough these days to figure out the max resolution on its own, so I simply commented out the Modes subsection.

Section "Screen"
    Identifier     "Screen0"
    Device         "Device0"
    Monitor        "Monitor0"
    DefaultDepth    24
    SubSection     "Display"
        Depth       24
#        Modes      "1920x1200" "1024x768" "800x600" "640x480"
    EndSubSection
EndSection

I setup the Monitor section with more descriptive names as well as putting in the exact horizontal sync rate and vertical refresh rate from the instruction manual for my Samsung display.

Section "Monitor"
    Identifier     "Monitor0"
    VendorName     "Samsung"
    ModelName      "Syncmaster P2250"
    HorizSync       30.0 - 81.0
    VertRefresh     56.0 - 60.0
    Option         "DPMS"
EndSection

To test the configuration, use the -config option on the X program, passing it the customized X configuration.

X -config /root/xorg.nvidia.conf

Once satisfied with the settings, I copied the config file to /etc/X11/xorg.conf and tried to start X Windows.

cp /root/xorg.nvidia.conf /etc/X11/xorg.conf
startx   # ugly tvm should start

Finally, with the driver working, it can be added to load automatically during boot, allowing the use of graphical log-in programs such as XDM.  To load the driver during the boot stage, add the following to /boot/loader.conf:

nvidia_load="YES"

Next I shall look at configuring XDM so that I don’t have to always manually start X.