Mellanox ConnectX-2 10GB Interface on FreeBSD 10.2

Although Mellanox's FreeBSD driver for the ConnectX-2 is included in the kernel source, the kernel modules are not included with the generic kernel. To use this card, build a custom kernel.

1. update the FreeBSD source using freebsd-update

sudo freebsd-update fetch
sudo freebsd-update install

2. copy the generic kernel config to a custom kernel
cd /usr/src/sys/amd64/conf

sudo cp GENERIC MYKERNEL01

4. edit the config file (sudo vi MYKERNEL01.) Add these lines to the bottom:

#### Mellanox ConnectX-2 support
options  OFED
options  IPOIB_CM
device  ipoib
device  mlx4ib
device  mlxen 

5. Compile the kernel and install it

cd /usr/src
sudo make buildkernel KERNCONF=MYKERNEL01
sudo make installkernel KERNCONF=MYKERNEL01

6. reboot

sudo shutdown -r now

Your new interface should show up when the machine comes back:

mlxen0: flags=8802 metric 0 mtu 1500
options=d07bb
ether 00:02:c9:52:ad:23
nd6 options=29
media: Ethernet autoselect (autoselect )
status: active


You can configure however you'd like (IP, MTU, etc.)

If you run freebsd-update on a regular basis, you may want to append a  kernel rebuild (step #5) to a script so you can rebuild the kernel automatically if there is a kernel source update.

FreeBSD NAS/SAN with ZFS (part 1)

Part 1: Initial Setup

I had to set up a NAS/SAN recently using a Dell PowerEdge 1900. The ultimate goal is to have CIFS (through Samba 3.x), NFS, and iSCSI support. With so few drives, performance will not be amazing and 8GB of RAM is a bit low for ZFS, but I needed cheap, fairly reliable storage. I can recover from a system crash fairly easily, and I can restore the data from backups, if necessary. I generally advise against using large drives for RAID devices because of the long rebuild times in the event of failure. However, this system is designed to provide cheap storage, so RAIDZ2 will have to do.

The pros of this sytem:

• Cheap
• Easy to set up
• fairly reliable (though there are single points of failure) - RAIDZ2 will permit two failures before data loss
• reasonable performance for a fileserver
• flexible - the system has the entire ports tree

The cons:

• at least two single points of failure (a single system drive, albeit a SSD and a single power supply, though I will be rectifying the power supply issue shortly)
• FreeBSD claims the system drive does not support TRIM. If I had chosen a traditional platter based drive, this wouldn't matter. This means I might see some performance issues on the system drives after a certain amount of time. Since I won't be doing much on the system drive, this may not be much of an issue eve
• Performance will be mediocre with so few spindles (I'd love to have 15 or more in the pool, but I don't have an external JBOD array available)
• rebuild time will be long, due to the 2TB drives
•  no dedicated hot spare (no free ports on the SAS controller)

It's important to note that Nexenta community edition is an option, as is FreeNAS (I've ruled out OpenFiler because I've had so many issues with it over the years.) However, I had the following issues:

1. Nexenta CE is not to be used for production systems
2. FreeNAS is on an older revision of ZFS than FreeBSD proper

The system is configured like so (it's not the greatest in terms of high availability):

• 1x Intel Xeon E5335 processor (quad core @2GHz)
• 8GB RAM (FB-DIMMs)
• 1 Dell SAS 5i card (LSI based)
• 7x Hitachi 2TB 7,200RPM SATA drives
• 1x OCZ Vertex 30GB SSD 
• 2x Intel Gigabit server NICs

It's important to note that the chassis really only supports 6 3.5" drives, so I had to use drive brackets to mount the remaining 2TB drive & SSD. It would be best to use a pair of SSD drives (the SSD drive is for the base OS) but I didn't have any free ports left on the SAS controller. I can highly recommend that you do a pair of drives for redundancy.

Alternatively, I could use the on board SATA, but I'm actually okay with the system functioning like an appliance (i.e., I'll back up base configuration (/etc /usr/local/etc) and the ZFS pools.) You can, of course, use a USB stick for the base OS, if you'd like. Ultimately, I'd probably want to use SSDs for the caches, but I ran out of drive

The steps were:

1. Download FreeBSD 9.0 AMD64 and burn to a DVD
1. DVD 1 ISO
2. Or... use the memory stick version
3. or... use the boot only version and do a net install
2. Boot the machine  from the DVD
3. Install the system. I prefer to partition unix systems if possible. I really dislike have an errant log file filling up the single filesystem. It's certainly convenient to not have to carve out space, but if I got that route, I prefer to have a properly managed filesystem like ZFS so I can create new volumes and set quotas on them

 I partitioned it like so:

Filesystem           Size    Used   Avail Capacity  Mounted on
/dev/da0a              2G    356M    1.5G    19%    /
devfs                1.0k    1.0k      0B   100%    /dev
/dev/da0d            503M    4.1M    459M     1%    /tmp
/dev/da0e            4.9G    303M    4.2G     7%    /var
/dev/da0f            7.9G    2.5G    4.7G    35%    /usr
/dev/da0g              2G     16M    1.8G     1%    /home

Your needs may vary. I enabled TRIM support on all the filesystems, but FreeBSD complains about that and claims that the drive does not support it. I'm pretty sure it does. I'll likely have to address this issue later.

After partitioning and installing the OS, I set up user accounts, used freebsd-update fetch and freebsd-update install to apply security patches, and finally used portsnap to create and update the ports tree.

3a. set up necessary components in /etc/rc.conf:

zfs_enable="YES"

 4. I set up all the necessary networking components I wanted, such as NTPD and NIC teaming/bonding (failover only.)

5. Zpool creation:

These 2TB drives have 4K sectors, but emulate 512bytes for maximum OS compatibility. I wanted to align the ZFS pool to match the 4K sectors for optimal performance. I found numerous discussions online, but this page was the most straightforward, at least for my purposes.

http://ivoras.net/blog/tree/2011-01-01.freebsd-on-4k-sector-drives.html

alternatively, there is this howto:

http://www.aisecure.net/2012/01/16/rootzfs/

basically, you create GNOP providers (man page) with 4096k sector sizes, export the ZFS pool, destroy the gnop providers and re-import the pool.

Here's the list of drives:

sudo camcontrol devlist
Password:
              at scbus0 target 0 lun 0 (da0,pass0)
        at scbus0 target 1 lun 0 (da1,pass1)
        at scbus0 target 2 lun 0 (da2,pass2)
        at scbus0 target 3 lun 0 (da3,pass3)
        at scbus0 target 4 lun 0 (da4,pass4)
        at scbus0 target 5 lun 0 (da5,pass5)
        at scbus0 target 6 lun 0 (da6,pass6)
        at scbus0 target 7 lun 0 (da7,pass7)
   at scbus2 target 0 lun 0 (cd0,pass8)
 

 I created my pool like so:

# use a for loop, if you'd like
# If I were provisioning many more than 7 drives, I'd probably just do a for loop, too
sudo gnop create -S 4096 /dev/da1
sudo gnop create -S 4096 /dev/da2
sudo gnop create -S 4096 /dev/da3
sudo gnop create -S 4096 /dev/da4
sudo gnop create -S 4096 /dev/da5
sudo gnop create -S 4096 /dev/da6
sudo gnop create -S 4096 /dev/da7
sudo zpool create dpool1 raidz2 /dev/da1.nop /dev/da2.nop /dev/da3.nop \

  /dev/da4.nop /dev/da5.nop /dev/da6.nop /dev/da7.nop
sudo zpool export dpool1
sudo gnop destroy /dev/ /dev/da1.nop /dev/da2.nop /dev/da3.nop \

  /dev/da4.nop /dev/da5.nop /dev/da6.nop /dev/da7.nop
sudo zpool import dpool1

sudo zdb -C dpool1 | grep ashift

should return 12...

I have a working pool:

sudo zpool



> sudo zpool status dpool1
  pool: dpool1
 state: ONLINE
 scan: scrub repaired 0 in 0h0m with 0 errors on Sat Jun 30 11:32:09 2012
config:

        NAME        STATE     READ WRITE CKSUM
        dpool1      ONLINE       0     0     0
          raidz2-0  ONLINE       0     0     0
            da1     ONLINE       0     0     0
            da2     ONLINE       0     0     0
            da3     ONLINE       0     0     0
            da4     ONLINE       0     0     0
            da5     ONLINE       0     0     0
            da6     ONLINE       0     0     0
            da7     ONLINE       0     0     0

errors: No known data errors

I did some benchmarks. Performance was acceptable for my purposes. I'd really want to get the spindle count much higher if I were using for something like a DB server or a large hypervisor.

The next section in this series will cover the teaming/bonding in FreeBSD.

Nagios Group Service Checks and Exclusions

Using hostgroup_name in service checks is very, very useful. However, I find that I often want to include the Nagios server in one of the hostgroups, but don't necessarily want to configure nrpe for the nagios host. An easy way around this is to use Nagios's exclude feature. You can exclude a group (or host) by prefacing it with "!"

For instance:

1. create a group that contains only your nagios host

define hostgroup {

        hostgroup_name localhost-monitor-hosts

        members         mynagiosserver

}

2. create the service:

define service {

        hostgroup_name          unix-hosts,!localhost-monitor-hosts

        use                             critical-service

        service_description             check_disk

        check_command                   check_nrpe!check_disk!80!85

        check_freshness                 1

        register                        1

        }

This will check all hosts in the unix-hosts group, excluding any in localhost-monitor-hosts. You can obviously create multiple groups, each with a different purpose.

Note, you can also define a service with more than one host, and exclude hosts that way:

define service {

         host_name www1, www2, !www3

         use generic-service

         service_description check_ping

         check_command check_ping!100.0,20%!500.0,60%

}

but I find that more taxing, as you have to maintain the host_name... which is kind ignoring the whole point of groups.

Nagios, FreeBSD, and Lilac

In the earlier post, I talked about setting up nconf on FreeBSD. I decided to give Fruity/Lilac a try. It's been around longer, anyway, and it supports imports of existing configs. It also can use nmap to generate a basic configuration.

Anyway, the rough steps are:

1. install mysql50-server

2. install apache22

3. install php5

4. install php5-extensions with the following modules enabled:

json, pcntl, posix, mysql, curl

5. install php5-pdo_mysql

6. install nmap

7. untar the lilac source to /usr/local/www/apache22/data/lilac

8. sudo chown www:www /usr/local/www/apache22/data/lilac/includes

9. restart apache.

Note: in order to get apache to work with php, I created a php config file in /usr/local/etc/apache22/Includes/php.conf:

DirectoryIndex index.php index.html index.htm
AddType application/x-httpd-php .php .htm .html
AddType application/x-httpd-php-source .phps

Nagios, FreeBSD, and nconf

I just noticed nconf on the Nagios.org site the other day. Another GUI for configuration. Anyway, I was building a new VM for use by our consultants to run nagios, RT, and cacti, and I figured I would give it a shot.

Nconf is yet another web frontend for the configuration of Nagios. It seems to work okay. It's PHP based. It's not in the ports tree, but then again, it hasn't been around very long. Here's what I did to install it:

1. install nagios, apache, php5, php5-extensions, mysql (It can't use PostgreSQL at this point) etc.

2. Create an appropriate php5.ini in /usr/local/etc and make sure it contains at least:

short_open_tag = On
register_globals = Off
magic_quotes_gpc = Off

3. untar nconf into /usr/local/www/apache22/data/nconf

4. cd to that dir

chown www config output temp

5. sudo mysql (-p if you have a root password set at this point, if not it's a good idea)

CREATE DATABASE nconf;
GRANT SELECT, INSERT, UPDATE, DELETE, CREATE, ALTER, DROP on nconf.* to 'nconf_user'@'localhost' identified by 'difficult_password';
flush privileges;


6. add www to nagios group

7. restart apache (I also had to add:

AddType application/x-httpd-php .php .htm .html
AddType application/x-httpd-php-source .phps

to /usr/local/etc/apache22/httpd.conf)

8. Browse to http://your-host-ip-or-name/nconf and go through the setup (set the path to nagios to be /usr/local/bin/nagios)

9. remove INSTALL.php and the INSTALL directory in nconf

10. Download the default imagepak-base.tar.tar from here.)

11. rename the image pack to imagepak-base.tar.gz

12. cd /usr/local/www/nagios/images/logos
sudo gunzip /path/to/the/imagepak.tar.gz
sudo tar -xf /path/to/the/imagepak.tar

The base install is done.

It will generate a tarball with the configs in /usr/local/www/apache22/data/nconf/output. You'll need to untar it into /usr/local/etc/nagios and configure nagios.conf to parse the directories for configs.

I hope the project lasts. There have certainly been a lot of webguis for Nagios that have died. Hopefully, they will add the ability to import existing configs at some point.