Dell Perc 5i RAID Controller - Two Drives Show Up as Foreign

I recently had a strange problem to troubleshoot. A client's ESXi 5.1 box crashed hard. Unfortunately, the server (a Dell PowerEdge 2950 g3) had no iDrac card.

When I arrived, I discovered that someone had unplugged two of the drives in a RAID 5 set of 5 drives. There was a hot spare, but both drives were pulled at the same time (long story.) The RAID 5 volume was down. I looked into the Perc BIOS. Two of the five drives now showed up as foreign. While I've certainly imported foreign sets, I've rarely seen this situation on some of the drives and not all of them.

When a foreign disk is detected in the Dell Perc BIOS (this is, of course, a rebadged LSI RAID card), a new tab shows up - foreign. I went into the foreign tab and imported the configs from both drives. The volume was now online (rebuilding, of course.) I waited a while then let ESXi boot. I stopped the VMs from starting and waited till the rebuild finished, just to be safe - a power outage at that moment might have been particularly bad (this building has unreliable power and the UPS can only keep a server up so long, of course.) After the disks had quieted down, I started the VMs. There was some data loss, of course, and the startup took over an hour (there was a snapshot in place - so the hard crash probably damaged the snapshot). However, the server was back. There's something to be said for lockable bezels.

Nagios MS SQL Server Plugin

I recently attempted to use the  check_mssql_database.py plugin. It doesn't give much feedback for errors. I was typically seeing something like this while attempting to connect to SQL Server 2005 from a Nagios Xi installation:

CRITICAL:Unable to access SQL Server.

I did some research, including looking at the Python script. After looking into the DB connection method, I discovered that I had to configure the freetds.conf file (in /etc, in my case.)

I added a stanza like this (you might need a more exotic configuration or configurations):

[MYSERVERNAME]

     host = myservers_IP_ADDRESS

     port = my_port_number 

     tds version = 7.0

After doing this, I was able to connect (not all the checks return sane values, but that's a problem for a different day.) My connection string goes like so:

check_mssql_server.py -H 'hostname_or_IP'    --warning 20 --critical 30 -U 'MYDOMAIN\username' -P 'MYPASSWORD' --deadlocks

You can specify instance with -I, but I was using the default instance of 'MSSQLSERVER'

The plugin's install document is here:

Install doc

Edit: it would appear that most of the tests are not going to be useful. Quite a few of them return nonsensical, large values and other return the same value consistently, no matter what the conditions on the server.

Also, Nagios Xi does something funny where it auto escapes the \ in the domain\username combo. The plugin ends up running DOMAINNAME\\username (which it can't handle.) I tried removing the quotes and several other things.

I ended up finally modifying the plugin to add my domain name to the user name and then I used the short username in Nagios Xi without the domain preface. If I knew Python better, I would have had it strip off the extra \ - but I had to get it working quickly. I might go back and make that change later.

 To be honest, I'm not very familiar with Nagios Xi - having used the original Nagios since 2002 or so, so it could be something I'm overlooking. 

Issues with Tape Changer in Bacula on FreeBSD

  I have an old Dell PowerVault 124T I use for home backups. It's a bit noisy, so I have it attached to an old APC master switch so I can turn it on when I want to do a backup. The catch is that the FreeBSD server I'm using might have been rebooted while the LTO array was off. So, I use camcontrol:


sudo camcontrol devlist

turns up nothing...

sudo camcontrol rescan all

now, we have an LTO drive and a changer:

            at scbus1 target 9 lun 0 (pass0,sa0)
               at scbus1 target 9 lun 1 (pass1,ch0)

So far, so good. A restart of bacula-sd should pick up the device, if needed. 

sudo service bacula-sd restart

However, I discovered that bacula couldn't see the slots in the changer when I did an update barcode slots

So, I checked the status of the storage daemon:

3301 Issuing autochanger "loaded? drive 0" command.
3991 Bad autochanger "loaded? drive 0" command: ERR=Child exited with code 127.
Results=mtx: not found

mtx not found! It was installed from ports, so I did a simple fix of 

ln -s /usr/local/bin/mtx /usr/bin

and now the command worked:

*update barcode slots
Automatically selected Catalog: MyCatalog
Using Catalog "MyCatalog"
The defined Storage resources are:
     1: File
     2: LTO3
Select Storage resource (1-2): 2
Connecting to Storage daemon LTO3 at fbsd1:9103 ...
3306 Issuing autochanger "slots" command.
Device "Autochanger" has 8 slots.
Connecting to Storage daemon LTO3 at fbsd1:9103 ...
3306 Issuing autochanger "list" command.
Catalog record for Volume "000150L3" updated to reference slot 1.
Catalog record for Volume "000186L3" updated to reference slot 2.
Catalog record for Volume "000140L3" updated to reference slot 3.
Catalog record for Volume "000138L3" updated to reference slot 4.
Catalog record for Volume "000137L3" updated to reference slot 5.
Catalog record for Volume "000131L3" updated to reference slot 6.
Catalog record for Volume "000136L3" updated to reference slot 7.
Catalog record for Volume "000162L3" updated to reference slot 8.
*exit

Coping with the "the directory service was unable to allocate a relative identifier" Error in Widnows

The error was:

 the directory service was unable to allocate a relative identifier


I had a client with this problem recently. He was trying to add an XP machine to his Windows 2003 based Active Directory domain.

Since he was unable to get a relative identifier from the RID master on the domain, his computer account could not be created. I tested and verified that no objects could be created, including new user accounts.

The corresponding technet article is:

KB 822053

There were several AD related errors in the event logs on the domain controllers, including event id 2042:

It has been too long since this machine last replicated with the 
named source machine. The time between replications with this source 
has exceeded the tombstone lifetime. Replication has been stopped 
with this source. 

The corresponding technet article is:

Event ID 2042

Looking at the domain controllers, it appeared that one of the domain controllers (call it domain controller B) seemed to be able to replicate to the other domain controller (domain controller A) , but not vice versa. The client said that they had discovered a firewall running on one of the domain controllers (DC B) and had turned it off. But, replication had worked from A to B (A was the RID master) in over a year.

  I checked replication with repadmin and did not find any lingering objects. After several false starts, I used the procedure (and I backed up the system state on both domain controllers, just to be safe) at the bottom of the technet article:

To restart replication following event ID 2042

1. Click Start, click Run, type regedit, and then click OK.
2. Navigate to HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\Services\NTDS\Parameters
3. In the details pane, create or edit the registry entry as follows:
   
   If the registry entry exists in the details pane, modify the entry as follows:
   1. In the details pane, right-click Allow Replication With Divergent and Corrupt Partner, and then click Modify.
   2. In the Value data box, type 1, and then click OK.
   If the registry entry does not exist, create the entry as follows:
   
   1. Right-click Parameters, click New, and then click DWORD Value.
   2. Type the name Allow Replication With Divergent and Corrupt Partner, and then press ENTER.
   3. Double-click the entry. In the Value data box, type 1, and then click OK.

Reset the Registry to Protect Against Outdated Replication

When you are satisfied that lingering objects have been removed and replication has occurred successfully from the source domain controller, edit the registry to return the value in Allow Replication With Divergent and Corrupt Partner to 0.

After restarting the ntfrs service on both domain controllers and forcing replication, replication between A and B started working correctly and I was able to join the machine to the domain.

VMware Workstation 9 and Windows 8 - Hyper-V issues

I recently installed Windows 8 on a laptop, as well as VMware workstation 9. Later on, I added Hyper-V roles. When I tried to power Workstation back up, I got the following error message:

VMware Workstation and Hyper-V are not compatible. Remove the Hyper-V role from the system before running VMware Workstation.


Removing the  Hyper-V role, entirely, seems to suffice. I tried removing the Hyper-V role and shutting down the Hyper-V management service without rebooting, but that did not help. After rebooting, the Hyper-V management service was gone and VMware Workstation could launch VMs properly again.

Updating Firmware on HP C7000 Blade Chassis

I recently had to update an HP C7000 series blade chassis. The onboard administrators were both on 1.3 firmware, and a new G7 blade required firmware in the late 2.x line of firmware. I had seen discussions of whether it was safe to upgrade only the OA firmware and not the blades. I contacted HP support, and they said that though while not ideal, you could safely update just the OA firmware.

I first ran the HP Proliant Service Pack ISO: http://h18004.www1.hp.com/products/servers/management/spp/index.html

The HP SUM utility was not able to upgrade the firmware, due to a web API error. I tried several times. The fix that worked for me was to download the last of the Smart Update ISO images, here:


http://h18004.www1.hp.com/products/blades/bladesystemupdate.html

I then used that version of HP SUM to update the C7000 OA to version 9.3. From there, I was able to update the firmware using the Proliant SP disk I linked above. I also used the SP disk to update the ILO's on the blades. I didn't update the firmware on the interconnect devices (in my case, a Cisco switch and a Brocade fibre switch) nor did I update the bios of the blades, although that was certainly a possibility.

RSA Appliance: Administrative Tasks

I've decided to post a bunch of quick tips about navigating around the appliance - mainly because I don't do it that often and quickly forget. I'll add more as time permits.

• To change the network settings of the appliance (DNS, gateway, etc.)
• go to https://yourappliance:7072/operations-console
• go to administration -> networking -> configure network settings