Tuesday, November 27, 2018

Cisco Catalyst 3850 Upgrade Issue - Internal Error

While upgrading a pair of non-stacked 3850-48Ps, I encountered this error message:


[1]: % An internal error was encountered. Operation aborted.

I did have some space issues, so I deleted previous versions. Even that did not fix the issue. I tried installing it from tftp, to no avail.

Eventually, I ran the software install command with "new force"

software install file tftp://10.20.30.40/cat3k_caa-universalk9.SPA.03.06.09.E.152-2.E9.bin new force

Success. I rebooted both switches. One crashed on reboot, but rebooted cleanly afterward.

Sunday, November 25, 2018

Cisco 3850 WS-C3850-48P-E Switch with Wrong License Level Lanbase

I received a couple of refurbished 3850 PoE Enterprise switches. One switch was perfectly fine, with IP services enabled permanently.

The other:


Slot#  License name   Type     Count   Period left
----------------------------------------------------------
1      lanbase      permanent     N/A   Lifetime

License Level on Reboot: lanbase



I issued a request for an RMA - however, the switch was labeled correctly on the case: WS-C3850-48P-E.

On a whim, I ran this command:


license right-to-use activate ipservices



I rebooted as instructed from the switch, and now I have:

Switch#show license right-to-use
 Slot#  License name   Type     Count   Period left
----------------------------------------------------------
 1      ipservices   permanent     N/A   Lifetime
 1      lanbase      permanent     N/A   Lifetime

License Level on Reboot: ipservices


So, yay - no need for an RMA.


Friday, November 23, 2018

Upgrading pfsense 2.4.3_1 to 2.4.4 - Issue Replacing PHP 5.x with 7.x Due to FRR

I ran into a problem upgrading pfsense 2.4.3_1 to 2.4.4 due to a dependency on the old PHP 5.6.x from the older package of FRR .2_2 (Free Range Router.) The problem was that PHP56 was marked as vital due to the FRR dependency, and it was not possible to update FRR to .2_4, as FRR .2_4 needed pfsense 2.4.4 (I could not decide whether this was a modern O. Henry story or a missing epilogue from Catch 22.)

The fix:

1. connect to the pfsense server with SSH
2. Open a shell
3. Run this command:

pkg set -v 0 php56

Your upgrade should work - I had to do a couple of update attempts before it worked.

Wednesday, March 21, 2018

SSH No Matching Cipher Found With SSH to Older Cisco Gear

I've seen this error on recent SSH versions, including the OpenSSH used by OS X and others:

Unable to negotiate with 192.168.77.1 port 22: no matching cipher found. Their offer: aes128-cbc,3des-cbc,aes192-cbc,aes256-cbc

Several people suggest modifying your local ssh client config. However, I'd rather not use 3DES unless I have to. So, I created an alias and put it in .bash_profile:

alias ssho='ssh -c 3des-cbc'

after a quick . ~/.bash_profile

I can ssh with the alias:

ssho 192.168.77.1
password:

Of course, the best fix would be to update IOS on the Cisco gear.

Thursday, March 1, 2018

Openstack Ansible Pike - Cannot Create Image in Horizon

There is a bug with OpenStack Pike deployments using Ansible. When you attempt to create an image in Horizon, you will see the following error:



Followed by this entry in the images list after closing this:



You should still be able to create images via command line, but Horizon won't work. 

There is a bug filed for this:


Their fix is simple: connect to the horizon container from an infrastructure node:

lxc-attach -n `lxc-ls -1 | grep horizon | tail -1`


edit /etc/horizon/local_settings.py
and change:

HORIZON_IMAGES_UPLOAD_MODE = 'direct'

to

HORIZON_IMAGES_UPLOAD_MODE = 'remote'

After that, restart the horizon container

lxc-stop -n `lxc-ls -1 | grep horizon | tail -1`

lxc-start -n `lxc-ls -1 | grep horizon | tail -1`

And simply wait a few minutes. Image creation should work. 








Thursday, July 6, 2017

Powercenter Integration Service Won't Start: Error SF_34004 - Missing Environment Variables

I recently had an issue where I could not get the Informatica PowerCenter Integration Service to start after an install. The service generated this error code: SF_34004. There is an Informatica KB article describing this problem and the appropriate fix:

https://kb.informatica.com/solution/18/Pages/120251.aspx


Basically, it's suggesting that the Integration Service was configured to connect to the PowerCenter repo using the schema credentials and not the domain administrator.

However, I found that this did not help. the repo service was running correctly and the integration service was using the correct credentials. I checked the user running PowerCenter, and realized the INFA_HOME, INFA_DOMAIN_FILE, and other variables were not set correctly. After setting these and restarting PowerCenter, I was able to enable the Integration Service.


Friday, May 5, 2017

Abnormally Long First Boot Times on Cisco 3750x's with 15.2.x code



I recently upgraded some Cisco 3750x (WS-C3750X-48T-S to be precise) from 12.2.x code to 15.2 code (15.2(4)E4.)

The reboot took a rather long time due to microcode updates. It was about 25-30 minutes. The following messages were displayed:



ssssssssspssssssssssssssssssspssssssssssssssssssspssssssssssssssssssspssssssssssssssssssspsssssssssp
Front-end Microcode IMG MGR: Preparing to program device[0]...25166 bytes.
Front-end Microcode IMG MGR: Programming device 0...rrrrrrwssssssssssssssssssspssssssssssssssssssspp
Front-end Microcode IMG MGR: Preparing to program device[0]...86370 bytes.
Front-end Microcode IMG MGR: Programming device 0...rrrrrrwssssssssssssssssssspssssssssssssssssssspw

% Front-end Microcode IMG MGR: HW image is upgraded. MCU reset causes the switch to reload

If you're making such an upgrade, be prepared for a long wait. It will likely be worse on a stack.

While the 3750x is not a speedy switch, management-wise, this was impressively slow. The regular boot time after the upgrade was about 5-6 minutes, so be patient. Switches in a stack will take even longer to boot.

Tuesday, February 28, 2017

Android Nougat and OTG Sound Blaster E3 Sound Issues

I recently upgraded to Nougat and experienced sound issues using an OTG cable and my Sound Blaster E3 DAC. The sound was basically panned hard to the left, slower than normal, and there was some distortion.

I realized that ES File Explorer was activating each time I plugged the E3 in. The simple fix was to remove ES File Explorer. That being said, I'll probably experiment with its settings as it's a useful utility.

Sunday, October 30, 2016

Raspberry Pi and 1-Wire Temperature/Environmental Sensors

I've discussed 1-wire before. In this case, I needed a cheap temperature sensor to alert on environmental conditions in an server room.

I set up a Raspberry Pi with stock Raspian and install digitemp like so:

apt install digitemp

I used an iButtonlink usb master (here: http://www.ibuttonlink.com/products/linkusb and a t-sense sensor here:
http://www.ibuttonlink.com/products/t-sense )

The USB device shows up as /dev/ttyUSB0 as an FTDI serial port.

I can walk the 1-wire tree like so:

sudo digitemp_DS9097U -s /dev/ttyUSB0 -w
DigiTemp v3.6.0 Copyright 1996-2007 by Brian C. Lane
GNU General Public License v2.0 - http://www.digitemp.com
Turning off all DS2409 Couplers
..
Devices on the Main LAN
2149B214090000D1 : DS18B20 Temperature Sensor
01C3235419000000 : DS2401/DS1990A Serial Number iButton

That being said, you'll want to create a dot file to use this like so:

 sudo digitemp_DS9097U -s /dev/ttyUSB0 -i
DigiTemp v3.6.0 Copyright 1996-2007 by Brian C. Lane
GNU General Public License v2.0 - http://www.digitemp.com
Turning off all DS2409 Couplers
..
Searching the 1-Wire LAN
2149B214090000D1 : DS18B20 Temperature Sensor
ROM #0 : 2149B214090000D1
Wrote .digitemprc


I have one temperature sensor at the moment. After creating the .digitemprc, I can query it:

 sudo digitemp_DS9097U -s /dev/ttyUSB0 -t 0
DigiTemp v3.6.0 Copyright 1996-2007 by Brian C. Lane
GNU General Public License v2.0 - http://www.digitemp.com
Oct 30 20:16:45 Sensor 0 C: 21.00 F: 69.80


Thursday, October 6, 2016

Setting Up RAID Volumes on Cisco UCS C series with Avago/LSI/MegaRAID Controllers

On a C220M4, I discovered that the controller configures the drives as JBOD. Fixing this is easy, but not quite as straightforward as one would expect... especially considering that the controller has been a mainstay of servers for a very long time.

1. Go into the controller firmware (I had to do a control-M before I saw the control-R option):


As you can see, create virtual drives is not an option. Select "Make Unconfigured Good"




Select all the drives and click OK. 

Now you have the option to create a volume.







Cisco UCS C220M4s and the Purple Screen of Death - Could not start pcpu 1: TSC sync timed out

Recently, I had to install ESXi 5.5 on a Cisco rackmount server. VMware crashed with the purple screen of death.

Here's the display:


These are relatively new servers, so there was nothing especially helpful on the web. Other people had this problem, but I couldn't find any real solutions, other than a reference to C states on the CPU.

I experimented a bit and figured that CPU power savings might be the problem (I'd seen this before on Opteron servers.)

The fix is to set CPU performance to HPC (it defaults to Enterprise) in BIOS:


That's advanced -> CPU Performance -> HPC (from Enterprise)

This solved the problem. 

Wednesday, October 5, 2016

Avocent ACS 6016/6008 Renaming Ports From CLI

These console servers are fairly easy to setup, but the CLI is segmented to facilitate differing levels of access.

If you wish to rename a serial port (i.e., with hostname) from the CLI, you need to do the following

1. Figure out which port is being renamed (i.e., port 11)
2. log in with appropriate credentials via SSH (or direct serial)
3.

cd ports

4.

cd serial_ports

4b. You can check out the parameters here:

--:#- [serial_ports/physical] cli-> show
port: 11
enable_cisco_rj45_pin-out = yes
status = enabled
speed = 9600
parity = none
data_bits = 8
stop_bits = 1

flow_control = none 

5. set port name

set port_name=my_new_name

i.e., 

--:#- [serial_ports/cas] cli-> set port_name=unused 

8. save the config

save


That's it, basically.

Tuesday, August 30, 2016

AnyConnect 4.x on ASA 8.4.x: AnyConnect is not enabled on the VPN server



I recently had to do a failover on a pair of ASA5520s. On failing over to the secondary, AnyConnect stopped working. I did not see a license error, but users who connected received this error:

AnyConnect is not enabled on the VPN server

There weren't any smoking guns in the logs. I did notice that the newest version of the config was missing the client image definitions in webvpn. They were in the flash drive, but not copied to the secondary. So, when the secondary took over, this portion of the config was dropped.

The solution was to re-add the images:

config t
webvpn
anyconnect image disk0:/anyconnect-macosx-i386-4.3.00748-k9.pkg 1
anyconnect image disk0:/anyconnect-win-4.3.00748-k9.pkg 2







Tuesday, March 8, 2016

Cisco ASA tftp Copies and Issues with atftpd

while attempting to copy a new anyconnect pkg file from a CentOS based server running atftpd, I received the following error on a Cisco ASA 5505:

%Error reading tftp://mytftpserver/anyconnect-win-3.1.14018-k9.pkg (Undefined error code)

On the tftp server in /var/log/messages, I saw:


Mar  6 14:49:19 mytftpserver  atftpd[15147]: Serving anyconnect-win-3.1.14018-k9.pkg to 192.168.47.5:32947
Mar  6 14:49:19 mytftpserver atftpd[15147]: Requested file to big, increase BLKSIZE

I had installed tftp a while ago... specifically atftp-server. After looking at the man page for in.tftpd, I concluded there was no good way to set BLKSIZE on this package. The fix was:

sudo yum erase atftp-server
sudo yum install tftp-server

I then edited the xinet.d/tftp file to enable it and change the target directory and restarted xinetd. After this, I was able to copy the pkg file over successfully.

Monday, December 28, 2015

Mellanox ConnectX-2 10GB Interface on FreeBSD 10.2

Although Mellanox's FreeBSD driver for the ConnectX-2 is included in the kernel source, the kernel modules are not included with the generic kernel. To use this card, build a custom kernel.

1. update the FreeBSD source using freebsd-update

sudo freebsd-update fetch
sudo freebsd-update install

2. copy the generic kernel config to a custom kernel
cd /usr/src/sys/amd64/conf

sudo cp GENERIC MYKERNEL01

4. edit the config file (sudo vi MYKERNEL01.) Add these lines to the bottom:

#### Mellanox ConnectX-2 support
options  OFED
options  IPOIB_CM
device  ipoib
device  mlx4ib
device  mlxen 

5. Compile the kernel and install it

cd /usr/src
sudo make buildkernel KERNCONF=MYKERNEL01
sudo make installkernel KERNCONF=MYKERNEL01

6. reboot

sudo shutdown -r now

Your new interface should show up when the machine comes back:

mlxen0: flags=8802 metric 0 mtu 1500
options=d07bb
ether 00:02:c9:52:ad:23
nd6 options=29
media: Ethernet autoselect (autoselect )
status: active


You can configure however you'd like (IP, MTU, etc.)

If you run freebsd-update on a regular basis, you may want to append a  kernel rebuild (step #5) to a script so you can rebuild the kernel automatically if there is a kernel source update.



Thursday, November 19, 2015

2010 Macbook Pro and problems with Yosemite and/or El Capitan

I have had terrible issues with a mid-2010 MBP 15" core i7 laptop. Doing a clean installation of either Yosemite or El Capitan, the installer fails toward the end or crashes on first boot. I have not tried a fresh Mavericks install. However, a restore from a Mavericks time machine backup works perfectly fine. Alternatively, a linux installation also worked well with no crashes.

Quite a few posts suggest it is a log board problem and that Apple would fix it. However, being impatient, I tried my own work around. I noticed that the problem didn't happen if I reduced the RAM from 8GB to 4GB (1 stick of DDR 3 instead of 2.)

A prevailing theory for this failure is that the Nvidia 330M card was crashing the system when it switched from the onboard to the 330M.

A couple of suggested work arounds:

1. disable automatic video switching in power preferences. This will likely consume battery at a faster rate

2. use gfxCardStatus and force it to either onboard or discrete

 What I did was:

1. do a fresh install of El Capitan with only 1 stick of DDR 3 (4GB)

2. fully patched the OS

3. installed gfxcardstatus and forced it to onboard only.

 I haven't had a crash since.

Wednesday, May 27, 2015

Windows Netstat and findstr - Same as find | grep on Linux/Unix

Unix/Linux users are familiar with the following patterns of usage for finding listening ports:

looking for services listening on ports 443 or 444:
netstat -a | grep 44[34]


find a service listening on port 602, 612, 604, 614, 608, or 618:
netstat -an | grep 6[01][248]


To do the same on windows, use netstat and findstr:


find a window's service listening on port 602, 612, 604, 614, 608, or 618:
netstat -an | findstr /R 6[01][248]