Friday, January 31, 2014

Apache mod_jk to Tomcat/Jboss Connection Errors Related to Palo Alto Firewalls

I recently ran across a problem when sending mod_jk/AJP connections back to a Jboss app server running behind a Palo Alto firewall (PA500.) The error was a little mysterious as Jboss didn't really report anything interesting. I could see that traffic was passing (via tcpdump) but Apache generated 500 errors.
The only clue was the following error message in the mod_jk log:

[Wed Jan 29 17:23:44 2014][9283:16992576] [info] ajp_handle_cping_cpong::jk_ajp_common.c (876): awaited reply cpong, not received 
[Wed Jan 29 17:23:44 2014][9283:16992576] [error] ajp_connect_to_endpoint::jk_ajp_common.c (957): (WSERVICES) cping/cpong after connecting to the backend server failed (errno=104) 
[Wed Jan 29 17:23:44 2014][9283:16992576] [error] ajp_send_request::jk_ajp_common.c (1507): (WSERVICES) connecting to backend failed. Tomcat is probably not started or is listening on the wrong port (errno=104) 
[Wed Jan 29 17:23:44 2014][9283:16992576] [info] ajp_service::jk_ajp_common.c (2447): (WSERVICES) sending request to tomcat failed (recoverable), because of error during request sending (attempt=1)

The cping/cpong message is talking about a failed keep alive.

I figured it was something related to the way the PA firewall manipulates the data via a proxy. I then created a rule defining 8009 as a service (as opposed to an application), turned off AV inspection of the traffic, and the problem went away.