Zenoss ZenTech Community

I did a version upgrade for my Ubuntu server from 12.04 to 14.04 LTS. This server has Zenoss 4.2.4 riding on it, and it's tracking network activity on my routers and switches.

After the upgrade, the dashboard came up and I thought all was well. Of course not! After some checking later I found that there were no traffic recorded on the graphs for the time after I had done the upgrade. Also, the high utilization warnings stopped at the same time as the graphs stopped. The event console also shows that zentrap, zenperfsnmp and zenmodeler all show heartbeat failure.

When I look at the logs in /opt/zenoss/log, I find a couple of interesting things. First, the zeneventserver.log shows some general failures in Java. Here are some examples:

2017-02-08T12:53:31.267 [INDEXER_EVENT_SUMMARY] WARN org.zenoss.zep.index.impl.EventIndexerImpl - General failure indexing events
org.springframework.transaction.CannotCreateTransactionException: Could not open JDBC Connection for transaction; nested exception is com.mysql.jdbc.exceptions.jdbc4.CommunicationsException: Communications link failure

2017-02-08T12:53:28.266 [ZEP_HEARTBEAT_PROCESSOR] WARN org.zenoss.zep.impl.Application - Failed to process heartbeat events
org.springframework.transaction.CannotCreateTransactionException: Could not open JDBC Connection for transaction; nested exception is com.mysql.jdbc.exceptions.jdbc4.CommunicationsException: Communications link failure

Running zenoss status shows the three daemons above (zentrap, zenperfsnmp and zenmodeler) not running. However, the zentrap.log shows the daemon listening on SNMP trap port 162. Zenmodeler is running as a daemon, connected to localhost:8789 and connected to zenhub. However, about the time of the reboot after the upgrade zenperfsnmp.log threw a "DeadReferenceError: Calling Stale Broker" error. However, it to is connected to localhost:8789, Zenhub and is performing periodic maintenance.

I suspect the problem is authentication between Zenoss and MySql, as a result of the upgrade. However, I'm not sure where best to begin tracing the fix. I'm going over the installation manual again after this posting, but want to be sure I'm looking in the right place.

Any ideas on why the three daemons aren't starting or logging shutdown events, and where the authentication can be checked between Zenoss and MySql

Subject:	If you are lucky, it is just
Author:	Jane Curry
Posted:	2017-02-11 09:18

If you are lucky, it is just the index to the events database that is confused. You can safely delete these and they will be recreated when Zenoss is restarted. As the zenoss user, go to $ZENHOME/var/zeneventserver/index. You should find subdirectories for archive and summary. Delete both those subdirectories and restart Zenoss.

Cheers,

Jane

NB Please note that we are away from Feb 10 to March 10, 2017

Email: jane.curry@skills-1st.co.uk Web: https://www.skills-1st.co.uk

Subject:	Update - there's some segfaulting going on!
Author:	Gregg Hughes
Posted:	2017-02-14 10:18

So I'm going over some OSSEC logs this morning and I find this:

Feb 13 15:50:29 isc-nagios kernel: [442203.736394] python[30077]: segfault at 8 ip 00007f7ec8eb7d63 sp 00007fff2f9b8878 error 4 in libc-2.19.so[7f7ec8e31000+1ba000]

This looks very suspicious. I'm going to debug this as soon as I have time and see if this is what's biting on my Zenoss installation. Meanwhile, has anyone else seen this particular error

Thanks to all!

Subject:	Ubuntu version upgrade bashed Zenoss
Author:	Gregg Hughes
Posted:	2017-02-09 15:47

Ubuntu version upgrade bashed Zenoss