---

Subject:	ZenEventService Log output
Author:	[Not Specified]
Posted:	2014-04-17 21:23

2014-04-17T00:00:14.833 [main] INFO org.zenoss.zep.dao.ConfigDao - Maximum archive days: 1000
2014-04-17T00:00:15.492 [main] WARN org.zenoss.utils.impl.ZenPackImpl - No egg-info found for ZenPack in /opt/zenoss/ZenPacks/ZenPacks.zenoss.PySamba-1.0.2-py2.7-linux-x86_64.egg/ZenPacks/zenoss/PySamba/lib.
2014-04-17T00:00:15.501 [main] INFO org.zenoss.zep.impl.PluginServiceImpl - No external plug-ins found.
2014-04-17T00:00:16.171 [main] WARN org.zenoss.utils.impl.ZenPackImpl - No egg-info found for ZenPack in /opt/zenoss/ZenPacks/ZenPacks.zenoss.PySamba-1.0.2-py2.7-linux-x86_64.egg/ZenPacks/zenoss/PySamba/lib.
2014-04-17T00:00:16.346 [AmqpConnectionManager] INFO org.zenoss.amqp.AmqpConnectionManager - Attempting to connect to message broker at amqp://zenoss@localhost:5672/zenoss
2014-04-17T00:00:16.370 [AmqpConnectionManager] INFO org.zenoss.amqp.impl.ConnectionFactoryImpl - No AMQP connection heartbeat
2014-04-17T00:00:16.438 [AmqpConnectionManager] INFO org.zenoss.amqp.AmqpConnectionManager - Connected to message broker at amqp://zenoss@localhost:5672/zenoss
2014-04-17T00:00:16.541 [main] INFO org.zenoss.zep.impl.PluginServiceImpl - Starting plug-in: TriggerPlugin
2014-04-17T00:00:16.550 [main] INFO org.zenoss.zep.impl.PluginServiceImpl - Starting plug-in: EventFanOutPlugin
2014-04-17T00:00:16.551 [main] INFO org.zenoss.zep.impl.PluginServiceImpl - Initialized plug-ins
2014-04-17T00:00:16.551 [main] INFO org.zenoss.zep.impl.Application - Initializing ZEP
2014-04-17T00:00:16.688 [main] INFO org.zenoss.utils.dao.impl.MySqlRangePartitioner - Adding partition p20140420_000000 to table event_archive
2014-04-17T00:00:16.862 [main] INFO org.zenoss.utils.dao.impl.MySqlRangePartitioner - Adding partition p20140417_060000 to table event_time
2014-04-17T00:00:16.862 [main] INFO org.zenoss.utils.dao.impl.MySqlRangePartitioner - Adding partition p20140417_070000 to table event_time
2014-04-17T00:00:16.929 [main] ERROR org.springframework.web.context.ContextLoader - Context initialization failed
org.springframework.jdbc.UncategorizedSQLException: StatementCallback; uncategorized SQLException for SQL [ALTER TABLE event_time PARTITION BY RANGE(processed) (PARTITION p20140417_060000 VALUES LESS THAN(1397714400000),PARTITION p20140417_070000 VALUES LESS THAN(1397718000000))]; SQL state [HY000]; error code [1696]; Failed to read from the .par file; nested exception is java.sql.SQLException: Failed to read from the .par file
at org.springframework.jdbc.support.AbstractFallbackSQLExceptionTranslator.translate(AbstractFallbackSQLExceptionTranslator.java:83) ~[spring-jdbc-3.0.5.RELEASE.jar:3.0.5.RELEASE]
at org.springframework.jdbc.support.AbstractFallbackSQLExceptionTranslator.translate(AbstractFallbackSQLExceptionTranslator.java:80) ~[spring-jdbc-3.0.5.RELEASE.jar:3.0.5.RELEASE]
at org.springframework.jdbc.support.AbstractFallbackSQLExceptionTranslator.translate(AbstractFallbackSQLExceptionTranslator.java:80) ~[spring-jdbc-3.0.5.RELEASE.jar:3.0.5.RELEASE]
at org.springframework.jdbc.core.JdbcTemplate.execute(JdbcTemplate.java:406) ~[spring-jdbc-3.0.5.RELEASE.jar:3.0.5.RELEASE]
at org.springframework.jdbc.core.JdbcTemplate.update(JdbcTemplate.java:518) ~[spring-jdbc-3.0.5.RELEASE.jar:3.0.5.RELEASE]
at org.springframework.jdbc.core.simple.SimpleJdbcTemplate.update(SimpleJdbcTemplate.java:248) ~[spring-jdbc-3.0.5.RELEASE.jar:3.0.5.RELEASE]
at org.zenoss.utils.dao.impl.MySqlRangePartitioner.createPartitions(MySqlRangePartitioner.java:58) ~[utils-1.2-SNAPSHOT.jar:na]
at org.zenoss.utils.dao.impl.AbstractRangePartitioner.createPartitions(AbstractRangePartitioner.java:108) ~[utils-1.2-SNAPSHOT.jar:na]
at org.zenoss.zep.dao.impl.EventTimeDaoImpl.initializePartitions(EventTimeDaoImpl.java:79) ~[zep-core-1.2-SNAPSHOT.jar:na]
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) ~[na:1.6.0_31]
at sun.reflect.NativeMethodAccessorImpl.invoke(Unknown Source) ~[na:1.6.0_31]
at sun.reflect.DelegatingMethodAccessorImpl.invoke(Unknown Source) ~[na:1.6.0_31]
at java.lang.reflect.Method.invoke(Unknown Source) ~[na:1.6.0_31]
at org.springframework.aop.support.AopUtils.invokeJoinpointUsingReflection(AopUtils.java:309) ~[spring-aop-3.0.5.RELEASE.jar:3.0.5.RELEASE]
at org.springframework.aop.framework.ReflectiveMethodInvocation.invokeJoinpoint(ReflectiveMethodInvocation.java:183) ~[spring-aop-3.0.5.RELEASE.jar:3.0.5.RELEASE]
at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:150) ~[spring-aop-3.0.5.RELEASE.jar:3.0.5.RELEASE]
at org.springframework.transaction.interceptor.TransactionInterceptor.invoke(TransactionInterceptor.java:110) ~[spring-tx-3.0.5.RELEASE.jar:3.0.5.RELEASE]
at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:172) ~[spring-aop-3.0.5.RELEASE.jar:3.0.5.RELEASE]
at org.springframework.aop.framework.JdkDynamicAopProxy.invoke(JdkDynamicAopProxy.java:202) ~[spring-aop-3.0.5.RELEASE.jar:3.0.5.RELEASE]
at $Proxy29.initializePartitions(Unknown Source) ~[na:na]
at org.zenoss.zep.impl.Application.initializePartitions(Application.java:234) ~[zep-core-1.2-SNAPSHOT.jar:na]
at org.zenoss.zep.impl.Application.init(Application.java:174) ~[zep-core-1.2-SNAPSHOT.jar:na]
at org.zenoss.zep.impl.Application.onApplicationEvent(Application.java:462) ~[zep-core-1.2-SNAPSHOT.jar:na]
at org.zenoss.zep.impl.Application.onApplicationEvent(Application.java:60) ~[zep-core-1.2-SNAPSHOT.jar:na]
at org.springframework.context.event.SimpleApplicationEventMulticaster.multicastEvent(SimpleApplicationEventMulticaster.java:97) ~[spring-context-3.0.5.RELEASE.jar:3.0.5.RELEASE]
at org.springframework.context.support.AbstractApplicationContext.publishEvent(AbstractApplicationContext.java:303) ~[spring-context-3.0.5.RELEASE.jar:3.0.5.RELEASE]
at org.zenoss.zep.impl.PluginServiceImpl.onApplicationEvent(PluginServiceImpl.java:416) ~[zep-core-1.2-SNAPSHOT.jar:na]
at org.zenoss.zep.impl.PluginServiceImpl.onApplicationEvent(PluginServiceImpl.java:58) ~[zep-core-1.2-SNAPSHOT.jar:na]
at org.springframework.context.event.SimpleApplicationEventMulticaster.multicastEvent(SimpleApplicationEventMulticaster.java:97) ~[spring-context-3.0.5.RELEASE.jar:3.0.5.RELEASE]
at org.springframework.context.support.AbstractApplicationContext.publishEvent(AbstractApplicationContext.java:303) ~[spring-context-3.0.5.RELEASE.jar:3.0.5.RELEASE]
at org.springframework.context.support.AbstractApplicationContext.finishRefresh(AbstractApplicationContext.java:911) ~[spring-context-3.0.5.RELEASE.jar:3.0.5.RELEASE]
at org.springframework.context.support.AbstractApplicationContext.refresh(AbstractApplicationContext.java:428) ~[spring-context-3.0.5.RELEASE.jar:3.0.5.RELEASE]
at org.springframework.web.context.ContextLoader.createWebApplicationContext(ContextLoader.java:276) ~[spring-web-3.0.5.RELEASE.jar:3.0.5.RELEASE]
at org.springframework.web.context.ContextLoader.initWebApplicationContext(ContextLoader.java:197) ~[spring-web-3.0.5.RELEASE.jar:3.0.5.RELEASE]
at org.springframework.web.context.ContextLoaderListener.contextInitialized(ContextLoaderListener.java:47) [spring-web-3.0.5.RELEASE.jar:3.0.5.RELEASE]
at org.jboss.resteasy.plugins.spring.SpringContextLoaderListener.contextInitialized(SpringContextLoaderListener.java:44) [resteasy-spring-2.0.1.GA.jar:na]
at org.eclipse.jetty.server.handler.ContextHandler.startContext(ContextHandler.java:643) [jetty-server-7.5.3.v20111011.jar:7.5.3.v20111011]
at org.eclipse.jetty.servlet.ServletContextHandler.startContext(ServletContextHandler.java:233) [jetty-servlet-7.5.3.v20111011.jar:7.5.3.v20111011]
at org.eclipse.jetty.webapp.WebAppContext.startContext(WebAppContext.java:1213) [jetty-webapp-7.5.3.v20111011.jar:7.5.3.v20111011]
at org.eclipse.jetty.server.handler.ContextHandler.doStart(ContextHandler.java:589) [jetty-server-7.5.3.v20111011.jar:7.5.3.v20111011]
at org.eclipse.jetty.webapp.WebAppContext.doStart(WebAppContext.java:454) [jetty-webapp-7.5.3.v20111011.jar:7.5.3.v20111011]
at org.eclipse.jetty.util.component.AbstractLifeCycle.start(AbstractLifeCycle.java:59) [jetty-util-7.5.3.v20111011.jar:7.5.3.v20111011]
at org.eclipse.jetty.deploy.bindings.StandardStarter.processBinding(StandardStarter.java:36) [jetty-deploy-7.5.3.v20111011.jar:7.5.3.v20111011]
at org.eclipse.jetty.deploy.AppLifeCycle.runBindings(AppLifeCycle.java:183) [jetty-deploy-7.5.3.v20111011.jar:7.5.3.v20111011]
at org.eclipse.jetty.deploy.DeploymentManager.requestAppGoal(DeploymentManager.java:485) [jetty-deploy-7.5.3.v20111011.jar:7.5.3.v20111011]
at org.eclipse.jetty.deploy.DeploymentManager.addApp(DeploymentManager.java:138) [jetty-deploy-7.5.3.v20111011.jar:7.5.3.v20111011]
at org.eclipse.jetty.deploy.providers.ScanningAppProvider.fileAdded(ScanningAppProvider.java:140) [jetty-deploy-7.5.3.v20111011.jar:7.5.3.v20111011]
at org.eclipse.jetty.deploy.providers.ScanningAppProvider$1.fileAdded(ScanningAppProvider.java:53) [jetty-deploy-7.5.3.v20111011.jar:7.5.3.v20111011]
at org.eclipse.jetty.util.Scanner.reportAddition(Scanner.java:604) [jetty-util-7.5.3.v20111011.jar:7.5.3.v20111011]
at org.eclipse.jetty.util.Scanner.reportDifferences(Scanner.java:535) [jetty-util-7.5.3.v20111011.jar:7.5.3.v20111011]
at org.eclipse.jetty.util.Scanner.scan(Scanner.java:398) [jetty-util-7.5.3.v20111011.jar:7.5.3.v20111011]
at org.eclipse.jetty.util.Scanner.doStart(Scanner.java:332) [jetty-util-7.5.3.v20111011.jar:7.5.3.v20111011]
at org.eclipse.jetty.util.component.AbstractLifeCycle.start(AbstractLifeCycle.java:59) [jetty-util-7.5.3.v20111011.jar:7.5.3.v20111011]
at org.eclipse.jetty.deploy.providers.ScanningAppProvider.doStart(ScanningAppProvider.java:117) [jetty-deploy-7.5.3.v20111011.jar:7.5.3.v20111011]
at org.eclipse.jetty.util.component.AbstractLifeCycle.start(AbstractLifeCycle.java:59) [jetty-util-7.5.3.v20111011.jar:7.5.3.v20111011]
at org.eclipse.jetty.deploy.DeploymentManager.startAppProvider(DeploymentManager.java:546) [jetty-deploy-7.5.3.v20111011.jar:7.5.3.v20111011]
at org.eclipse.jetty.deploy.DeploymentManager.doStart(DeploymentManager.java:221) [jetty-deploy-7.5.3.v20111011.jar:7.5.3.v20111011]
at org.eclipse.jetty.util.component.AbstractLifeCycle.start(AbstractLifeCycle.java:59) [jetty-util-7.5.3.v20111011.jar:7.5.3.v20111011]
at org.eclipse.jetty.util.component.AggregateLifeCycle.doStart(AggregateLifeCycle.java:45) [jetty-util-7.5.3.v20111011.jar:7.5.3.v20111011]
at org.eclipse.jetty.server.handler.AbstractHandler.doStart(AbstractHandler.java:53) [jetty-server-7.5.3.v20111011.jar:7.5.3.v20111011]
at org.eclipse.jetty.server.handler.HandlerWrapper.doStart(HandlerWrapper.java:90) [jetty-server-7.5.3.v20111011.jar:7.5.3.v20111011]
at org.eclipse.jetty.server.Server.doStart(Server.java:261) [jetty-server-7.5.3.v20111011.jar:7.5.3.v20111011]
at org.eclipse.jetty.util.component.AbstractLifeCycle.start(AbstractLifeCycle.java:59) [jetty-util-7.5.3.v20111011.jar:7.5.3.v20111011]
at org.eclipse.jetty.xml.XmlConfiguration$1.run(XmlConfiguration.java:1203) [jetty-xml-7.5.3.v20111011.jar:7.5.3.v20111011]
at java.security.AccessController.doPrivileged(Native Method) [na:1.6.0_31]
at org.eclipse.jetty.xml.XmlConfiguration.main(XmlConfiguration.java:1126) [jetty-xml-7.5.3.v20111011.jar:7.5.3.v20111011]
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) ~[na:1.6.0_31]
at sun.reflect.NativeMethodAccessorImpl.invoke(Unknown Source) ~[na:1.6.0_31]
at sun.reflect.DelegatingMethodAccessorImpl.invoke(Unknown Source) ~[na:1.6.0_31]
at java.lang.reflect.Method.invoke(Unknown Source) ~[na:1.6.0_31]
at org.eclipse.jetty.start.Main.invokeMain(Main.java:469) [jetty-start-7.5.3.v20111011.jar:7.5.3.v20111011]
at org.eclipse.jetty.start.Main.start(Main.java:612) [jetty-start-7.5.3.v20111011.jar:7.5.3.v20111011]
at org.eclipse.jetty.start.Main.parseCommandLine(Main.java:265) [jetty-start-7.5.3.v20111011.jar:7.5.3.v20111011]
at org.eclipse.jetty.start.Main.main(Main.java:79) [jetty-start-7.5.3.v20111011.jar:7.5.3.v20111011]
Caused by: java.sql.SQLException: Failed to read from the .par file
at com.mysql.jdbc.SQLError.createSQLException(SQLError.java:1073) ~[mysql-connector-java-5.1.15.jar:na]
at com.mysql.jdbc.MysqlIO.checkErrorPacket(MysqlIO.java:3603) ~[mysql-connector-java-5.1.15.jar:na]
at com.mysql.jdbc.MysqlIO.checkErrorPacket(MysqlIO.java:3535) ~[mysql-connector-java-5.1.15.jar:na]
at com.mysql.jdbc.MysqlIO.sendCommand(MysqlIO.java:1989) ~[mysql-connector-java-5.1.15.jar:na]
at com.mysql.jdbc.MysqlIO.sqlQueryDirect(MysqlIO.java:2150) ~[mysql-connector-java-5.1.15.jar:na]
at com.mysql.jdbc.ConnectionImpl.execSQL(ConnectionImpl.java:2620) ~[mysql-connector-java-5.1.15.jar:na]
at com.mysql.jdbc.StatementImpl.executeUpdate(StatementImpl.java:1664) ~[mysql-connector-java-5.1.15.jar:na]
at com.mysql.jdbc.StatementImpl.executeUpdate(StatementImpl.java:1583) ~[mysql-connector-java-5.1.15.jar:na]
at org.springframework.jdbc.core.JdbcTemplate$1UpdateStatementCallback.doInStatement(JdbcTemplate.java:508) ~[spring-jdbc-3.0.5.RELEASE.jar:3.0.5.RELEASE]
at org.springframework.jdbc.core.JdbcTemplate$1UpdateStatementCallback.doInStatement(JdbcTemplate.java:1) ~[spring-jdbc-3.0.5.RELEASE.jar:3.0.5.RELEASE]
at org.springframework.jdbc.core.JdbcTemplate.execute(JdbcTemplate.java:395) ~[spring-jdbc-3.0.5.RELEASE.jar:3.0.5.RELEASE]
... 70 common frames omitted

---

Subject:	Fun Stuff
Author:	[Not Specified]
Posted:	2014-04-18 09:07

What OS and Zenoss version are you running

When you ran 'zenchkrels -r -x1' did you get any output Also did you verify with 'zenoss status' that all the daemons were running (obviously except the one you stopped).

Use 'top' to make sure the system is being maxed out and 'df -h' to verify the drive isn't full.

Does 'service rabbitmq-server restart' give you any errors

Does 'rabbitmqctl -p /zenoss list_queues' list the zenoss queues (can see an example here http://wiki.zenoss.org/Working_with_Queues)

A lot of basic steps yes, but it will help pinpoint the issue!

Hydruid

---

Subject:	Hi Hydruid
Author:	[Not Specified]
Posted:	2014-04-18 09:37

Hi Hydruid
1) OS = Centos 6.5, Zenos = 4.2.4
2) zenchkrels -r -x1 does not produce any output outside of 'INFO:zen.CheckRelates:Checking relations ...'
3) all zenoss daemons running except the one I stopped
4) top and df -h come back looking good (lots of available capacity)
5) service rabbitmg-server restart, restarts the service sucessfully
6) I have 4 queues listed

Honestly I think I have a corrupted database as you can see from my log output that I have "Failed to read from the .par file; nested exception is java.sql.SQLException: Failed to read from the .par file". I tried multiple steps to drop and recreate the zenoss_zep database but I keep getting the error 'ERROR 1051 (42S02): Unknown table 'event_time'

thanks,
Kevin

---

Subject:	Hmmm
Author:	[Not Specified]
Posted:	2014-04-18 09:46

You should have more queue's than 4 but I believe you're right about the corrupted database. Make sure to stop zenoss before trying to drop/import tables.

You can also try to repair the database by creating the table and whatever else is missing, my .DEB package (http://master.dl.sourceforge.net/project/zenossforubuntu/zenoss-core-424...) has the default databases in it, so does the RPM.

Hydruid

---

Subject:	I think at this point I will
Author:	[Not Specified]
Posted:	2014-04-18 12:10

I think at this point I will have to rebuild it from scratch. Thanks for your help!