Zenoss ZenTech Community

We are running into an issue with Zenoss 5.X where HBase starts, but OpenTSDB, MetricShipper, MetricConsumer, and Central Query all fail to start.

2015-07-24 16:44:31,227 INFO RPC interface 'supervisor' initialized
2015-07-24 16:44:31,227 CRIT Server 'inet_http_server' running without any HTTP authentication checking
2015-07-24 16:44:31,227 INFO supervisord started with pid 25
2015-07-24 16:44:32,230 INFO spawned: 'tsdbwatchdog' with pid 28
2015-07-24 16:44:32,232 INFO spawned: 'opentsdb' with pid 29
2015-07-24 16:44:33,424 INFO success: tsdbwatchdog entered RUNNING state, process has stayed up for > than 1 seconds (startsecs)
2015-07-24 16:44:33,841 INFO exited: opentsdb (exit status 1; not expected)
2015-07-24 16:44:34,844 INFO spawned: 'opentsdb' with pid 125
2015-07-24 16:44:35,618 INFO exited: opentsdb (exit status 1; not expected)
2015-07-24 16:44:37,622 INFO spawned: 'opentsdb' with pid 220
2015-07-24 16:44:38,930 INFO exited: opentsdb (exit status 1; not expected)
2015/07/24 16:44:40.034339 Launching harvester on new file: /opt/zenoss/log/opentsdb.log
2015/07/24 16:44:40.034406 Starting harvester: /opt/zenoss/log/opentsdb.log
2015/07/24 16:44:40.034522 Current file offset: 23789
2015-07-24 16:44:41,936 INFO spawned: 'opentsdb' with pid 315
2015/07/24 16:44:42.537381 Registrar received 1 events
2015-07-24 16:44:42,703 INFO exited: opentsdb (exit status 1; not expected)
2015-07-24 16:44:43,705 INFO gave up: opentsdb entered FATAL state, too many start retries too quickly

Subject:	would you paste the logs for
Author:	Andrew Kirch
Posted:	2015-07-24 15:55

would you paste the logs for the others I'm guessing you might have come across a race condition during startup.

Andrew Kirch

akirch@gvit.com

Need Zenoss support, consulting or custom development Look no further. Email or PM me!

Ready for Distributed Topology (collectors) for Zenoss 5 Coming May 1st from GoVanguard

Subject:	RE: would you paste the logs for
Author:	Devon Solomon
Posted:	2017-05-16 08:13

central Query
2017/05/16 11:34:20 200 15.925974ms POST /api/metrics/store
E0516 11:34:22.790191 00001 proxy.go:268] Error Local (net.Dial): dial tcp4 172.17.0.13:4242: getsockopt: connection refused
2017/05/16 11:34:35 200 84.755037ms POST /api/metrics/store
2017/05/16 11:34:44 200 98.259009ms POST /api/metrics/store
2017/05/16 11:34:50 200 37.97995ms POST /api/metrics/store
2017/05/16 11:35:05 200 10.740934ms POST /api/metrics/store
2017/05/16 11:35:15 200 74.234302ms POST /api/metrics/store
2017/05/16 11:35:20 200 72.108008ms POST /api/metrics/store
2017/05/16 11:35:38 200 16.65833ms POST /api/metrics/store
E0516 11:35:44.106492 00001 proxy.go:268] Error Local (net.Dial): dial tcp4 172.17.0.13:4242: getsockopt: connection refused
2017/05/16 11:35:47 200 49.657818ms POST /api/metrics/store

Metric Shipper
2017/05/16 09:51:31 200 168.180295ms POST /api/metrics/store
W0516 09:51:39.718773 00001 proxy.go:169] No remote services available for prxying proxy[redis-6379; &{%!s(*net.netFD=&{{0 0 0} 25 2 1 false tcp4 0xc420015d40 <nil> {140319838058424}})}]=>[]
W0516 09:51:39.719047 00001 proxy.go:169] No remote services available for prxying proxy[redis-6379; &{%!s(*net.netFD=&{{0 0 0} 25 2 1 false tcp4 0xc420015d40 <nil> {140319838058424}})}]=>[]
2017/05/16 09:51:46 200 201.545233ms POST /api/metrics/store
2017/05/16 09:51:53 200 22.145615ms POST /api/metrics/store

Subject:	RE: would you paste the logs for
Author:	Devon Solomon
Posted:	2017-05-16 08:33

i get this message on the mariadb every now and then but is passing health checks

I0516 11:48:43.751450 00001 nfs.go:25] Distributed storage temporarily unavailable (EREMDEV). Waiting for it to return.
2017/05/16 11:48:49 200 19.294708ms POST /api/metrics/store

------------------------------
Devon Solomon
------------------------------

Subject:	OpenTSDB won't start
Author:	[Not Specified]
Posted:	2015-07-24 11:50

OpenTSDB won't start