TECHZEN Zenoss User Community ARCHIVE  

Zenoss5 does NOT record ANY data or display ANY graphs

Subject: Zenoss5 does NOT record ANY data or display ANY graphs
Author: Simon Lange
Posted: 2017-05-17 11:36

Fresh docker deployed zenoss5 installation.

zenoss does NOT record ANY information of ANY device.

All services did start and are running.

the journal just says:

May 17 17:28:52 HOST.DOMAIN.TLD serviced[3236]: time="2017-05-17T15:28:52Z" level=warning msg="Unable to report stats to OpenTSDB" destination="http://127.0.0.1:8443/api/metrics/store" error="stats: Received response status: \"422 Unprocessable Entity\" from metric POST" location="statsreporter.go:78" logger=stats


this line comes twice every minute!

in the docker console CentralQuery has a red expression mark ("answering") but its "started".

the system zenoss5 was deployed on is centos7.

ANY ideas? im out of them.

Simon

------------------------------
Simon Lange
Head of DevOps
Spectacle Games Inc
Berlin
------------------------------


Subject: RE: Zenoss5 does NOT record ANY data or display ANY graphs
Author: Simon Lange
Posted: 2017-05-22 09:12

Nobody?! So zenoss5 has been destroyed by too many changes on working features?

if my problem is not solved in the next days, we will ban zenoss and i have to use another monitoring software, which i dont want, since i like zenoss. But if zenoss does not work anymore, and noone feels responsible to make it running (hey, its a god damn FRESH install by YOUR docker-deployment), zenoss and us will go seperate ways in the future after years of success. :(

best regards

Simon

------------------------------
Simon Lange
Head of DevOps
Spectacle Games Inc
Berlin
------------------------------

Subject: RE: Zenoss5 does NOT record ANY data or display ANY graphs
Author: Michael De Simone
Posted: 2017-05-22 15:59

Hello Simon,

I think this message is key. It is telling us that

in the docker console CentralQuery has a red expression mark ("answering") but its "started".

Central Query is not listening, which is consistent with your other message. What log are you seeing that other message in? Are you seeing any other errors about Central Query? Are you looking at journalctl -u serviced -f -o cat?






Subject: RE: Zenoss5 does NOT record ANY data or display ANY graphs
Author: Simon Lange
Posted: 2017-05-22 21:59

Hi Michael,

thanks for your try to help.

Yes, im usin the journal to view whats goin on, and no there are no other messages except some http proxy errors which show up when i try to watch graphs in zenoss5.

[root@arneot ~]# journalctl -u serviced -f -o cat
time="2017-05-23T01:51:33Z" level=info msg="Received new authentication token" expiration=1495507893 location="token.go:62" logger=auth
time="2017-05-23T01:51:56Z" level=warning msg="Unable to report stats to OpenTSDB" destination="http://127.0.0.1:8443/api/metrics/store" error="stats: Received response status: \"422 Unprocessable Entity\" from metric POST" location="statsreporter.go:78" logger=stats
time="2017-05-23T01:52:26Z" level=warning msg="Unable to report stats to OpenTSDB" destination="http://127.0.0.1:8443/api/metrics/store" error="stats: Received response status: \"422 Unprocessable Entity\" from metric POST" location="statsreporter.go:78" logger=stats
time="2017-05-23T01:52:56Z" level=warning msg="Unable to report stats to OpenTSDB" destination="http://127.0.0.1:8443/api/metrics/store" error="stats: Received response status: \"422 Unprocessable Entity\" from metric POST" location="statsreporter.go:78" logger=stats
time="2017-05-23T01:53:26Z" level=warning msg="Unable to report stats to OpenTSDB" destination="http://127.0.0.1:8443/api/metrics/store" error="stats: Received response status: \"422 Unprocessable Entity\" from metric POST" location="statsreporter.go:78" logger=stats
time="2017-05-23T01:54:02Z" level=warning msg="Unable to report stats to OpenTSDB" destination="http://127.0.0.1:8443/api/metrics/store" error="stats: Received response status: \"422 Unprocessable Entity\" from metric POST" location="statsreporter.go:78" logger=stats
time="2017-05-23T01:54:26Z" level=warning msg="Unable to report stats to OpenTSDB" destination="http://127.0.0.1:8443/api/metrics/store" error="stats: Received response status: \"422 Unprocessable Entity\" from metric POST" location="statsreporter.go:78" logger=stats
time="2017-05-23T01:54:56Z" level=warning msg="Unable to report stats to OpenTSDB" destination="http://127.0.0.1:8443/api/metrics/store" error="stats: Received response status: \"422 Unprocessable Entity\" from metric POST" location="statsreporter.go:78" logger=stats
time="2017-05-23T01:55:26Z" level=warning msg="Unable to report stats to OpenTSDB" destination="http://127.0.0.1:8443/api/metrics/store" error="stats: Received response status: \"422 Unprocessable Entity\" from metric POST" location="statsreporter.go:78" logger=stats
time="2017-05-23T01:55:56Z" level=warning msg="Unable to report stats to OpenTSDB" destination="http://127.0.0.1:8443/api/metrics/store" error="stats: Received response status: \"422 Unprocessable Entity\" from metric POST" location="statsreporter.go:78" logger=stats
time="2017-05-23T01:56:26Z" level=warning msg="Unable to report stats to OpenTSDB" destination="http://127.0.0.1:8443/api/metrics/store" error="stats: Received response status: \"422 Unprocessable Entity\" from metric POST" location="statsreporter.go:78" logger=stats


this is all i usually get. btw: even if i stop zenoss5 from docker.
anyhow, zenoss does not record ANY data and graphs usually show N/A values or get "Response Timeout".

pretty frustrating.

------------------------------
Simon Lange
Head of DevOps
Spectacle Games Inc
Berlin
------------------------------


Subject: RE: Zenoss5 does NOT record ANY data or display ANY graphs
Author: Michael De Simone
Posted: 2017-05-24 12:11

Hi Simon,

No metrics are displayed because the interface to HBASE, OpenTSDB or Central Query is not coming up clean.

I'd probably start with the Corruption of Zenoss HBase section of this KB article.

https://support.zenoss.com/hc/en-us/articles/204643769-How-to-Recover-Control-Center-from-Hardware-Failure




Subject: RE: Zenoss5 does NOT record ANY data or display ANY graphs
Author: Simon Lange
Posted: 2017-06-26 08:54

Hi,

just saw your post right now, but solved the issue meanwhile myself.
You are right, hadoop was the evil part. As always. hadoop is a hell of unstable crap and i would usually never use it on productive system. well zenoss in its wise and foreseeing capabilities using it on zenoss5 so i have to live with Damocles's sword over our monitorng system.

Anyhow, i stopped hbase, deleted EVERYTHING of hbase, and let hbase start again, rebuilding itself. That did work.

However, the startup process of zenoss5 under docker is sloppy. it takes far too long to startup and openTSDB as HBASE are throwing errors while waiting for eachother, disturbing the whole startup process.

Also you should keep in mind that the HBASE error was OUT OF THE BOX. so the deployment of zenoss5 ITSELF did CREATE the error.

docker is a disease not a cure.
so is hadoop


------------------------------
Simon Lange
Head of DevOps
Spectacle Games Inc
Berlin
------------------------------


Subject: RE: Zenoss5 does NOT record ANY data or display ANY graphs
Author: Simon Lange
Posted: 2017-07-10 00:08

aaaaaaaaaaaaand again:
zenoss5's hadoop storage failed. again.

again hbase is not starting
again opentsdb isnt working
again all on these depending services throw errors.

out of the blue, without any changes to the environment, the machines, the monitoring machine. just by RUNNING docker/zenoss5 and it was EXACTLY setup as written in the wiki.

sorry, but zenoss5 is unstable, never was stable since it uses hadoop because hadoop itself is unstable as hell.

roll back to mysql/maria or release a REAL STABLE version and a REAL self-repairing zenoss instance because right now this -imagine lots of mad names for zenoss 5- does consume MORE admin-time than ALL monitored servers and infrastructures on two continents together.

so its not economic to user zenoss5 with hadoop crap too.

the promised tech support for our "zenoss5 hadoop problem aka dead on arrival" never happened. we did never get ANY satisfying answer. after a couple of reminder we did give up. and before you ask: the did not even want our money for monitoring us.

so is hits intentional destroying zenoss's reputation? dunno why, but it looks like.

anyhow, right now zenoss5 is unstable, not for production use and we started - today - lookin for alternatives and will tell all our partners not to use zenoss5 for planned monitoring projects. bad support, bad product, no perspective for a stable version.

i mean, hey, if you follow THEIR docs and specifications and using THEIR prefered way of distribution and all you get is a unstable version with no support and fixes.... well, as painful it is - at some point you gotta quit and movin to the next product. it may not as nice as zenoss anymore, its maybe ugly, and oldfashioned, but it is stable.

frustrated and mad on zenoss dev and support staff

Simon Lange

------------------------------
Simon Lange
Head of DevOps
Spectacle Games Inc
Berlin
------------------------------


Subject: RE: Zenoss5 does NOT record ANY data or display ANY graphs
Author: rop rop
Posted: 2017-07-10 09:27

Hi
I have encountered similar problems on fresh Zenoss 5.2.4.
Ihave switched from 3.2 version, which was quite good, but now i can't get it to work.
1. After some time (let say half an hour) the Control Center stop displaying all messagess (like services instances, graphs) etc.
2. I can observe at start two errors - opentsdb health check is red with "answering" state, CentralQuery service has the same red status.
3. I've added one switch to infrastructure and have no graphs. The message is "request timeout".

Any hints?
best regards
rop

Subject: RE: Zenoss5 does NOT record ANY data or display ANY graphs
Author: amir BEN HAMADI
Posted: 2018-06-11 04:59

Hello Simon,

Finally, have you managed to solve your problem because I have the same error message that appears and no return from them after asking the question on the forum.

Amir

------------------------------
amir BEN HAMADI
System engineer
Thales Air Operation
------------------------------


< Previous
How can i change time and date Zenos 4.3.2?
  Next
422 Unprocessable Entity
>