TECHZEN Zenoss User Community ARCHIVE  
FORUMS / Configuration & Administration / DISCUSSION

Zenoss 6.1.1 graphs show no data, zenhub and MetricShipper failing some health c ...

Subject: Zenoss 6.1.1 graphs show no data, zenhub and MetricShipper failing some health checks
Author: Joseph Meslovich
Posted: 2018-09-17 14:21

Since installing Zenoss6 last year we have had random times when MetricShipper and Zope would fail and we would have to restart them to get back into Zenoss Core.  Normally restarting those two containers in Control Center would do the trick.  Last week we did this and while Zope came back up we haven't gotten graphs back.

In Control Center zenhub and MetricShipper are showing health issues.  We are getting a failed health check for metric_consumer_answering for zenhub,  and fails for store_answering and websocket_opened on MetricShipper.  

Are these failed health checks related to the loss of graphs?  What should we be looking into to troubleshoot this issue further?


------------------------------
Joseph Meslovich
Network Administrator & IT Security Officer
Bridgewater College
Bridgewater VA
540-828-5343
------------------------------

Subject: RE: Zenoss 6.1.1 graphs show no data, zenhub and MetricShipper failing some health checks
Author: Solomon Hill
Posted: 2018-09-18 14:27

I'm having the exact same issue with Zenoss 6.2... not sure why.

Solomon Hill
Director of Technology
Ravenswood City School District
East Palo Alto, CA

------------------------------
Solomon Hill
------------------------------
Subject: RE: Zenoss 6.1.1 graphs show no data, zenhub and MetricShipper failing some health checks
Author: Steven Leong
Posted: 2018-09-19 18:55

I am also experiencing this issue since 13/9/18.  Restarting zenhub and metricshipper not helping.

------------------------------
Steven
------------------------------

Subject: RE: Zenoss 6.1.1 graphs show no data, zenhub and MetricShipper failing some health checks
Author: Arthur
Posted: 2018-09-20 15:49

Hi Joseph

Yes, this could be.

MetricShipper
Inserts metrics into OpenTSDBWriter.
This component ships with a default threshold. The maximum number of seconds MetricShipper needs to
process its Redis queue at the current rate is 300. If this value is exceeded, the MetricShipper node on the
metric pipeline turns gray and flashes.

OpenTSDB
Resource Manager no longer uses RRD files on the collectors for data storage. We have created a centralized
storage framework for this data which uses a Redis key-value store on the collector and then ships that data to
an OpenTSDB (time series database) instance that runs on Hadoop and HBase.

Source:
https://www.zenoss.com/sites/default/files/zenoss-doc/9856/base/admin/monself/self-monitor-components.html

Cheers

------------------------------
Arthur
------------------------------

Subject: RE: Zenoss 6.1.1 graphs show no data, zenhub and MetricShipper failing some health checks
Author: Arthur
Posted: 2018-09-20 15:52

To solve it

Do a backup from the GUI but don't delete or overwrite older, perhabs good ones.

then try:

https://support.zenoss.com/hc/en-us/articles/211783563-Zenoss-Master-Staged-Startup-and-Shutdown-Best-Practices-for-Maintenance-

If it does not help restore a known good backup taken before the failure occured.

------------------------------
Arthur
------------------------------

Subject: RE: Zenoss 6.1.1 graphs show no data, zenhub and MetricShipper failing some health checks
Author: Steven Leong
Posted: 2018-09-25 19:30

I have followed the guidance from the staged shutdown/startup article provided.

I encounter issues starting the MetricShipper service, it fails health checks for store_answering.  Consequently, when trying to start the zenhub service afterwards, it fails health checks for metric_consumer_answering. 

I am not seeing any other problems.

Any guidance on how to get these services healthy would be greatly appreciated, as I don't have a good backup to resort to... I believe that fixing this will resolve my graphing issues.

Thanks


------------------------------
Steven
------------------------------

Subject: RE: Zenoss 6.1.1 graphs show no data, zenhub and MetricShipper failing some health checks
Author: Steven Leong
Posted: 2018-10-01 22:45

Must have been a resource issue in my case, as I was seeing various memory related errors in the logs.

After reducing the number of monitored devices from 212 down to 190 (single master host), I tweaked the RAM requested for various services including zenhub, metricconsumer, zenpython, opentsdb and zenmodeler.  Doubled the default amount. 

Then after following the shutdown/startup guide it all came right.

Very relieved, and have now made a good backup :)


------------------------------
Steven
------------------------------

Subject: RE: Zenoss 6.1.1 graphs show no data, zenhub and MetricShipper failing some health checks
Author: Joseph Meslovich
Posted: 2018-11-12 10:59

So it appears we were also having a resource issue here.  We were monitoring 85 Windows servers, 24 Linux servers, and 121 switches.  We had some staffing changes and the new Systems Administrator decided to go with Zabbix instead of Zenoss for server monitoring.  So after removing the Windows servers from Zenoss, the resource requirements dropped enough that MetricShipper and zenhub containers started working normally again.

So if we had wanted to keep monitoring everything we would have also had to increase the resources of those containers.  When we initially installed we had also gone with the minimum recommend resources.  We did not explore what we would have needed to increase the resources to to properly size the Zenoss master for our environment.  We were only running the master and had not added any other hosts to the Zenoss cluster.


------------------------------
Joseph Meslovich
Network Administrator & IT Security Officer
Bridgewater College
Bridgewater VA
540-828-5343
------------------------------

Subject: RE: Zenoss 6.1.1 graphs show no data, zenhub and MetricShipper failing some health checks
Author: Jane Curry
Posted: 2018-11-14 05:38

Thanks for these updates.  It would appear that Zenoss 6 doesn't always  "degrade nicely" when short of resources.

It would be helpful if Zenoss could publish guidance on this sort of scenario.

Cheers,
Jane

------------------------------
Jane Curry
Skills 1st United Kingdom
jane.curry@skills-1st.co.uk
------------------------------

Subject: RE: Zenoss 6.1.1 graphs show no data, zenhub and MetricShipper failing some health checks
Author: Jason Olson
Posted: 2018-11-14 10:04

Heh. I think this is the only guidance we're going to get on things like this, Jane. :)

------------------------------
Jason Olson
------------------------------

Subject: RE: Zenoss 6.1.1 graphs show no data, zenhub and MetricShipper failing some health checks
Author: Jane Curry
Posted: 2018-11-14 10:11

But there's no harm in asking ;)
If a management system doesn't "degrade nicely" then I would hope that the vendor is addressing such an issue and would provide advice and guidance meantime.

Is that unreasonable?

Cheers,
Jane

------------------------------
Jane Curry
Skills 1st United Kingdom
jane.curry@skills-1st.co.uk
------------------------------

Subject: RE: Zenoss 6.1.1 graphs show no data, zenhub and MetricShipper failing some health checks
Author: Jason Olson
Posted: 2018-11-14 10:20

Not in the slightest. Here's hoping they scan the forums every so often for feedback like this.

------------------------------
Jason Olson
------------------------------

Subject: RE: Zenoss 6.1.1 graphs show no data, zenhub and MetricShipper failing some health checks
Author: Jad Baz
Posted: 2019-03-29 07:43

I'm also having this issue with Zenoss 6.2.1, check out my post from December:
Zenoss 6.2.1, Zope stops answering on its own, unprovoked

------------------------------
Jad
------------------------------


< Previous
threshold of zenmodeler cycle time exceeded 		  Next
Help with event transform 		>