TECHZEN Zenoss User Community ARCHIVE  

Redis not answering to collection services

Subject: Redis not answering to collection services
Author: Devon Solomon
Posted: 2018-12-15 20:39

We are on RM 5.3.1 and after restarting today all of our collection service is failing the health check of redis answering. Redis seems fine and is passing all health checks. All other services are passing health checks as well except the collection services. What could be the issue? We didn't have to increase memory on collectorredis from 8 to 10gb and seem to continue to increase. We have restarted all services and services and docker to see if that would help and hasn't. We are seeing events but don't see any data on the graphs. Please help!

------------------------------
Devon Solomon
------------------------------


Subject: RE: Redis not answering to collection services
Author: Jason Olson
Posted: 2018-12-17 11:14

Memory leaks. Memory leaks everywhere in version 5.x daemons. I'd recommend moving to 6.2.1, as there are far fewer issues with the various collectors. It's still not perfect, nor as stable as version 4.2.5, but far better than 5.x.

------------------------------
Jason Olson
------------------------------

Subject: RE: Redis not answering to collection services
Author: Ryan Matte
Posted: 2018-12-26 13:02

That could be caused by a number of things.  Maybe the .rdb file for that redis instance is corrupted and needs to be re-created.  You would do that by stopping redis then finding the corresponding .rdb file for it and deleting it, then starting it again.  Example...

For collectorredis on the localhost collector:

/opt/serviced/var/volumes/6zbnssjd86cva5rtoa5ifuhsa/localhost/localhost_collectorredis/dump.rdb

For the main redis instance:

/opt/serviced/var/volumes/6zbnssjd86cva5rtoa5ifuhsa/redis/dump.rdb

If the redis instance is for a remote collector on a delegate host it may be stored on that host's filesystem rather than inside the main serviced volume if NFS / DFS isn't mounted on the remote host.

If it's a collectorredis that's failing it might be maxed out with metrics.  You could check the graphs for it in Control Center to see if the queue graph shows that it's filled up.  If it's the main redis instance you could check the same thing on it's graphs.  If it's getting backed up it means that there's likely a problem somewhere in the pipeline.



------------------------------
Ryan Matte
------------------------------


< Previous
How to get more info on a network outbound spike using Zenoss
  Next
Zenoss 6.2.1 zPingMonitorIgnore not taking effect
>