Zenoss ZenTech Community

I have had two of my ZenOSS disks mysteriously fill up after at least one them has consistently been at about 50% use.

I have extended the disk storage capacity and rebooted the server only to find that I'm getting stuck because possibly of one services is not starting:
serviced-isvcs_logstash.

I'm not sure exactly how to resolve this as I've been following the steps in: https://support.zenoss.com/hc/en-us/articles/204643769-How-to-Recover-Control-Center-from-Hardware-Failure

thin_check /var/lib/docker/devicemapper/devicemapper/metadata (no error)
thin_check --clear-needs-check-flag /var/lib/docker/devicemapper/devicemapper/metadata

After that I tried to start to no avail so I followed the guide to run: docker rm $(docker ps -qa)

After that same thing. I don't quite understand the message as the outcome is just the one service failing.

Unfortunately, given the changes going on, the loss of ZenOSS is proving to be pretty troublesome.

Subject:	I have been trying to post
Author:	J K
Posted:	2016-08-10 21:58

I have been trying to post this extra code from the logs:

856:Aug 11 08:55:46 myhost.domain.com[8443]: time="2016-08-11T08:55:46.658466145+08:00" level=info msg="POST /containers/createname=serviced-isvcs_logstash"

857-Aug 11 08:55:46 myhost.domain.com[8443]: time="2016-08-11T08:55:46.662420790+08:00" level=error msg="Handler for POST /containers/create returned error: Could not find container f
or entity id 84788db658d9da2636da4ec71358691344160869f6c1d210433f83807ec30b47"

858-Aug 11 08:55:46 myhost.domain.com[8443]: time="2016-08-11T08:55:46.662449788+08:00" level=error msg="HTTP Error" err="Could not find container for entity id 84788db658d9da2636da4e
c71358691344160869f6c1d210433f83807ec30b47" statusCode=500

Subject:	Solution to Control Center offline after disk full
Author:	J K
Posted:	2016-08-10 23:50

Well, I've been at this on and off for two days even before the first posting. There were some details I did omit and one of them was that this is running inside an ESXi environment and since I took a snapshot at the failure point, I was able to keep trying different things.

Finally I found this: https://github.com/docker/docker/issues/17691

In checking for the container entity id "84788db658d9da2636da4ec71358691344160869f6c1d210433f83807ec30b47" I found nothing in /var/lib/docker/containers

The final solution to bring control center up in this case was (as root):

thin_check --clear-needs-check-flag /var/lib/docker/devicemapper/devicemapper/metadata

sqlite3 linkgraph.db
update edge set name='oldserviced-isvcs_logstash' where name='serviced-isvcs_logstash';
.exit

^^^

service serviced start

At the end it started up as if nothing happened and I still have a working ZenOSS install with all data in it. Funny thing is the disk consumption halved on its own.

The only downside is that I am running on CentOS 7 (while I've always had ZenOSS on Ubuntu) and at least for me, it does not seem to have any updates for docker.

I hope this helps someone as troubleshooting with the docker layer is not as easy as it was in the pre 5.x versions.

Subject:	Control Center offline after disk full: serviced-isvcs_logstash
Author:	J K
Posted:	2016-08-10 20:20

Control Center offline after disk full: serviced-isvcs_logstash