TECHZEN Zenoss User Community ARCHIVE  

Zenoss refuses to connect

Subject: Zenoss refuses to connect
Author: Gregg Hughes
Posted: 2018-01-17 15:35

Good afternoon, all!

I have (HAD) a Zenoss 5 installation running. I haven't checked on it for a couple of months, and when I needed to get some information found that Zenoss refuses to connect to either the main console or to the Control Center. I can and have root access to the appliance. However, after that I don't have a clear path to troubleshooting this issue. Reboots have not helped. I find the behaviour confusing since it has run flawlessly up to now.

Is there a document on troubleshooting this issue? Haven't found one, but my Google-fu may be weak. 

Thanks to all for looking!

Gregg


Subject: RE: Zenoss refuses to connect
Author: Arthur
Posted: 2018-01-18 15:27

Hi Gregg

// Please post

# cat /etc/centos-release && uname -a
# rpm -qa |grep docker && rpm -qa |grep serviced


// Take a time stamp and restart docker & serviced

# myDate=$(date "+%F %T")

# systemctl stop docker

# systemctl daemon-reload

# systemctl start docker

# docker ps >/tmp/serviced.log

# systemctl status docker >> /tmp/serviced.log
# systemctl status serviced >>/tmp/serviced.log

# journalctl --since "$myDate" >> /tmp/serviced.log

//Attach /tmp/serviced.log

Cheers

------------------------------
Arthur
------------------------------


Subject: RE: Zenoss refuses to connect
Author: Gregg Hughes
Posted: 2018-01-18 17:18

Hello, Arthur!

Here are the answers to the first questions
  cat /etc/centos-release && uname -a
   CentOS Linux release 7.3.1611 (Core)
   Linux ucspm 3.10.0-514.2.2.el7.x86_64 #1 SMP Tue Dec 6 23:06:41 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux

  rpm -qa |grep docker && rpm -qa |grep serviced
   docker-engine-selinux-1.12.5-1.el7.centos.noarch
   docker-engine-1.12.1-1.el7.centos.x86_64
   serviced-1.2.2-1.x86_64

There's something in the log showing something with a Thin Pool not having enough space. Here's the df -h:
   [root@ucspm ~]# df -h
   Filesystem Size Used Avail Use% Mounted on
   /dev/sda1 30G 6.3G 24G 22% /
   devtmpfs 7.8G 0 7.8G 0% /dev
   tmpfs 7.8G 0 7.8G 0% /dev/shm
   tmpfs 7.8G 548K 7.8G 1% /run
   tmpfs 7.8G 0 7.8G 0% /sys/fs/cgroup
   /dev/sdf1 147G 3.7G 143G 3% /opt/serviced/var/backups
   /dev/sde1 49G 4.5G 45G 10% /opt/serviced/var/isvcs
   /dev/sdd1 16G 34M 16G 1% /tmp
   tmpfs 1.6G 0 1.6G 0% /run/user/0
   [root@ucspm ~]#

I don't know about managing this Thin Pool, but google is my friend.....

Thanks!

Gregg

------------------------------
Gregg Hughes
Sr Systems Engineer
ISC International Limited
Milwaukee WI
4147210301
------------------------------

Attachments:

serviced.log.txt



Subject: RE: Zenoss refuses to connect
Author: Arthur
Posted: 2018-01-20 08:29

Hi Gregg

Zenoss V5 uses two thin-pools

docker-pool  for Docker Data and serviced-pool for Application Data

https://www.zenoss.com/sites/default/files/Control_Center_Planning_Guide_r1.5.0_d1930.17.331.pdf

These blocks are requested and mounted on demand. That's why you don't see it in your df –h output.

As from the logs I can't tell you which one has not enough free blocks. I would start with the following command to figure out which thin-pool is causing the issue:

# lsblk

# vgs

# lvs

Cheers



------------------------------
Arthur
------------------------------


Subject: RE: Zenoss refuses to connect
Author: Gregg Hughes
Posted: 2018-01-22 13:00

Hello, Arthur!

Thanks for the information. I've attached the results of the commands you outlined and I'm trawling through the Docker pages to see if I can find out what the recommended values should be.  So far, everything looks to be in order. 

FYI, this was the appliance I deployed, not an ISO or a repository installation.

Thanks very much!

Gregg



------------------------------
Gregg Hughes
Sr Systems Engineer
ISC International Limited
Milwaukee WI
4147210301
------------------------------

Attachments:

docker_virtual_disks.txt



Subject: RE: Zenoss refuses to connect
Author: Gregg Hughes
Posted: 2018-01-22 14:52

Hello, Arthur!

Fixed it. Linky to the magic buttons.....

Apparently every time I've rebooted the VM or restarted Docker it's started a new snapshot, not just the application. Figured it couldn't get more broken, so gave it a try and it's back - almost. There haven't been any statistics collected for about a month so I have some empty graphs that should get filled in as things go.

Thanks for the direction - going to get "Docker for Dummies" out and start studying.....

Gregg

------------------------------
Gregg Hughes
Sr Systems Engineer
ISC International Limited
Milwaukee WI
4147210301
------------------------------


Subject: RE: Zenoss refuses to connect
Author: Arthur
Posted: 2018-01-23 15:11

Hi Gregg

You're welcome. Appreciated your feedback and how you fixed it. I hope it will help others too.

Cheers



------------------------------
Arthur
------------------------------


< Previous
Upgrading Zenoss Core 6.0.0 to 6.1.0
  Next
Unable to get the Zenoss core UI
>