TECHZEN Zenoss User Community ARCHIVE  
FORUMS / Configuration & Administration / DISCUSSION

Delegate host doesn't start services.

Subject: Delegate host doesn't start services.
Author: VP
Posted: 2020-05-28 16:24

Hi,

I just installed a Zenoss core 6.3.2 on a VM and wanted to add 2 additional delegate hosts. I was able to add the hosts to a pool successfully but the services do not start on the delegate hosts, If i switch the zenoss services to default pool they just start fine.
In the control centre UI I see the delegate hosts Active/Authenticatin Blue tick is grayed out.

I was also not able to find docker images on the delegate hosts when i run docker images. DFS seems to be running fine but I am clueless as to where and what I should be checking.

------------------------------
vinay paduru
------------------------------

Subject: RE: Delegate host doesn't start services.
Author: Michael Rogers
Posted: 2020-06-02 15:23

Vinay,

You say you were able to deploy 2 delegate hosts and add them to a pool.  When you mention switching the Zenoss services to the default pool, do you mean deploying the application from its template?  I.e., after you added the delegate hosts to their own pool, did you create a new installation of Zenoss Core on that pool through the Deployment Wizard?

If so, you may wish to try again while tailing the systemd journal on those delegates to look for error messages.  From the command line of those hosts, you can issue a journalctl -flu serviced to determine what the Control Center service is doing and where it might be failing.  If you get some error messages that you don't immediately understand, feel free to paste them here.



------------------------------
Michael J. Rogers
Senior Instructor - Zenoss
Austin TX
------------------------------

Subject: RE: Delegate host doesn't start services.
Author: VP
Posted: 2020-06-03 09:27

Hello Michael,
Yes, I have deployed the application from its template after I added delegate hosts to their own pool.
Here is what I see on the Master, which are warnings/errors other events seem fine to me.. 
serviced[47649]: E0603 12:56:41.850593 47649 docker.go:34] Error checking Docker Hub login: config.json is not populated

serviced[47649]: time="2020-06-03T13:08:51Z" level=info msg="Updating Host" action=update id=c3f709fe success=true type=host user=system

serviced[47649]: time="2020-06-03T13:13:37Z" level=warning msg="Timeout waiting for service batch to reach desired state" batchsize=8 desiredstate=go emergency=false location="servicestatemanager.go:1059" logger=scheduler.servicestatemanager queue=go tenantid=acmm3lxfq8uatk27ql4w6ixx6 timeout=10m0s


Here is what I see on one delegate host.. after I tried restarting the serviceD service.
Started Zenoss ServiceD.
time="2020-06-03T13:08:46Z" level=info msg="Established connection to Docker" address="unix:///var/run/docker.sock" location="daemon.go:293" logger=cli.api

time="2020-06-03T13:08:46Z" level=info msg="Initialized application storage" args=[dm.basesize=100G] driver=nfs location="daemon.go:315" logger=cli.api options=map[] path="/opt/serviced/var/volumes"

time="2020-06-03T13:08:46Z" level=info msg="Listening for incoming RPC requests" address=":4979" ciphersuite="TLS_ECDHE_RSA_WITH_AES_128_CBC_SHA (49171),TLS_RSA_WITH_AES_128_CBC_SHA (47),TLS_RSA_WITH_AES_256_CBC_SHA (53),TLS_ECDHE_RSA_WITH_AES_256_CBC_SHA (49172),TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256 (49199)" location="daemon.go:261" logger=cli.api server=rpc tls=true

time="2020-06-03T13:08:46Z" level=info msg="Established ZooKeeper connection" location="daemon.go:328" logger=cli.api

time="2020-06-03T13:08:46Z" level=info msg="Started internal services" location="daemon.go:203" logger=cli.api services=[]

time="2020-06-03T13:08:46Z" level=info msg="Started TCP multiplexer" address=[::]:22250 location="mux.go:59" logger=proxy

time="2020-06-03T13:08:46Z" level=info msg="Determined delegate's outbound IP address" address=192.168.2.101 location="daemon.go:737" logger=cli.api

time="2020-06-03T13:08:46Z" level=info msg="Created host" hostid=c3f709fe ipaddress=192.168.2.101 location="host.go:242" logger=domain.host

time="2020-06-03T13:08:46Z" level=info msg="Loaded delegate keys from file" keyfile="/opt/serviced/etc/delegate.keys" location="localkeys.go:339" logger=auth

time="2020-06-03T13:08:46Z" level=warning msg="Unable to find pool assignment for this delegate. Has it been added via `serviced host add`? Will continue to retry silently" hostid=c3f709fe location="daemon.go:853" logger=cli.api master="192.168.2.100:4979"

time="2020-06-03T13:08:46Z" level=info msg="Received new authentication token" expiration=1591193326 location="token.go:62" logger=auth

time="2020-06-03T13:08:51Z" level=info msg="Determined pool assignment for this delegate" hostid=c3f709fe location="daemon.go:862" logger=cli.api master="192.168.2.100:4979" poolid=RM
W0603 13:09:46.557278 15031 connection.go:233] timed out waiting for connection​


------------------------------
vinay paduru
------------------------------

Vinay,

You say you were able to deploy 2 delegate hosts and add them to a pool.  When you mention switching the Zenoss services to the default pool, do you mean deploying the application from its template?  I.e., after you added the delegate hosts to their own pool, did you create a new installation of Zenoss Core on that pool through the Deployment Wizard?

If so, you may wish to try again while tailing the systemd journal on those delegates to look for error messages.  From the command line of those hosts, you can issue a journalctl -flu serviced to determine what the Control Center service is doing and where it might be failing.  If you get some error messages that you don't immediately understand, feel free to paste them here.



------------------------------
Michael J. Rogers
Senior Instructor - Zenoss
Austin TX
------------------------------

Subject: RE: Delegate host doesn't start services.
Author: Michael Rogers
Posted: 2020-06-03 15:24

Vinay,

The usual causes in situations like these can be traced to configuration issues or time synchronization.  It would help to see the contents of /etc/default/serviced on the master and at least one of the delegates, along with the contents of /etc/sysconfig/docker for the same.

While pulling those files, check that all 3 hosts are pointed to the same time server and that their clocks are synchronized.

------------------------------
Michael J. Rogers
Senior Instructor - Zenoss
Austin TX
------------------------------

Subject: RE: Delegate host doesn't start services.
Author: VP
Posted: 2020-06-03 16:35

Hi Michael,

Here is the config attached below 

Master 

 SERVICED_ADMIN_GROUP=sds_sec_sa
 SERVICED_DM_BASESIZE=450G
 SERVICED_DM_THINPOOLDEV=/dev/mapper/serviced-serviced--pool
 SERVICED_DOCKER_REGISTRY=192.168.2.100:5000
 SERVICED_ENDPOINT=192.168.2.100:4979
 SERVICED_FS_TYPE=devicemapper
 SERVICED_MASTER=1
 SERVICED_ZK=192.168.2.100:2181

Delegate

SERVICED_DOCKER_REGISTRY=192.168.2.100:5000
SERVICED_ENDPOINT=192.168.2.100:4979
SERVICED_LOG_ADDRESS=192.168.2.100:5042
SERVICED_LOGSTASH_ES=192.168.2.100:9100
SERVICED_MASTER=0
SERVICED_MASTER_IP=192.168.2.100
SERVICED_STATS_PORT=192.168.2.100:8443

SERVICED_DOCKER_REGISTRY=192.168.2.100:5000
SERVICED_ENDPOINT=192.168.2.100:4979
SERVICED_LOG_ADDRESS=192.168.2.100:5042
SERVICED_LOGSTASH_ES=192.168.2.100:9100
SERVICED_MASTER=0
SERVICED_MASTER_IP=192.168.2.100
SERVICED_STATS_PORT=192.168.2.100:8443

Master
OPTIONS="--log-level=error --storage-driver devicemapper --exec-opt native.cgroupdriver=cgroupfs --storage-opt dm.mountopt=discard --storage-opt dm.thinpooldev=/dev/mapper/rootdg-docker--pool --dns=172.17.0.1 --bip=172.17.0.1/16 --insecure-registry=192.168.2.100:5000"

Delegate

OPTIONS="--log-level=error --storage-driver devicemapper --exec-opt native.cgroupdriver=cgroupfs --storage-opt dm.mountopt=discard --storage-opt dm.thinpooldev=/dev/mapper/rootdg-docker--pool --dns=172.17.0.1 --bip=172.17.0.1/16 --insecure-registry=192.168.2.100:5000"

OPTIONS="--log-level=error --storage-driver devicemapper --exec-opt native.cgroupdriver=cgroupfs --storage-opt dm.mountopt=discard --storage-opt dm.thinpooldev=/dev/mapper/rootdg-docker--pool --dns=172.17.0.1 --bip=172.17.0.1/16 --insecure-registry=192.168.2.100:5000"

​

NTP seems to be in sync but not sure if its synced to the same server..

1591216196
1591216196
1591216196

timedatectl output; but I need to double confirm if they are pointing to same NTP server
NTP synchronized: yes
NTP synchronized: yes
NTP synchronized: yes

Is there any other config which I should be taking a look at ?



------------------------------
vinay paduru
------------------------------ 
serviced[47649]: E0603 12:56:41.850593 47649 docker.go:34] Error checking Docker Hub login: config.json is not populated

serviced[47649]: time="2020-06-03T13:08:51Z" level=info msg="Updating Host" action=update id=c3f709fe success=true type=host user=system

serviced[47649]: time="2020-06-03T13:13:37Z" level=warning msg="Timeout waiting for service batch to reach desired state" batchsize=8 desiredstate=go emergency=false location="servicestatemanager.go:1059" logger=scheduler.servicestatemanager queue=go tenantid=acmm3lxfq8uatk27ql4w6ixx6 timeout=10m0s


Here is what I see on one delegate host.. after I tried restarting the serviceD service.
Started Zenoss ServiceD.
time="2020-06-03T13:08:46Z" level=info msg="Established connection to Docker" address="unix:///var/run/docker.sock" location="daemon.go:293" logger=cli.api

time="2020-06-03T13:08:46Z" level=info msg="Initialized application storage" args=[dm.basesize=100G] driver=nfs location="daemon.go:315" logger=cli.api options=map[] path="/opt/serviced/var/volumes"

time="2020-06-03T13:08:46Z" level=info msg="Listening for incoming RPC requests" address=":4979" ciphersuite="TLS_ECDHE_RSA_WITH_AES_128_CBC_SHA (49171),TLS_RSA_WITH_AES_128_CBC_SHA (47),TLS_RSA_WITH_AES_256_CBC_SHA (53),TLS_ECDHE_RSA_WITH_AES_256_CBC_SHA (49172),TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256 (49199)" location="daemon.go:261" logger=cli.api server=rpc tls=true

time="2020-06-03T13:08:46Z" level=info msg="Established ZooKeeper connection" location="daemon.go:328" logger=cli.api

time="2020-06-03T13:08:46Z" level=info msg="Started internal services" location="daemon.go:203" logger=cli.api services=[]

time="2020-06-03T13:08:46Z" level=info msg="Started TCP multiplexer" address=[::]:22250 location="mux.go:59" logger=proxy

time="2020-06-03T13:08:46Z" level=info msg="Determined delegate's outbound IP address" address=192.168.2.101 location="daemon.go:737" logger=cli.api

time="2020-06-03T13:08:46Z" level=info msg="Created host" hostid=c3f709fe ipaddress=192.168.2.101 location="host.go:242" logger=domain.host

time="2020-06-03T13:08:46Z" level=info msg="Loaded delegate keys from file" keyfile="/opt/serviced/etc/delegate.keys" location="localkeys.go:339" logger=auth

time="2020-06-03T13:08:46Z" level=warning msg="Unable to find pool assignment for this delegate. Has it been added via `serviced host add`? Will continue to retry silently" hostid=c3f709fe location="daemon.go:853" logger=cli.api master="192.168.2.100:4979"

time="2020-06-03T13:08:46Z" level=info msg="Received new authentication token" expiration=1591193326 location="token.go:62" logger=auth

time="2020-06-03T13:08:51Z" level=info msg="Determined pool assignment for this delegate" hostid=c3f709fe location="daemon.go:862" logger=cli.api master="192.168.2.100:4979" poolid=RM
W0603 13:09:46.557278 15031 connection.go:233] timed out waiting for connection​


------------------------------
vinay paduru

Vinay,

You say you were able to deploy 2 delegate hosts and add them to a pool.  When you mention switching the Zenoss services to the default pool, do you mean deploying the application from its template?  I.e., after you added the delegate hosts to their own pool, did you create a new installation of Zenoss Core on that pool through the Deployment Wizard?

If so, you may wish to try again while tailing the systemd journal on those delegates to look for error messages.  From the command line of those hosts, you can issue a journalctl -flu serviced to determine what the Control Center service is doing and where it might be failing.  If you get some error messages that you don't immediately understand, feel free to paste them here.



------------------------------
Michael J. Rogers
Senior Instructor - Zenoss
Austin TX

Subject: RE: Delegate host doesn't start services.
Author: Ryan Matte
Posted: 2020-06-05 16:49

You are hitting the following error on your delegate...

W0603 13:09:46.557278 15031 connection.go:233] timed out waiting for connection​

Line 233 in that connection.go code is where it attempts to connect to zookeeper.  Based on the config you provided for your delegate in your last note you are missing the SERVICED_ZK which means your delegate doesn't know what it should be connecting to for zookeeper.  SERVICED_ZK needs to be defined on the delegates as well, not just on the master.  You need to fix that and then double check the rest of your config to make sure it's correct.

See: https://help.zenoss.com/zsd/cc/installing-control-center/configuring-and-starting-delegate-hosts/setting-internal-services-endpoints



------------------------------
Ryan Matte
------------------------------ 
Master 

 SERVICED_ADMIN_GROUP=sds_sec_sa
 SERVICED_DM_BASESIZE=450G
 SERVICED_DM_THINPOOLDEV=/dev/mapper/serviced-serviced--pool
 SERVICED_DOCKER_REGISTRY=192.168.2.100:5000
 SERVICED_ENDPOINT=192.168.2.100:4979
 SERVICED_FS_TYPE=devicemapper
 SERVICED_MASTER=1
 SERVICED_ZK=192.168.2.100:2181

Delegate

SERVICED_DOCKER_REGISTRY=192.168.2.100:5000
SERVICED_ENDPOINT=192.168.2.100:4979
SERVICED_LOG_ADDRESS=192.168.2.100:5042
SERVICED_LOGSTASH_ES=192.168.2.100:9100
SERVICED_MASTER=0
SERVICED_MASTER_IP=192.168.2.100
SERVICED_STATS_PORT=192.168.2.100:8443

SERVICED_DOCKER_REGISTRY=192.168.2.100:5000
SERVICED_ENDPOINT=192.168.2.100:4979
SERVICED_LOG_ADDRESS=192.168.2.100:5042
SERVICED_LOGSTASH_ES=192.168.2.100:9100
SERVICED_MASTER=0
SERVICED_MASTER_IP=192.168.2.100
SERVICED_STATS_PORT=192.168.2.100:8443

Master
OPTIONS="--log-level=error --storage-driver devicemapper --exec-opt native.cgroupdriver=cgroupfs --storage-opt dm.mountopt=discard --storage-opt dm.thinpooldev=/dev/mapper/rootdg-docker--pool --dns=172.17.0.1 --bip=172.17.0.1/16 --insecure-registry=192.168.2.100:5000"

Delegate

OPTIONS="--log-level=error --storage-driver devicemapper --exec-opt native.cgroupdriver=cgroupfs --storage-opt dm.mountopt=discard --storage-opt dm.thinpooldev=/dev/mapper/rootdg-docker--pool --dns=172.17.0.1 --bip=172.17.0.1/16 --insecure-registry=192.168.2.100:5000"

OPTIONS="--log-level=error --storage-driver devicemapper --exec-opt native.cgroupdriver=cgroupfs --storage-opt dm.mountopt=discard --storage-opt dm.thinpooldev=/dev/mapper/rootdg-docker--pool --dns=172.17.0.1 --bip=172.17.0.1/16 --insecure-registry=192.168.2.100:5000"

​

NTP seems to be in sync but not sure if its synced to the same server..

1591216196
1591216196
1591216196

timedatectl output; but I need to double confirm if they are pointing to same NTP server
NTP synchronized: yes
NTP synchronized: yes
NTP synchronized: yes

Is there any other config which I should be taking a look at ?



------------------------------
vinay paduru
------------------------------ 
serviced[47649]: E0603 12:56:41.850593 47649 docker.go:34] Error checking Docker Hub login: config.json is not populated

serviced[47649]: time="2020-06-03T13:08:51Z" level=info msg="Updating Host" action=update id=c3f709fe success=true type=host user=system

serviced[47649]: time="2020-06-03T13:13:37Z" level=warning msg="Timeout waiting for service batch to reach desired state" batchsize=8 desiredstate=go emergency=false location="servicestatemanager.go:1059" logger=scheduler.servicestatemanager queue=go tenantid=acmm3lxfq8uatk27ql4w6ixx6 timeout=10m0s


Here is what I see on one delegate host.. after I tried restarting the serviceD service.
Started Zenoss ServiceD.
time="2020-06-03T13:08:46Z" level=info msg="Established connection to Docker" address="unix:///var/run/docker.sock" location="daemon.go:293" logger=cli.api

time="2020-06-03T13:08:46Z" level=info msg="Initialized application storage" args=[dm.basesize=100G] driver=nfs location="daemon.go:315" logger=cli.api options=map[] path="/opt/serviced/var/volumes"

time="2020-06-03T13:08:46Z" level=info msg="Listening for incoming RPC requests" address=":4979" ciphersuite="TLS_ECDHE_RSA_WITH_AES_128_CBC_SHA (49171),TLS_RSA_WITH_AES_128_CBC_SHA (47),TLS_RSA_WITH_AES_256_CBC_SHA (53),TLS_ECDHE_RSA_WITH_AES_256_CBC_SHA (49172),TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256 (49199)" location="daemon.go:261" logger=cli.api server=rpc tls=true

time="2020-06-03T13:08:46Z" level=info msg="Established ZooKeeper connection" location="daemon.go:328" logger=cli.api

time="2020-06-03T13:08:46Z" level=info msg="Started internal services" location="daemon.go:203" logger=cli.api services=[]

time="2020-06-03T13:08:46Z" level=info msg="Started TCP multiplexer" address=[::]:22250 location="mux.go:59" logger=proxy

time="2020-06-03T13:08:46Z" level=info msg="Determined delegate's outbound IP address" address=192.168.2.101 location="daemon.go:737" logger=cli.api

time="2020-06-03T13:08:46Z" level=info msg="Created host" hostid=c3f709fe ipaddress=192.168.2.101 location="host.go:242" logger=domain.host

time="2020-06-03T13:08:46Z" level=info msg="Loaded delegate keys from file" keyfile="/opt/serviced/etc/delegate.keys" location="localkeys.go:339" logger=auth

time="2020-06-03T13:08:46Z" level=warning msg="Unable to find pool assignment for this delegate. Has it been added via `serviced host add`? Will continue to retry silently" hostid=c3f709fe location="daemon.go:853" logger=cli.api master="192.168.2.100:4979"

time="2020-06-03T13:08:46Z" level=info msg="Received new authentication token" expiration=1591193326 location="token.go:62" logger=auth

time="2020-06-03T13:08:51Z" level=info msg="Determined pool assignment for this delegate" hostid=c3f709fe location="daemon.go:862" logger=cli.api master="192.168.2.100:4979" poolid=RM
W0603 13:09:46.557278 15031 connection.go:233] timed out waiting for connection​


------------------------------
vinay paduru

Vinay,

You say you were able to deploy 2 delegate hosts and add them to a pool.  When you mention switching the Zenoss services to the default pool, do you mean deploying the application from its template?  I.e., after you added the delegate hosts to their own pool, did you create a new installation of Zenoss Core on that pool through the Deployment Wizard?

If so, you may wish to try again while tailing the systemd journal on those delegates to look for error messages.  From the command line of those hosts, you can issue a journalctl -flu serviced to determine what the Control Center service is doing and where it might be failing.  If you get some error messages that you don't immediately understand, feel free to paste them here.



------------------------------
Michael J. Rogers
Senior Instructor - Zenoss
Austin TX
Subject: RE: Delegate host doesn't start services.
Author: Ryan Matte
Posted: 2020-06-05 17:01

You also need to check and make sure that your delegates are able to hit port 2181 on 192.168.2.100.

------------------------------
Ryan Matte
------------------------------

Subject: RE: Delegate host doesn't start services.
Author: VP
Posted: 2020-06-06 02:38

Thank you Ryan, It is working as expected now..

------------------------------
vinay paduru
------------------------------


< Previous
Community Distributed collector on Zenoss CORE 6 		  Next
Configure Trusted Domain Servers 		>