TECHZEN Zenoss User Community ARCHIVE  

Bare metal control center install problem

Subject: Bare metal control center install problem
Author: Kevin Kirmse
Posted: 2017-09-03 14:06

I am attempting to install the latest version of the control center onto a clean install of Centos 7.3

I have run through the install procedure several times and it looks like something is missing.

serviced is complaining about files not found in /opt/serviced/templates, this directory does not exist.

docker is complaining about missing serviced containers

docker journal

Started Docker Application Container Engine.
time="2017-09-03T10:49:31.891537140-06:00" level=error msg="Handler for GET /containers/serviced-isvcs_docker-registry/json returned error: No such container: serviced-isvcs_docker-registry"
time="2017-09-03T10:49:31.891546485-06:00" level=error msg="Handler for GET /containers/serviced-isvcs_zookeeper/json returned error: No such container: serviced-isvcs_zookeeper"
time="2017-09-03T10:49:31.891567839-06:00" level=error msg="Handler for GET /containers/serviced-isvcs_elasticsearch-logstash/json returned error: No such container: serviced-isvcs_elasticsearch-logstash"
time="2017-09-03T10:49:31.891659529-06:00" level=error msg="Handler for GET /containers/serviced-isvcs_opentsdb/json returned error: No such container: serviced-isvcs_opentsdb"
time="2017-09-03T10:49:31.893749287-06:00" level=error msg="Handler for GET /containers/serviced-isvcs_elasticsearch-serviced/json returned error: No such container: serviced-isvcs_elasticsearch-serviced"
time="2017-09-03T10:50:22.801987329-06:00" level=error msg="Handler for GET /containers/serviced-isvcs_logstash/json returned error: No such container: serviced-isvcs_logstash"
time="2017-09-03T10:50:22.803267933-06:00" level=error msg="Handler for GET /containers/serviced-isvcs_kibana/json returned error: No such container: serviced-isvcs_kibana"

lsblk --output=NAME,SIZE,TYPE,FSTYPE,MOUNTPOINT

NAME SIZE TYPE FSTYPE MOUNTPOINT
sda 931.5G disk
├─sda1 1G part xfs /boot
├─sda2 115.8G part LVM2_member
│ ├─cl-root 50G lvm xfs /
│ ├─cl-swap 15.8G lvm swap [SWAP]
│ └─cl-home 50G lvm xfs /home
├─sda3 500G part LVM2_member
│ ├─zenoss-docker--pool_tmeta 1G lvm
│ │ └─zenoss-docker--pool 50G lvm
│ │ ├─docker-253:0-100813006-66bcad99dc907fdaebdfdd16dbe8dcf6c38a54b2e9b9cffc6d3d1e8242aa5746 10G dm xfs /var/lib/docker/devicemapper/mnt/66bcad99dc907fdaebdfdd16dbe8dcf6c38a54b2e9b9cffc6d3d1e8242aa5746
│ │ ├─docker-253:0-100813006-ba3f1a94767a37060cba36039ec9ddea036ec46dc54fe50a419ae754f0cf7b80 10G dm xfs /var/lib/docker/devicemapper/mnt/ba3f1a94767a37060cba36039ec9ddea036ec46dc54fe50a419ae754f0cf7b80
│ │ ├─docker-253:0-100813006-b054551634c0945b2ffb137eb0a5a35f2e82e768e5fa4c108dcb7236b7d24146 10G dm xfs /var/lib/docker/devicemapper/mnt/b054551634c0945b2ffb137eb0a5a35f2e82e768e5fa4c108dcb7236b7d24146
│ │ ├─docker-253:0-100813006-1006057d0f9d0c2c27089666541ee3a45f10f496380f6da0df882efad7a5a0f5 10G dm xfs /var/lib/docker/devicemapper/mnt/1006057d0f9d0c2c27089666541ee3a45f10f496380f6da0df882efad7a5a0f5
│ │ ├─docker-253:0-100813006-b491ec3f28bf00272c2215dc4e13e6ddb34eab6850044ef8b3b1c31a317935bc 10G dm xfs /var/lib/docker/devicemapper/mnt/b491ec3f28bf00272c2215dc4e13e6ddb34eab6850044ef8b3b1c31a317935bc
│ │ ├─docker-253:0-100813006-f39eefb6121db6957a42d36af6859b43dea969a9900aea97a186bc68e3472b5a 10G dm xfs /var/lib/docker/devicemapper/mnt/f39eefb6121db6957a42d36af6859b43dea969a9900aea97a186bc68e3472b5a
│ │ └─docker-253:0-100813006-6897dfee27f160477260b6a3d166ab02879c4160b5ee0f45a3e770282dadc140 10G dm xfs /var/lib/docker/devicemapper/mnt/6897dfee27f160477260b6a3d166ab02879c4160b5ee0f45a3e770282dadc140
│ ├─zenoss-docker--pool_tdata 50G lvm
│ │ └─zenoss-docker--pool 50G lvm
│ │ ├─docker-253:0-100813006-66bcad99dc907fdaebdfdd16dbe8dcf6c38a54b2e9b9cffc6d3d1e8242aa5746 10G dm xfs /var/lib/docker/devicemapper/mnt/66bcad99dc907fdaebdfdd16dbe8dcf6c38a54b2e9b9cffc6d3d1e8242aa5746
│ │ ├─docker-253:0-100813006-ba3f1a94767a37060cba36039ec9ddea036ec46dc54fe50a419ae754f0cf7b80 10G dm xfs /var/lib/docker/devicemapper/mnt/ba3f1a94767a37060cba36039ec9ddea036ec46dc54fe50a419ae754f0cf7b80
│ │ ├─docker-253:0-100813006-b054551634c0945b2ffb137eb0a5a35f2e82e768e5fa4c108dcb7236b7d24146 10G dm xfs /var/lib/docker/devicemapper/mnt/b054551634c0945b2ffb137eb0a5a35f2e82e768e5fa4c108dcb7236b7d24146
│ │ ├─docker-253:0-100813006-1006057d0f9d0c2c27089666541ee3a45f10f496380f6da0df882efad7a5a0f5 10G dm xfs /var/lib/docker/devicemapper/mnt/1006057d0f9d0c2c27089666541ee3a45f10f496380f6da0df882efad7a5a0f5
│ │ ├─docker-253:0-100813006-b491ec3f28bf00272c2215dc4e13e6ddb34eab6850044ef8b3b1c31a317935bc 10G dm xfs /var/lib/docker/devicemapper/mnt/b491ec3f28bf00272c2215dc4e13e6ddb34eab6850044ef8b3b1c31a317935bc
│ │ ├─docker-253:0-100813006-f39eefb6121db6957a42d36af6859b43dea969a9900aea97a186bc68e3472b5a 10G dm xfs /var/lib/docker/devicemapper/mnt/f39eefb6121db6957a42d36af6859b43dea969a9900aea97a186bc68e3472b5a
│ │ └─docker-253:0-100813006-6897dfee27f160477260b6a3d166ab02879c4160b5ee0f45a3e770282dadc140 10G dm xfs /var/lib/docker/devicemapper/mnt/6897dfee27f160477260b6a3d166ab02879c4160b5ee0f45a3e770282dadc140
│ ├─zenoss-serviced--pool_tmeta 3.5G lvm
│ │ └─zenoss-serviced--pool 350G lvm
│ └─zenoss-serviced--pool_tdata 350G lvm
│ └─zenoss-serviced--pool 350G lvm
├─sda4 1K part
├─sda5 50G part xfs /opt/serviced/var/isvcs
└─sda6 264.8G part xfs /opt/serviced/var/backups

serviced journal

Started Zenoss ServiceD.
time="2017-09-03T16:49:31Z" level=info msg="Established connection to Docker" address="unix:///var/run/docker.sock" location="daemon.go:291" logger=cli.api
time="2017-09-03T16:49:31Z" level=info msg="Initialized application storage" args=[dm.thinpooldev=/dev/mapper/zenoss-serviced--pool dm.basesize=100G] driver=devicemapper location="daemon.go:313" logger=cli.api options=map[] path="/opt/serviced/var/volumes"
time="2017-09-03T16:49:31Z" level=info msg="Listening for incoming RPC requests" address=":4979" ciphersuite="TLS_ECDHE_RSA_WITH_AES_128_CBC_SHA (49171),TLS_RSA_WITH_AES_128_CBC_SHA (47),TLS_RSA_WITH_AES_256_CBC_SHA (53),TLS_ECDHE_RSA_WITH_AES_256_CBC_SHA (49172),TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256 (49199)" location="daemon.go:259" logger=cli.api server=rpc tls=true
time="2017-09-03T16:49:31Z" level=info msg="Established ZooKeeper connection" location="daemon.go:326" logger=cli.api
time="2017-09-03T16:49:42Z" level=info msg="Internal service checked in healthy" elapsed=764ns isvc=docker-registry location="container.go:793" logger=isvcs
time="2017-09-03T16:50:02Z" level=warning msg="Unable to validate health of ZooKeeper. Retrying silently" location="zk.go:86" logger=isvcs
time="2017-09-03T16:50:08Z" level=info msg="ZooKeeper checked in healthy" location="zk.go:132" logger=isvcs
time="2017-09-03T16:50:08Z" level=info msg="Internal service checked in healthy" elapsed=410ns isvc=zookeeper location="container.go:793" logger=isvcs
time="2017-09-03T16:50:13Z" level=warning msg="Elastic health reported below minimum" active_primary_shards=0 active_shards=0 cluster_name=edbhxkbjnwq4v6eyuzag8j4mf initializing_shards=1 location="es.go:264" logger=isvcs minhealth=yellow number_of_data_nodes=1 number_of_nodes=1 relocating_shards=0 reported=red timed_out=false unassigned_shards=0 url="http://localhost:9200/_cluster/health"
time="2017-09-03T16:50:14Z" level=warning msg="Elastic health reported below minimum" active_primary_shards=0 active_shards=0 cluster_name=edbhxkbjnwq4v6eyuzag8j4mf initializing_shards=1 location="es.go:264" logger=isvcs minhealth=yellow number_of_data_nodes=1 number_of_nodes=1 relocating_shards=0 reported=red timed_out=false unassigned_shards=0 url="http://localhost:9200/_cluster/health"
time="2017-09-03T16:50:15Z" level=info msg="Internal service checked in healthy" isvc=elasticsearch-serviced location="container.go:885" logger=isvcs
time="2017-09-03T16:50:15Z" level=info msg="Internal service checked in healthy" elapsed=22.000637394s isvc=elasticsearch-serviced location="container.go:793" logger=isvcs
time="2017-09-03T16:50:20Z" level=warning msg="Elastic health reported below minimum" active_primary_shards=0 active_shards=0 cluster_name=3jggywnoksmsq6hra86o2fde8 initializing_shards=0 location="es.go:264" logger=isvcs minhealth=yellow number_of_data_nodes=1 number_of_nodes=1 relocating_shards=0 reported=red timed_out=false unassigned_shards=1 url="http://localhost:9100/_cluster/health"
time="2017-09-03T16:50:22Z" level=info msg="Internal service checked in healthy" isvc=elasticsearch-logstash location="container.go:885" logger=isvcs
time="2017-09-03T16:50:22Z" level=info msg="Internal service checked in healthy" elapsed=22.000665403s isvc=elasticsearch-logstash location="container.go:793" logger=isvcs
time="2017-09-03T16:50:27Z" level=info msg="Started internal services" location="daemon.go:189" logger=cli.api
time="2017-09-03T16:50:27Z" level=info msg="Determined outbound IP address" address=192.168.167.105 location="daemon.go:435" logger=cli.api
time="2017-09-03T16:50:27Z" level=info msg="Created host" hostid=a8c069a7 ipaddress=192.168.167.105 location="host.go:238" logger=domain.host
time="2017-09-03T16:50:27Z" level=info msg="Loaded master keys from disk" keyfile="/opt/serviced/var/isvcs/.keys/master.keys" location="daemon.go:456" logger=cli.api
I0903 16:50:28.345300 04632 nfs.go:78] started nfs server:
time="2017-09-03T16:50:28Z" level=info msg="Established connection to metrics server" location="daemon.go:1121" logger=cli.api metricsaddr="http://127.0.0.1:8888"
I0903 16:50:28.797111 04632 controlplanedao.go:124] Opening ElasticSearch ControlPlane Dao: hostName=localhost, port=9200
time="2017-09-03T16:50:28Z" level=info msg="Started ServiceStateManager" location="servicestatemanager.go:281" logger=scheduler.servicestatemanager tenantids=[]
time="2017-09-03T16:50:28Z" level=info msg="Started service state manager" leveltimeout=10m0s location="daemon.go:1437" logger=cli.api
time="2017-09-03T16:50:29Z" level=info msg="Adding ResourcePool" action=add id=default success=true type=resourcepool user=system
time="2017-09-03T16:50:29Z" level=info msg="Checking docker registry" location="dfs.go:538" logger=facade
time="2017-09-03T16:50:29Z" level=info msg="No previous version of the docker registry exists; nothing to migrate" force=false fromregistryhost= location="dfs.go:483" logger=facade
time="2017-09-03T16:50:29Z" level=info msg="Started Control Center UI server" location="daemon.go:1335" logger=cli.api master="192.168.167.105:4979" uiport=":443"
time="2017-09-03T16:50:29Z" level=info msg="Started serviced master" location="daemon.go:558" logger=cli.api
time="2017-09-03T16:50:29Z" level=warning msg="Unable to autoload templates from the filesystem" error="lstat /opt/serviced/templates: no such file or directory" location="daemon.go:1380" logger=cli.api path="/opt/serviced/templates"
time="2017-09-03T16:50:29Z" level=info msg="Started TCP multiplexer" address=[::]:22250 location="mux.go:59" logger=proxy
time="2017-09-03T16:50:29Z" level=info msg="Determined delegate's outbound IP address" address=192.168.167.105 location="daemon.go:707" logger=cli.api
I0903 16:50:29.027604 04632 zzk.go:170] Starting a listener at /net/pub/master
I0903 16:50:29.030640 04632 zzk.go:170] Starting a listener at /pools
I0903 16:50:29.032391 04632 zzk.go:170] Starting a listener at /net/vhost/master
time="2017-09-03T16:50:29Z" level=info msg="Started service scheduler" location="daemon.go:1421" logger=cli.api
time="2017-09-03T16:50:29Z" level=info msg="Created host" hostid=a8c069a7 ipaddress=192.168.167.105 location="host.go:238" logger=domain.host
time="2017-09-03T16:50:29Z" level=warning msg="Unable to load delegate keys from file. Continuing to watch for changes" error="open /opt/serviced/etc/delegate.keys: no such file or directory" keyfile="/opt/serviced/etc/delegate.keys" location="localkeys.go:318" logger=auth
time="2017-09-03T16:50:29Z" level=warning msg="Unable to find pool assignment for this delegate. Has it been added via `serviced host add`? Will continue to retry silently" hostid=a8c069a7 location="daemon.go:823" logger=cli.api master="192.168.167.105:4979"
time="2017-09-03T16:50:29Z" level=info msg="Creating HTTP server" bindport=":443" ciphersuite=[TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256 (49199) TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256 (49195) TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384 (49200) TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384 (49196) TLS_ECDHE_RSA_WITH_AES_256_CBC_SHA (49172) TLS_ECDHE_RSA_WITH_AES_128_CBC_SHA (49171) TLS_ECDHE_RSA_WITH_3DES_EDE_CBC_SHA (49170) TLS_ECDHE_ECDSA_WITH_AES_256_CBC_SHA (49162) TLS_ECDHE_ECDSA_WITH_AES_128_CBC_SHA (49161) TLS_RSA_WITH_AES_256_CBC_SHA (53) TLS_RSA_WITH_AES_128_CBC_SHA (47) TLS_RSA_WITH_3DES_EDE_CBC_SHA (10) TLS_RSA_WITH_AES_128_GCM_SHA256 (156) TLS_RSA_WITH_AES_256_GCM_SHA384 (157)] location="cpserver.go:232" logger=web
I0903 16:50:29.150822 04632 scheduler.go:157] Resetting service locks
I0903 16:50:29.154077 04632 scheduler.go:165] DFS locks are all reset
I0903 16:50:29.154558 04632 zzk.go:170] Starting a listener at /pools
I0903 16:50:29.157227 04632 scheduler.go:244] Entering lead for realm default!
I0903 16:50:29.157268 04632 scheduler.go:245] Host Master successfully started
I0903 16:50:29.157378 04632 zzk.go:170] Starting a listener at /docker/registry/tags
I0903 16:50:29.157944 04632 zzk.go:170] Starting a listener at /pools/default/services
I0903 16:50:29.256807 04632 server.go:378] serviced exports:
/exports *(rw,fsid=0,no_root_squash,insecure,no_subtree_check,async,crossmnt)
I0903 16:50:29.283696 04632 nfs.go:78] started nfs server:
I0903 16:50:29.312282 04632 nfs.go:62] reloaded nfs server:




------------------------------
Kevin Kirmse
Electrical Engineer
FIC
------------------------------


< Previous
Zenoss 4.2.4 core download
  Next
Zenoss 5.2.6 services stuck starting in Docker
>