![]() |
![]() |
Subject: | serviced-isvcs_elasticsearch-logstash container will not start |
Author: | [Not Specified] |
Posted: | 2015-10-28 02:28 |
Hi all,
Recently my /opt/serviced/var/volumes filled up (on btrfs).
I resized the LVM and filesystem.
However the elasticsearch-logstash container will not start.
This is from /var/log/upstart/serviced.logv:
I1028 07:20:50.430346 09240 container.go:630] waiting for elasticsearch-logstash to start, checking health status again in 1 second
E1028 07:21:01.431465 09240 container.go:655] healthcheck timed out for elasticsearch-logstash
E1028 07:21:01.431716 09240 container.go:372] Healthcheck for elasticsearch-logstash failed: healthcheck timed out
W1028 07:21:01.435610 09240 container.go:397] Stopping isvc container serviced-isvcs_elasticsearch-logstash
E1028 07:21:46.582181 09240 manager.go:418] Error starting isvc elasticsearch-logstash: healthcheck timed out
I1028 07:21:46.582354 09240 container.go:473] Starting isvc logstash
I1028 07:21:46.593573 09240 container.go:334] isvc serviced-isvcs_logstash found but not running; removing container c2f132fde0c1db7ee8d5681e01e2bc4deabcf7cf30ea9c3b4e2474e1dfab51b5
I1028 07:21:46.681597 09240 container.go:347] Creating a new container for isvc serviced-isvcs_logstash
E1028 07:21:46.778671 09240 manager.go:418] Error starting isvc logstash: API error (500): Could not get container for serviced-isvcs_elasticsearch-logstash
F1028 07:21:46.778968 09240 daemon.go:145] Could not start isvcs: manager: could not start 2 isvcs
And from the docker logs for that container I get this error
[2015-10-28 07:19:10,742][INFO ][discovery ] [elasticsearch-serviced] dadlwag0fgu46sdqoo8hczuz9/E5FiuuQrTjyHLgzZEQwLbw
[2015-10-28 07:19:10,883][INFO ][http ] [elasticsearch-serviced] bound_address {inet[/0:0:0:0:0:0:0:0:9200]}, publish_address {inet[/172.17.0.58:9200]}
[2015-10-28 07:19:10,884][INFO ][node ] [elasticsearch-serviced] started
[2015-10-28 07:19:11,881][INFO ][gateway ] [elasticsearch-serviced] recovered [1] indices into cluster_state
[2015-10-28 07:19:02,578][INFO ][node ] [elasticsearch-logstash] version[1.3.1], pid[7], build[2de6dc5/2014-07-28T14:45:15Z]
[2015-10-28 07:19:02,579][INFO ][node ] [elasticsearch-logstash] initializing ...
[2015-10-28 07:19:02,589][INFO ][plugins ] [elasticsearch-logstash] loaded [], sites [head]
[2015-10-28 07:19:07,811][INFO ][node ] [elasticsearch-logstash] initialized
[2015-10-28 07:19:07,811][INFO ][node ] [elasticsearch-logstash] starting ...
[2015-10-28 07:19:07,961][INFO ][transport ] [elasticsearch-logstash] bound_address {inet[/0:0:0:0:0:0:0:0:9300]}, publish_address {inet[/172.17.0.57:9300]}
[2015-10-28 07:19:08,011][INFO ][discovery ] [elasticsearch-logstash] 1kslyxzbk0zlnqabik9gotuas/f84x74iORm-rR54z-8r4tw
[2015-10-28 07:19:09,175][WARN ][discovery.zen.ping.multicast] [elasticsearch-logstash] failed to read requesting data from /172.17.0.58:54328
java.io.IOException: No transport address mapped to [25964]
at org.elasticsearch.common.transport.TransportAddressSerializers.addressFromStream(TransportAddressSerializers.java:71)
at org.elasticsearch.cluster.node.DiscoveryNode.readFrom(DiscoveryNode.java:267)
at org.elasticsearch.cluster.node.DiscoveryNode.readNode(DiscoveryNode.java:257)
at org.elasticsearch.discovery.zen.ping.multicast.MulticastZenPing$Receiver.onMessage(MulticastZenPing.java:319)
at org.elasticsearch.common.network.MulticastChannel$Plain$Receiver.run(MulticastChannel.java:364)
at java.lang.Thread.run(Thread.java:745)
[2015-10-28 07:19:11,090][INFO ][cluster.service ] [elasticsearch-logstash] new_master [elasticsearch-logstash][f84x74iORm-rR54z-8r4tw][21e16b624ce7][inet[/172.17.0.57:9300]], reason: zen-disco-join (elected_as_master)
[2015-10-28 07:19:11,393][INFO ][http ] [elasticsearch-logstash] bound_address {inet[/0:0:0:0:0:0:0:0:9100]}, publish_address {inet[/172.17.0.57:9100]}
[2015-10-28 07:19:11,399][INFO ][node ] [elasticsearch-logstash] started
[2015-10-28 07:19:14,114][INFO ][gateway ] [elasticsearch-logstash] recovered [10] indices into cluster_state
Any help would be greatly appreciated.
Subject: | I have the same issues that |
Author: | Tony H |
Posted: | 2015-10-28 08:46 |
I have the same issues that occurred yesterday for the same reason (full volume). This is a production box on my end so I am working to try and find a solution, but have yet to. Hopefully we come across something soon and if so I will share.
< |
Previous Zenoss core 4.2.5 WINRM no data collection |
Next Event Transforms Not Working |
> |