TECHZEN Zenoss User Community ARCHIVE  

Logasth_Elastic Search not working

Subject: Logasth_Elastic Search not working
Author: [Not Specified]
Posted: 2015-07-10 12:08

All my graphs/logs in CC UI are not working. When I attach to the elasticsearch container this is what I get

015-07-07 15:08:23,691][DEBUG][action.bulk ] [elasticsearch-logstash] observer: timeout notification from cluster service. timeout setting [1m], time since start [1m]
[2015-07-07 15:08:23,691][DEBUG][action.bulk ] [elasticsearch-logstash] observer: timeout notification from cluster service. timeout setting [1m], time since start [1m]
[2015-07-07 15:08:23,691][DEBUG][action.bulk ] [elasticsearch-logstash] observer: timeout notification from cluster service. timeout setting [1m], time since start [1m]
[2015-07-07 15:08:23,691][DEBUG][action.bulk ] [elasticsearch-logstash] observer: timeout notification from cluster service. timeout setting [1m], time since start [1m]
[2015-07-07 15:08:23,692][DEBUG][action.bulk ] [elasticsearch-logstash] observer: timeout notification from cluster service. timeout setting [1m], time since start [1m]
[2015-07-07 15:08:33,785][WARN ][indices.cluster ] [elasticsearch-logstash] [logstash-2015.07.02][1] failed to start shard
org.elasticsearch.index.gateway.IndexShardGatewayRecoveryException: [logstash-2015.07.02][1] failed to recover shard
at org.elasticsearch.index.gateway.local.LocalIndexShardGateway.recover(LocalIndexShardGateway.java:269)
at org.elasticsearch.index.gateway.IndexShardGatewayService$1.run(IndexShardGatewayService.java:132)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:745)
Caused by: org.elasticsearch.ElasticsearchIllegalArgumentException: No version type match [34]
at org.elasticsearch.index.VersionType.fromValue(VersionType.java:307)
at org.elasticsearch.index.translog.Translog$Create.readFrom(Translog.java:364)
at org.elasticsearch.index.translog.TranslogStreams.readTranslogOperation(TranslogStreams.java:52)
at org.elasticsearch.index.gateway.local.LocalIndexShardGateway.recover(LocalIndexShardGateway.java:241)
... 4 more
[2015-07-07 15:08:33,824][WARN ][cluster.action.shard ] [elasticsearch-logstash] [logstash-2015.07.02][1] sending failed shard for [logstash-2015.07.02][1], node[iA5GdFdmTciLfH9eg2XQ1Q], [P], s[INITIALIZING], indexUUID [mx__bjdYQbO9eViXOi9REQ], reason [Failed to start shard, message [IndexShardGatewayRecoveryException; ]]
[2015-07-07 15:08:33,824][WARN ][cluster.action.shard ] [elasticsearch-logstash] [logstash-2015.07.02][1] received shard failed for [logstash-2015.07.02][1], node[iA5GdFdmTciLfH9eg2XQ1Q], [P], s[INITIALIZING], indexUUID [mx__bjdYQbO9eViXOi9REQ], reason [Failed to start shard, message [IndexShardGatewayRecoveryException; ]]
^C[2015-07-07 15:08:34,295][WARN ][indices.cluster ] [elasticsearch-logstash] [logstash-2015.07.03][1] failed to start shard
org.elasticsearch.index.gateway.IndexShardGatewayRecoveryException: [logstash-2015.07.03][1] failed to recover shard
at org.elasticsearch.index.gateway.local.LocalIndexShardGateway.recover(LocalIndexShardGateway.java:269)
at org.elasticsearch.index.gateway.IndexShardGatewayService$1.run(IndexShardGatewayService.java:132)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:745)
Caused by: org.elasticsearch.ElasticsearchIllegalArgumentException: No version type match [122]
at org.elasticsearch.index.VersionType.fromValue(VersionType.java:307)
at org.elasticsearch.index.translog.Translog$Create.readFrom(Translog.java:364)
at org.elasticsearch.index.translog.TranslogStreams.readTranslogOperation(TranslogStreams.java:52)
at org.elasticsearch.index.gateway.local.LocalIndexShardGateway.recover(LocalIndexShardGateway.java:241)
... 4 more
[2015-07-07 15:08:34,314][WARN ][cluster.action.shard ] [elasticsearch-logstash] [logstash-2015.07.03][1] sending failed shard for [logstash-2015.07.03][1], node[iA5GdFdmTciLfH9eg2XQ1Q], [P], s[INITIALIZING], indexUUID [YICCx6eTSROaoLPVDuopuw], reason [Failed to start shard, message [IndexShardGatewayRecoveryException; ]]
[2015-07-07 15:08:34,314][WARN ][cluster.action.shard ] [elasticsearch-logstash] [logstash-2015.07.03][1] received shard failed for [logstash-2015.07.03][1], node[iA5GdFdmTciLfH9eg2XQ1Q], [P], s[INITIALIZING], indexUUID [YICCx6eTSROaoLPVDuopuw], reason [Failed to start shard, message [IndexShardGatewayRecoveryException; ]]
[2015-07-07 15:08:34,904][WARN ][indices.cluster ] [elasticsearch-logstash] [logstash-2015.07.03][3] failed to start shard
org.elasticsearch.index.gateway.IndexShardGatewayRecoveryException: [logstash-2015.07.03][3] failed to recover shard
at org.elasticsearch.index.gateway.local.LocalIndexShardGateway.recover(LocalIndexShardGateway.java:269)
at org.elasticsearch.index.gateway.IndexShardGatewayService$1.run(IndexShardGatewayService.java:132)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:745)
Caused by: org.elasticsearch.ElasticsearchIllegalArgumentException: No version type match [48]
at org.elasticsearch.index.VersionType.fromValue(VersionType.java:307)
at org.elasticsearch.index.translog.Translog$Create.readFrom(Translog.java:364)
at org.elasticsearch.index.translog.TranslogStreams.readTranslogOperation(TranslogStreams.java:52)
at org.elasticsearch.index.gateway.local.LocalIndexShardGateway.recover(LocalIndexShardGateway.java:241)
... 4 more

Something is corrupted or broken inside my elasticsearch container. How do you go about troubleshooting this Is there a way to get a fresh image of elasticsearch and redeploy it without losing all my previous logs



Subject: file a bug, lets see what
Author: Andrew Kirch
Posted: 2015-07-20 12:00

file a bug, lets see what engineering says. http://jira.zenoss.com

Andrew Kirch

akirch@gvit.com

Need Zenoss support, consulting or custom development Look no further. Email or PM me!

Ready for Distributed Topology (collectors) for Zenoss 5 Coming May 1st from GoVanguard



Subject: This has been fixed looks
Author: [Not Specified]
Posted: 2015-07-28 10:55

This has been fixed looks like it was a corrupted Elasticsearch_logstash had to curl -xdelete logstash indexes. Then restart Control Center. That fixes the problem :D



Subject: awesome. If you can figure
Author: Andrew Kirch
Posted: 2015-07-29 09:03

awesome. If you can figure out why it was corrupted please file a bug so that we can repair it!

Andrew Kirch

akirch@gvit.com

Need Zenoss support, consulting or custom development Look no further. Email or PM me!

Ready for Distributed Topology (collectors) for Zenoss 5 Coming May 1st from GoVanguard



< Previous
Zenoss 5 graph thresholds
  Next
[zenoss 4.2.4] How to purge ALL archived events??
>