![]() |
![]() |
Subject: | Logasth_Elastic Search not working |
Author: | [Not Specified] |
Posted: | 2015-07-10 12:08 |
All my graphs/logs in CC UI are not working. When I attach to the elasticsearch container this is what I get
015-07-07 15:08:23,691][DEBUG][action.bulk ] [elasticsearch-logstash] observer: timeout notification from cluster service. timeout setting [1m], time since start [1m]
[2015-07-07 15:08:23,691][DEBUG][action.bulk ] [elasticsearch-logstash] observer: timeout notification from cluster service. timeout setting [1m], time since start [1m]
[2015-07-07 15:08:23,691][DEBUG][action.bulk ] [elasticsearch-logstash] observer: timeout notification from cluster service. timeout setting [1m], time since start [1m]
[2015-07-07 15:08:23,691][DEBUG][action.bulk ] [elasticsearch-logstash] observer: timeout notification from cluster service. timeout setting [1m], time since start [1m]
[2015-07-07 15:08:23,692][DEBUG][action.bulk ] [elasticsearch-logstash] observer: timeout notification from cluster service. timeout setting [1m], time since start [1m]
[2015-07-07 15:08:33,785][WARN ][indices.cluster ] [elasticsearch-logstash] [logstash-2015.07.02][1] failed to start shard
org.elasticsearch.index.gateway.IndexShardGatewayRecoveryException: [logstash-2015.07.02][1] failed to recover shard
at org.elasticsearch.index.gateway.local.LocalIndexShardGateway.recover(LocalIndexShardGateway.java:269)
at org.elasticsearch.index.gateway.IndexShardGatewayService$1.run(IndexShardGatewayService.java:132)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:745)
Caused by: org.elasticsearch.ElasticsearchIllegalArgumentException: No version type match [34]
at org.elasticsearch.index.VersionType.fromValue(VersionType.java:307)
at org.elasticsearch.index.translog.Translog$Create.readFrom(Translog.java:364)
at org.elasticsearch.index.translog.TranslogStreams.readTranslogOperation(TranslogStreams.java:52)
at org.elasticsearch.index.gateway.local.LocalIndexShardGateway.recover(LocalIndexShardGateway.java:241)
... 4 more
[2015-07-07 15:08:33,824][WARN ][cluster.action.shard ] [elasticsearch-logstash] [logstash-2015.07.02][1] sending failed shard for [logstash-2015.07.02][1], node[iA5GdFdmTciLfH9eg2XQ1Q], [P], s[INITIALIZING], indexUUID [mx__bjdYQbO9eViXOi9REQ], reason [Failed to start shard, message [IndexShardGatewayRecoveryException; ]]
[2015-07-07 15:08:33,824][WARN ][cluster.action.shard ] [elasticsearch-logstash] [logstash-2015.07.02][1] received shard failed for [logstash-2015.07.02][1], node[iA5GdFdmTciLfH9eg2XQ1Q], [P], s[INITIALIZING], indexUUID [mx__bjdYQbO9eViXOi9REQ], reason [Failed to start shard, message [IndexShardGatewayRecoveryException; ]]
^C[2015-07-07 15:08:34,295][WARN ][indices.cluster ] [elasticsearch-logstash] [logstash-2015.07.03][1] failed to start shard
org.elasticsearch.index.gateway.IndexShardGatewayRecoveryException: [logstash-2015.07.03][1] failed to recover shard
at org.elasticsearch.index.gateway.local.LocalIndexShardGateway.recover(LocalIndexShardGateway.java:269)
at org.elasticsearch.index.gateway.IndexShardGatewayService$1.run(IndexShardGatewayService.java:132)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:745)
Caused by: org.elasticsearch.ElasticsearchIllegalArgumentException: No version type match [122]
at org.elasticsearch.index.VersionType.fromValue(VersionType.java:307)
at org.elasticsearch.index.translog.Translog$Create.readFrom(Translog.java:364)
at org.elasticsearch.index.translog.TranslogStreams.readTranslogOperation(TranslogStreams.java:52)
at org.elasticsearch.index.gateway.local.LocalIndexShardGateway.recover(LocalIndexShardGateway.java:241)
... 4 more
[2015-07-07 15:08:34,314][WARN ][cluster.action.shard ] [elasticsearch-logstash] [logstash-2015.07.03][1] sending failed shard for [logstash-2015.07.03][1], node[iA5GdFdmTciLfH9eg2XQ1Q], [P], s[INITIALIZING], indexUUID [YICCx6eTSROaoLPVDuopuw], reason [Failed to start shard, message [IndexShardGatewayRecoveryException; ]]
[2015-07-07 15:08:34,314][WARN ][cluster.action.shard ] [elasticsearch-logstash] [logstash-2015.07.03][1] received shard failed for [logstash-2015.07.03][1], node[iA5GdFdmTciLfH9eg2XQ1Q], [P], s[INITIALIZING], indexUUID [YICCx6eTSROaoLPVDuopuw], reason [Failed to start shard, message [IndexShardGatewayRecoveryException; ]]
[2015-07-07 15:08:34,904][WARN ][indices.cluster ] [elasticsearch-logstash] [logstash-2015.07.03][3] failed to start shard
org.elasticsearch.index.gateway.IndexShardGatewayRecoveryException: [logstash-2015.07.03][3] failed to recover shard
at org.elasticsearch.index.gateway.local.LocalIndexShardGateway.recover(LocalIndexShardGateway.java:269)
at org.elasticsearch.index.gateway.IndexShardGatewayService$1.run(IndexShardGatewayService.java:132)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:745)
Caused by: org.elasticsearch.ElasticsearchIllegalArgumentException: No version type match [48]
at org.elasticsearch.index.VersionType.fromValue(VersionType.java:307)
at org.elasticsearch.index.translog.Translog$Create.readFrom(Translog.java:364)
at org.elasticsearch.index.translog.TranslogStreams.readTranslogOperation(TranslogStreams.java:52)
at org.elasticsearch.index.gateway.local.LocalIndexShardGateway.recover(LocalIndexShardGateway.java:241)
... 4 more
Something is corrupted or broken inside my elasticsearch container. How do you go about troubleshooting this Is there a way to get a fresh image of elasticsearch and redeploy it without losing all my previous logs
Subject: | file a bug, lets see what |
Author: | Andrew Kirch |
Posted: | 2015-07-20 12:00 |
file a bug, lets see what engineering says. http://jira.zenoss.com
Andrew Kirch
akirch@gvit.com
Need Zenoss support, consulting or custom development Look no further. Email or PM me!
Ready for Distributed Topology (collectors) for Zenoss 5 Coming May 1st from GoVanguard
Subject: | This has been fixed looks |
Author: | [Not Specified] |
Posted: | 2015-07-28 10:55 |
This has been fixed looks like it was a corrupted Elasticsearch_logstash had to curl -xdelete logstash indexes. Then restart Control Center. That fixes the problem :D
Subject: | awesome. If you can figure |
Author: | Andrew Kirch |
Posted: | 2015-07-29 09:03 |
awesome. If you can figure out why it was corrupted please file a bug so that we can repair it!
Andrew Kirch
akirch@gvit.com
Need Zenoss support, consulting or custom development Look no further. Email or PM me!
Ready for Distributed Topology (collectors) for Zenoss 5 Coming May 1st from GoVanguard
< |
Previous Zenoss 5 graph thresholds |
Next [zenoss 4.2.4] How to purge ALL archived events?? |
> |