TECHZEN Zenoss User Community ARCHIVE  

OpenTSDB stuck in "starting"

Subject: OpenTSDB stuck in "starting"
Author: Paul Giordano
Posted: 2018-05-08 16:51

Running Zenoss 6.1.2, upgraded from 5.2.3 - this was running for a bit over a week. 
So we hit a wall where the VMWare datastore ran out of space, and Zenoss shutdown. Went to How to Recover Control Center from Hardware Failure and followed the stuff there. Got much farther coming back up, but opentsdb goes into "starting" but in the Control Center it never shows the icon indicating startup. It seems to start/restart/restart, but never goes anywhere. I can't find a log that shows anything - in the serviced-isvcs_opentsdb container, the /var/log/opentsdb logs are empty. So,
1st: Where is there a more definitive log? Am I looking at the right place?
2nd: What else can I do to that might fix opentsdb?
Help appreciated. Sigh.

------------------------------
Paul Giordano
Senior Systems Engineer
Zethcon Corporation
------------------------------


Subject: RE: OpenTSDB stuck in "starting"
Author: Paul Giordano
Posted: 2018-05-09 08:33

OK, found the logs. opentsdb not starting, and the logs indicate that HBase is up. Running the startup script manually and the hang is in the /opt/opentsdb/src/create_table.sh script, hangs doing the "create 'tsdb-uid'" command. Will try that manually.
------------------------------
Paul Giordano
Senior Systems Engineer
Zethcon Corporation
------------------------------


Subject: RE: OpenTSDB stuck in "starting"
Author: Paul Giordano
Posted: 2018-05-09 09:24

Yep, that's the issue. HBase is hanging on the table creates.

------------------------------
Paul Giordano
Senior Systems Engineer
Zethcon Corporation
------------------------------


Subject: RE: OpenTSDB stuck in "starting"
Author: Paul Giordano
Posted: 2018-05-10 12:40

Well, it looks like the emergency shutdown totally hosed the system. No need to answer this. We're going to look into a different direction.

------------------------------
Paul Giordano
Senior Systems Engineer
Zethcon Corporation
------------------------------


Subject: RE: OpenTSDB stuck in "starting"
Author: rick parker
Posted: 2018-05-10 12:43

Hi Paul,
What I found after a lot of effort, to run everything on a single server a lot of memory configured for the 
host is required to start like 32GB. After running for some time I think usage drops

------------------------------
rick parker
systems engineer
SmartAction
el segundo CA
------------------------------


< Previous
Received wrong download links in the e-mail (Zenoss Core)
  Next
Core login screen just flashes bad user / password
>