TECHZEN Zenoss User Community ARCHIVE  

Zenoss 4.2.5 Database issue

Subject: Zenoss 4.2.5 Database issue
Author: Greg Sullivan
Posted: 2021-01-21 08:54

I'm having a SQL issue with Zenoss Core 4.2.5 on Centos likely as the result of a disk full and a few power events.  It's a pretty standard install and monitors about 300 devices.  It runs really well without persistent issues.

****

There is a serious problem with the MySQL event database

Below is the detailed information provided:

(145, "Table './zodb/new_oid' is marked as crashed and should be repaired")

****

The installation is mostly functional but I get an error page intermittently on new page views and can't add/remove/modify devices.  ("unable to connect to server" prompt).

I'm not super interested in preserving event or performance data.  Can I just drop and add a database? which one?

My SQL skills are modest, but I did get through POSKey error (dropping and adding zodb_session).



------------------------------
Greg Sullivan
Appoquinimink School District
Odessa DE
3024493832
------------------------------



Subject: RE: Zenoss 4.2.5 Database issue
Author: Pheripheral Pheripheral
Posted: 2021-01-22 09:51

​Hi,

I'm pretty sure i've seen this error in amougst the various errors i've seen with corrupted event databases.

I've got a pair of scripts that totally removes then rebuilds the event database if you want to give them a try...

Main script, save this with whatever name you fancy in a top level dir called /fixme/ (or edit script to work in whatever location you want)
#!/bin/sh
set -x #echo on

# STOP ZENOSS
sudo -i -u zenoss zenoss stop

# TRY AND REMOVE ZENOSS_ZEP
# MAY WELL FAIL AS THIS SCRIPT IS TO BE USED WHEN ZENOSS_ZEP CORRUPTED
mysql -u root << DROPZEPSQL
show databases;
drop database zenoss_zep;
\! echo "zenoss_zep drop run"
show databases;
DROPZEPSQL

# STOP MYSQL
service mysql stop

# EMPTY AND REMOVE ZENOSS_ZEP DIRECTORY ENTIRELY
rm /var/lib/mysql/zenoss_zep/*.*
rmdir /var/lib/mysql/zenoss_zep/

# START MYSQL
service mysql start

# DROP ZENOSS_ZEP IF IT STILL EXISTS
mysql -u root << DROPZEPSQL
show databases;
drop database zenoss_zep;
\! echo "zenoss_zep drop run"
show databases;
DROPZEPSQL

# FORCE CREATE ZENOSS_ZEP - Often fails after full remove
sudo -i -u zenoss /fixme/create_zepdb.sh

# STOP MYSQL
service mysql stop

# EMPTY POSSIBLE PARTIALLY CREATED ZENOSS_ZEP DIRECTORY
rm /var/lib/mysql/zenoss_zep/*.*

# START MYSQL
service mysql start

# FORCE CREATE ZENOSS_ZEP
sudo -i -u zenoss /fixme/create_zepdb.sh

# START ZENOSS
sudo -i -u zenoss zenoss start​


Secondary script (called via sudo as zenoss user from main script) - save this as create_zepdb.sh in the same /fixme/ directory

#/bin/sh
set -x #echo on

zengc=/opt/zenoss/bin/zenglobalconf

# read configuration from global.conf
dbtype=$(${zengc} -p zodb-db-type)
host=$(${zengc} -p zodb-host)
port=$(${zengc} -p zodb-port)
user=$(${zengc} -p zodb-user)
userpass=$(${zengc} -p zodb-password)
admin=$(${zengc} -p zodb-admin-user)
adminpass=$(${zengc} -p zodb-admin-password)
dbname=$(${zengc} -p zodb-db)

# Drop and recreate the ZEP event database
zeneventserver-create-db --dbtype $dbtype --dbhost $host --dbport $port --dbadminuser $admin --dbadminpass "${adminpass}" --dbuser $user --dbpass "${userpass}" --force
​

Run the main script as root and it should drop the event database, remove the directories and recreate.
It does the recreate twice as during tests I saw it fail under some circumstances on the first go. This one seems to work everytime.

Hope this helps.

Cheers

------------------------------
Pheripheral Pheripheral
------------------------------


Subject: RE: Zenoss 4.2.5 Database issue
Author: Greg Sullivan
Posted: 2021-01-22 13:07

That worked ... I attached the output.  

Those error pages went away I believe, but I'm still not able to modify or delete devices.  I still get the small black window dead center on the screen that says "server exception" ... unable to connect to server.  

The UI responds to navigation but not functions that modify devices (the radio buttons to turn services off/on in advanced >> daemons work fine though).

Can't change production state, IP address etc.

Could this just be missing host info?


------------------------------
Greg Sullivan
Appoquinimink School District
Odessa DE
3024493832
------------------------------


Subject: RE: Zenoss 4.2.5 Database issue
Author: Greg Sullivan
Posted: 2021-02-03 14:27

Yeah that script did complete but it didn't fix the problem.  
**********************************

[zenoss@appomonitor01 ~]$ zodbscan

Traceback (most recent call last):

  File "/opt/zenoss/bin/zodbscan", line 8, in <module>

    load_entry_point('zenoss.toolbox==2.3.1', 'console_scripts', 'zodbscan')()

  File "/opt/zenoss/lib/python/pkg_resources.py", line 318, in load_entry_point

    return get_distribution(dist).load_entry_point(group, name)

  File "/opt/zenoss/lib/python/pkg_resources.py", line 2221, in load_entry_point

    return ep.load()

  File "/opt/zenoss/lib/python/pkg_resources.py", line 1954, in load

    entry = __import__(self.module_name, globals(),globals(), ['__name__'])

  File "/opt/zenoss/lib/python2.7/site-packages/zenoss.toolbox-2.3.1-py2.7.egg/zenoss/toolbox/zodbscan.py", line 28, in <module>

    from zodbpickle.pickle import Unpickler as UnpicklerBase

ImportError: No module named zodbpickle.pickle

******************************************************
I downloaed the zenoss toolbox and zodbscan gives an error.  zenrelationscan and findposkeyerror are having issues too with fixing.

I took it from the top... dropped and created zodb_session and went back to the toolbox but still no luck.  I'm attaching the logs from this 


------------------------------
Greg Sullivan
Appoquinimink School District
Odessa DE
3024493832
------------------------------


< Previous
HTML Links within events, presented on the events screen
  Next
Migrating to Zenoss 4.2.5 ......
>