---

Subject:	Verify all the daemons are
Author:	[Not Specified]
Posted:	2014-04-28 12:32

Verify all the daemons are running with 'zenoss status' as the zenoss user.

Also check the rabbit queue's: 'sudo rabbitmqctl -p /zenoss list_queues' (can compare to the list here: http://wiki.zenoss.org/Working_with_Queues)

does this command throw any errors: 'sudo service rabbitmq-server restart'

Did you change the hostname or IP of the server at any point

Hydruid

---

Subject:	Reply
Author:	[Not Specified]
Posted:	2014-04-28 12:47

Argh, I had that info, sorry.

zenoss@zenoss:~$ zenoss status
Daemon: zeneventserver program running; pid=21481
Daemon: zopectl program running; pid=21555
Daemon: zenhub program running; pid=21603
Daemon: zenjobs program running; pid=21644
Daemon: zeneventd program running; pid=21710
Daemon: zenping program running; pid=21752
Daemon: zensyslog program running; pid=21849
Daemon: zenstatus program running; pid=21831
Daemon: zenactiond program running; pid=21869
Daemon: zentrap program running; pid=21952
Daemon: zenmodeler program running; pid=21958
Daemon: zenperfsnmp program running; pid=21989
Daemon: zencommand program running; pid=22020
Daemon: zenprocess program running; pid=22051
Daemon: zenrrdcached program running; pid=22055
Daemon: zeneventlog program running; pid=22102
Daemon: zenwinperf program running; pid=22146
Daemon: zenwin program running; pid=22218
Daemon: zenjmx program running; pid=22252
=====================================================================

root@zenoss:~# rabbitmqctl -p /zenoss list_queues
Listing queues ...
celery 0
zenoss.celeryd.pidbox 0
zenoss.queues.zep.rawevents 0
zenoss.queues.zep.signal 0
...done.

So it looks like some are missing here
=====================================================================
rabbitmq-server restarts with "SUCCESS"
=====================================================================
Hostname/IP have not changed since install. Assigned statically in /etc/network/interfaces

Thanks for the help!

---

Subject:	Try rebuilding the queue's
Author:	[Not Specified]
Posted:	2014-04-28 12:51

Try rebuilding the queue's using this link: http://wiki.zenoss.org/Working_with_Queues

Also check the credentials in $ZENHOME/etc/global.conf to make sure you can use them to connect to MySQL. the default username/password is usually zenoss: mysql -u zenoss -p

Hydruid

---

Subject:	Obee kaybee, I followed those
Author:	[Not Specified]
Posted:	2014-04-28 14:04

Obee kaybee, I followed those steps. Now there are items in one of the queues, but there are still only 4 queues, not sure if that's a problem.

=======================
root@zenoss:~# rabbitmqctl -p /zenoss list_queues
Listing queues ...
celery 0
zenoss.celeryd.pidbox 0
zenoss.queues.zep.rawevents 111
zenoss.queues.zep.signal 0
...done.
========================

I forgot to mention (edited), I checked and I have the default MySQL user/pass, and they connected without error.

---

Subject:	I don't know if this is bad,
Author:	[Not Specified]
Posted:	2014-04-28 15:19

I don't know if this is bad, but this queue has been creeping up:
zenoss.queues.zep.rawevents 1080

---

Subject:	Hmmm
Author:	[Not Specified]
Posted:	2014-04-29 08:33

Pastebin your /etc/hosts file

Hydruid

---

Subject:	Last updated July 1, 2013
Author:	[Not Specified]
Posted:	2014-04-29 08:55

Last updated July 1, 2013
----
verus@zenoss:~$ more /etc/hosts
127.0.0.1 localhost
127.0.1.1 zenoss.vt.loc zenoss

# The following lines are desirable for IPv6 capable hosts
::1 ip6-localhost ip6-loopback
fe00::0 ip6-localnet
ff00::0 ip6-mcastprefix
ff02::1 ip6-allnodes
ff02::2 ip6-allrouters

---

Subject:	Try This
Author:	[Not Specified]
Posted:	2014-04-29 09:12

# Give these commands a try to see what happens:
sudo service zenoss stop
sudo service rabbitmq-server stop
sudo service mysql stop

# Add the following to /etc/hosts (remember to tab between columns)
ip-of-zenoss-server hostname
ip-0f-zenoss-server hostname.fqdn
#Examples
172.16.5.5 AwesomeZenoss
172.16.5.5 AwesomeZenoss.Android.local

# Start everything back up with a reboot
sudo reboot now

Hydruid

---

Subject:	Same results.
Author:	[Not Specified]
Posted:	2014-04-29 09:36

Same results.
verus@zenoss:~$ more /etc/hosts
127.0.0.1 localhost
#127.0.1.1 zenoss.vt.loc zenoss
192.168.203.18 zenoss.verustech.com
192.168.203.18 zenoss

(If I haven't said it enough before, I REALLY appreciate your help!)

---

Subject:	Hmm
Author:	[Not Specified]
Posted:	2014-04-29 09:48

You're very welcome......this issue is very strange lol.

We may have to reinstall zenoss over the top of what is there now, lets try a few more things:

# Check ZODB Integrity
sudo su zenoss
zenoss stop
zenchkrels -r -x1
zenoss start

# If the above zenchkrels reported no issues, lets try upgrading rabbit
sudo su zenoss
zenoss stop
exit
sudo service rabbitmq-server stop
sudo dpkg --list | grep rabbit #shows which version is installed
sudo apt-get remove rabbitmq-server
sudo dpkg --list | grep rabbit #verify that it's been removed
# might have to run: sudo apt-get purge rabbitmq-server if it's still listed after a remove
cd ~
wget http://www.rabbitmq.com/releases/rabbitmq-server/v3.3.1/rabbitmq-server_...
sudo dpkg -i rabbitmq-server_3.3.1-1_all.deb
sudo service rabbitmq-server restart
# Recreate the queue's from this link: http://wiki.zenoss.org/Working_with_Queues
sudo su zenoss
zenoss restart
exit
sudo rabbitmqctl -p /zenoss list_queues

That is a lot of commands lol but hopefully it will take care of it.

Hydruid

---

Subject:	Same
Author:	[Not Specified]
Posted:	2014-04-29 10:04

OK, zenchkrels -r -x1 returns nothing.
I upgraded rabbit to 3.3.1
Recreated queues, restarted zenoss.
Same error. list_queues gives:
root@zenoss:~# rabbitmqctl -p /zenoss list_queues
Listing queues ...
celery 0
zenoss.celeryd.pidbox 0
zenoss.queues.zep.rawevents 43
zenoss.queues.zep.signal 0
...done.

---

Subject:	I feel like it's something
Author:	[Not Specified]
Posted:	2014-04-29 10:05

I feel like it's something simple, but I just can't find it. :)

---

Subject:	Try starting the zenhub
Author:	[Not Specified]
Posted:	2014-04-29 10:09

Try starting the zenhub daemon with a higher verbosity to see if we can get more info on what is wrong:

sudo su zenoss
zenhub run -v10

Pastebin the output!

Hydruid

---

Subject:	Address already in use
Author:	[Not Specified]
Posted:	2014-04-29 10:14

http://pastebin.com/DjB13zvS

---

Subject:	Try:
Author:	[Not Specified]
Posted:	2014-04-29 10:23

Try:
sudo su zenoss
zenhub stop
zenhub run -v10

Hydruid

---

Subject:	I ran it and let it run for a
Author:	[Not Specified]
Posted:	2014-04-29 10:32

I ran it and let it run for a bit. Tried logging in also in case that would generate something.

http://pastebin.com/GP4jhFMp

---

Subject:	Looks like zenhub is
Author:	[Not Specified]
Posted:	2014-04-29 10:40

Looks like zenhub is functioning properly! Are the raw events or any of the rabbit queue's still consistently growing

Also have you tried access zenoss from a different PC to see what happens when you try to login

Hydruid

---

Subject:	rawevents seems to have
Author:	[Not Specified]
Posted:	2014-04-29 11:04

rawevents seems to have stopped at 638.
I've tried from Mac/Chrome, Windows/Chrome, Windows/IE both inside the LAN and outside.

---

Subject:	MySQL
Author:	[Not Specified]
Posted:	2014-04-29 11:08

Try exporting each of the MySQL databases to see if that gives you any errors:

mysqldump -u zenoss zenoss_zep -p > zenoss_zep.sql
mysqldump -u zenoss zodb -p > zodb.sql
mysqldump -u zenoss zodb_session -p > zodb_session.sql

how many devices and/or customizations did you put into this install What happened around the time it stopped working At this point it might be faster to rebuild and to ensure that you do a regular backup ;)

Hydruid

---

Subject:	Finally an error. :)
Author:	[Not Specified]
Posted:	2014-04-29 11:24

Finally an error. :)

root@zenoss:~/mysql# mysqldump -u zenoss zenoss_zep -p > zenoss_zep.sql
Enter password:
mysqldump: Got error: 1696: Failed to read from the .par file when using LOCK TABLES

---

Subject:	Also, we only have maybe 20
Author:	[Not Specified]
Posted:	2014-04-29 11:27

Also, we only have maybe 20 devices in there. I really have no idea when it stopped. We checked it one day, and found it broken, and it got put in the "We'll look at it later" pile. :)

I can definitely rebuild. But, if it's been capturing data all this time it would be nice to have. If not, no biggie.

BTW, the other two mysqldump command worked without error.

---

Subject:	About time we got an error ;)
Author:	[Not Specified]
Posted:	2014-04-29 11:34

About time we got an error ;)

Stop zenoss and then try this: sudo mysqlcheck --repair --all-databases

Hydruid

---

Subject:	Several errors
Author:	[Not Specified]
Posted:	2014-04-29 11:54

Several errors

http://pastebin.com/fNBTbxaX

---

Subject:	Ahh looks like zenoss_zep is
Author:	[Not Specified]
Posted:	2014-04-29 12:32

Ahh looks like zenoss_zep is corrupt

#Stop Zenoss
sudo su zenoss
zenoss stop

#Jump into mysql and drop that database
mysql -u zenoss -p
drop database zenoss_zep;
FLUSH PRIVILEGES;
create database zenoss_zep;
exit

#Import the original db
sudo su zenoss
cd ~
mysql -uzenoss -pzenoss zenoss_zep < zenoss_zep.sql

Let me know if you don't have the zenoss_zep.sql at /home/zenoss

Hydruid

---

Subject:	Nope.
Author:	[Not Specified]
Posted:	2014-04-29 12:39

Nope.
These are the only sql files I have in /home/zenoss

zenoss@zenoss:~$ find . -name *.sql
./zenoss-inst/build/zenoss.extjs-4.1.0-r57826/build/lib/zenoss/extjs/src/examples/simple-tasks/db/schema.sql
./zenoss-inst/build/zenoss.extjs-4.1.0-r57826/build/lib/zenoss/extjs/src/examples/grid-filtering/grid-demo.sql
./zenoss-inst/build/zenoss.extjs-4.1.0-r57826/zenoss/extjs/src/examples/simple-tasks/db/schema.sql
./zenoss-inst/build/zenoss.extjs-4.1.0-r57826/zenoss/extjs/src/examples/grid-filtering/grid-demo.sql
./zenoss-inst/build/java/zep/core/src/main/sql/mysql/005.sql
./zenoss-inst/build/java/zep/core/src/main/sql/mysql/002.sql
./zenoss-inst/build/java/zep/core/src/main/sql/mysql/001.sql
./zenoss-inst/build/java/zep/core/src/main/sql/mysql/003.sql
./zenoss-inst/build/java/zep/core/src/main/sql/mysql/004.sql
./zenoss-inst/build/java/zep/core/src/main/sql/postgresql/002.sql
./zenoss-inst/build/java/zep/core/src/main/sql/postgresql/001.sql
./zenoss-inst/build/Twisted-11.0.0/doc/core/examples/row_schema.sql
./zenoss-inst/build/RelStorage-1.5.1/notes/migrate.sql

---

Subject:	Link
Author:	[Not Specified]
Posted:	2014-04-29 12:50

sudo su zenoss
cd ~
wget http://master.dl.sourceforge.net/project/zenossforubuntu/misc/zenoss_zep...

Then follow my commands from my previous comment

Hydruid

---

Subject:	Argh.
Author:	[Not Specified]
Posted:	2014-04-29 13:22

Argh.

mysql> drop database zenoss_zep;
ERROR 1051 (42S02): Unknown table 'event_archive,event_time'

---

Subject:	Do some research to see if
Author:	[Not Specified]
Posted:	2014-04-29 13:39

Do some research to see if there is a way you can force it to be dumped, alternatively you could install phpmyadmin and use the GUI to repair or drop it.

Hydruid

---

Subject:	a fix?
Author:	[Not Specified]
Posted:	2014-07-30 17:07

I had the same problem. Here's what I did:

- stop mysql - 'mysqld stop'
- go to /var/lib/mysql and execute 'rm -rf zenoss_zep'
- start mysql - 'mysqld start'
- zenoss must be running for the next step
- as zenoss user, execute 'zeneventserver-create-db --dbtype=mysql --dbadminuser=root --dbadminpass='
- restart zenoss - 'zenoss restart'
- check rabbitmq - 'sudo rabbitmqctl -p /zenoss list_queues'

you should see all the queues now and hopefully Zenoss works . This fixed the same problem for me. No promises on data loss or anything, as I have no idea what the affect of dropping the zenoss_zep database is. However, Zenoss is now working again.

URLs I used:
http://stackoverflow.com/questions/10538908/schr%C3%B6dingers-mysql-tabl...
http://zenpackers.readthedocs.org/en/latest/databases.html

Best regards,
Jeff