---

Subject:	What method did you use to
Author:	[Not Specified]
Posted:	2014-03-07 10:31

What method did you use to install Zenoss on Squeeze In my testing, I was unable to get it working without basically upgrading the system to mostly Wheezy/Testing packages.

I had similar issues as you have, daemons not running and failing. Check your /var/log/syslog and I'm sure that you will find python segfaults.

Hydruid

---

Subject:	I used this method
Author:	[Not Specified]
Posted:	2014-03-10 12:20

http://hydruid-blog.com/p=485

It went really well tbh, fresh install. It went south when I actually switched the IP from dynamic to static. if that keeps up i will just wipe it out, re-install squeeze, stick it to static IP and THEN do the script install. but if someone could avoid me this, i would be thankful

Matt

---

Subject:	That is my script :)
Author:	[Not Specified]
Posted:	2014-03-10 13:59

That is my script :)

Make sure that your /etc/hosts is correct. Also do the following commands to see what happens:
/etc/init.d/zenoss stop
/etc/init.d/rabbitmq-server restart
rabbitmqctl -p /zenoss list_queues

Then pastebin the results :)

Hydruid

---

Subject:	On it!
Author:	[Not Specified]
Posted:	2014-03-10 15:51

And thanks for that script, grabbing all of that and pastebinning it to you in a moment.

TMR

---

Subject:	There you go!
Author:	[Not Specified]
Posted:	2014-03-10 16:02

Copypasta here: http://pastebin.com/ga3hxxKD

The really weird thing is that it all starts working magically when manually restarting zenoss deamon. I'm just at lost at why it doesn't bring everything up after a reboot.

TMR

---

Subject:	Proof
Author:	[Not Specified]
Posted:	2014-03-10 16:09

see below at the bottom of that pastebin: http://pastebin.com/bxi0JX4Z

---

Subject:	Rabbit
Author:	[Not Specified]
Posted:	2014-03-11 10:41

Rabbit is your culprit, specifically this output:
root@sentinel:~# rabbitmqctl -p /zenoss list_queues
Listing queues ...
celery 0
...done.

You can see at this link: http://wiki.zenoss.org/Working_with_Queues that you should have the zenoss queue's listed.

However it may also be a bug from my script that I fixed a few months ago. Check your /etc/init.d/zenoss to verify it contains the following lines:
#Custom Ubuntu Variables
export ZENHOME=/usr/local/zenoss
export RRDCACHED=/usr/local/zenoss/bin/rrdcached
export PATH="/usr/local/zenoss/bin:$PATH"

If those lines are there, then it's time to work on rabbit. Go to the rabbit link above and follow the guide "to recreate all Zenoss exchanges and queues." That should get you fixed up!

Rabbit is super picky about host names and IP address changes.

Hydruid

---

Subject:	I think you're right!
Author:	[Not Specified]
Posted:	2014-03-11 11:56

Okay i checked the INIT.D script and it's definitely not that, the 3 lines are right there as stated. It totally makes sense about Rabbit. However, the strange thing is that when I restarted ZenOSS, the queues are now listed in Rabbit:

root@sentinel:~# rabbitmqctl -p /zenoss list_queues
Listing queues ...
celery 0
sentinel.celeryd.pidbox 0
zenoss.queues.zep.heartbeats 0
zenoss.queues.zep.migrated.archive 0
zenoss.queues.zep.migrated.summary 0
zenoss.queues.zep.modelchange 0
zenoss.queues.zep.rawevents 0
zenoss.queues.zep.signal 0
zenoss.queues.zep.zenevents 0
...done.

So do I really need to re-create them Or should I check Rabbit's config to see as how it doesn't recreate them itself on boot

TMR

---

Subject:	That confirms that rabbit is
Author:	[Not Specified]
Posted:	2014-03-11 12:16

That confirms that rabbit is your issue. On a reboot the zenoss queue's don't exist, after you manually restart zenoss, they exist.....very strange indeed.

Make a backup and try the following:
make another backup ;)
sudo su zenoss
zenoss stop
exit
sudo apt-get remove rabbitmq-server
sudo apt-get purge rabbitmq-server
dpkg --list | grep rabbit
wget -N http://www.rabbitmq.com/releases/rabbitmq-server/v3.2.3/rabbitmq-server_...
sudo dpkg -i rabbitmq-server_3.2.3-1_all.deb
sudo chown -R zenoss:zenoss /usr/local/zenoss
sudo rabbitmqctl add_user zenoss zenoss
sudo rabbitmqctl add_vhost /zenoss
sudo rabbitmqctl set_permissions -p /zenoss zenoss '.*' '.*' '.*'os
sudo su zenoss
zenoss start

If everything starts up correctly, reboot to see what happens!

Hydruid

---

Subject:	Followup
Author:	[Not Specified]
Posted:	2014-03-11 12:27

I followed the exact instructions here and recreated the queues: http://wiki.zenoss.org/Working_with_Queues

However the last step gave me this:

root@sentinel:~# rabbitmqctl -p $VHOST list_queues # check if all queues exist
Listing queues ...
...done.

But upon restarting Zenoss, they seem to finally re-appear:

root@sentinel:~# rabbitmqctl -p /zenoss list_queues
Listing queues ...
celery 0
sentinel.celeryd.pidbox 0
zenoss.queues.zep.heartbeats 0
zenoss.queues.zep.migrated.archive 0
zenoss.queues.zep.migrated.summary 0
zenoss.queues.zep.modelchange 0
zenoss.queues.zep.rawevents 0
zenoss.queues.zep.signal 0
zenoss.queues.zep.zenevents 0
...done.

Attempting a reboot to see if it fixed the issue.

TMR

---

Subject:	Unfortunately...
Author:	[Not Specified]
Posted:	2014-03-11 14:23

After reconstructing the message queues, and after a fresh reboot, situation is still the same, sadly

TMR

---

Subject:	Did you try what I suggested
Author:	[Not Specified]
Posted:	2014-03-11 14:40

Did you try what I suggested earlier..........
============================================================================

That confirms that rabbit is your issue. On a reboot the zenoss queue's don't exist, after you manually restart zenoss, they exist.....very strange indeed.

Make a backup and try the following:
make another backup ;)
sudo su zenoss
zenoss stop
exit
sudo apt-get remove rabbitmq-server
sudo apt-get purge rabbitmq-server
dpkg --list | grep rabbit
wget -N http://www.rabbitmq.com/releases/rabbitmq-server/v3.2.3/rabbitmq-server_...
sudo dpkg -i rabbitmq-server_3.2.3-1_all.deb
sudo chown -R zenoss:zenoss /usr/local/zenoss
sudo rabbitmqctl add_user zenoss zenoss
sudo rabbitmqctl add_vhost /zenoss
sudo rabbitmqctl set_permissions -p /zenoss zenoss '.*' '.*' '.*'os
sudo su zenoss
zenoss start

If everything starts up correctly, reboot to see what happens!

Hydruid

---

Subject:	Oh!
Author:	[Not Specified]
Posted:	2014-03-11 14:46

i'm sorry, unsure why but i completely missed that part! Let me try that and i'll get right back atcha!

thanks again for all the help, it's mucho appreciated

TMR

---

Subject:	You're welcome, I want to get
Author:	[Not Specified]
Posted:	2014-03-11 14:47

You're welcome, I want to get it working! I understand what it's like to use the non-preferred OS for zenoss and other things :)

Hydruid

---

Subject:	Darn it, forum!
Author:	[Not Specified]
Posted:	2014-03-11 14:48

The link for the rabbit server compilable seems to be truncated in the forum.. looking for it

TMR

---

Subject:	Right package
Author:	[Not Specified]
Posted:	2014-03-11 14:51

I'm gonna assume it's this one, right rabbitmq-server_3.2.3-1_all.deb

As for ZenOSS "non prefered OS", Debian certainly is my first choice, way before any RPM based distro!

TMR

---

Subject:	I'm dumb
Author:	[Not Specified]
Posted:	2014-03-11 14:56

Forget that I asked for the .deb filename, I just figured one line after that it was right there (facepalms).

It seems to be restarting correctly, now going to try a reboot and see how that's going on.

TMR

---

Subject:	I prefer Ubuntu but it's
Author:	[Not Specified]
Posted:	2014-03-11 14:58

I prefer Ubuntu but it's basically Debian with a few minor changes and newer packages!

Hydruid

---

Subject:	Ugh
Author:	[Not Specified]
Posted:	2014-03-11 15:01

Sadly, no dice. It's like it's mocking me. Same exact thing, same processes not running. Did I mention the mysql error that it throws in the logs at startup

TMR

---

Subject:	Yes I noticed that. I think
Author:	[Not Specified]
Posted:	2014-03-11 15:12

Yes I noticed that. I think we've confirmed that is the issue. Modify your /etc/init.d/zenoss and below the Custom Ubuntu Variables section add in:

sleep 60

I just tested it on a VM and it seemed to work fine. The bad news is that everytime you do a /etc/init.d/zenoss restart it will wait 60 seconds before it does anything LOL.

Hydruid

---

Subject:	Tried that already
Author:	[Not Specified]
Posted:	2014-03-11 15:15

I did try that with a 30 seconds delay and it failed previously, but going to try it with 60. TBH if it works, i won't even care about the delay haha

TMR

---

Subject:	If that doesn't work, the
Author:	[Not Specified]
Posted:	2014-03-11 15:20

If that doesn't work, the next step is to troubleshoot your MySQL issue. Here is an old but good link to get you started with that: http://ubuntuforums.org/showthread.phpt=1676913

Hydruid

---

Subject:	Thing is
Author:	[Not Specified]
Posted:	2014-03-11 15:23

Mysql isn't perticularily slow, it just seems like some kind of racing condition. MySQL needs to be properly and finally started BEFORE ZenOSS even attempts to start. Checked the INIT order and i'm very poor at it but it seems that ZenOSS attempts starting up even before Mysql startup is initiated.

And this said, failure again, the processes are not started properly with or without the 60 seconds delay.

Sadness.

TMR

---

Subject:	This is what you want to do:
Author:	[Not Specified]
Posted:	2014-03-11 15:25

This is what you want to do: http://serverfault.com/questions/340408/debian-squeeze-startup-order-of-...

Hydruid

---

Subject:	Disabled mysql crashed table checks and looking into init
Author:	[Not Specified]
Posted:	2014-03-11 15:27

This hurts my fragile little mind. I was okay with RC inits but this lsb thing is weirding me out.

---

Subject:	ooooo
Author:	[Not Specified]
Posted:	2014-03-11 15:33

i think i found something. there's already an override for zenoss! it just lacks the proper required-start header stating it depends on mysql.. fiddling with it

TMR

---

Subject:	Preliminary logs
Author:	[Not Specified]
Posted:	2014-03-11 15:56

I get this in both the logs for zensyslog and zentrap. Clearing them and rebooting to get a better view.

2014-03-11 16:42:15,510 WARNING zen.zensyslog: No service named 'EventService': ZenHub may be disconnected
2014-03-11 16:42:15,510 ERROR zen.maintenance: Maintenance failed. Message from hub: ZenHub is down

However, strangely, zenhub sure is running and is connectable.

---

Subject:	Weird
Author:	[Not Specified]
Posted:	2014-03-11 16:08

Just cleaned the logs and did a reboot and zensyslog and zentrap are still not running, but most bizarre, absolutely nothing negative in the logs.

http://pastebin.com/w0GqUNiY

---

Subject:	Weirder
Author:	[Not Specified]
Posted:	2014-03-11 16:28

Those two processes won't restart after a zenoss restart

TMR

---

Subject:	Are those even necessary?
Author:	[Not Specified]
Posted:	2014-03-11 16:34

Considering i'm probably not going to get into coding zenpacks anytime soon, are those even essential Reading this, makes me think they aren't: http://wiki.zenoss.org/Zenoss_Processes

---

Subject:	Okay!
Author:	[Not Specified]
Posted:	2014-03-11 16:48

The situation changed a bit but imoho, it's already a lot better. Now I only have 2 services not running:
zensyslog and zentrap

The rest seems to be running A-okay! So i'll check the logs for those two and see what is going on. I suspect another racing condition that I will probably be able to sort the same way.

To recap, what I did is simply edit /etc/insserv/overrides/zenoss like this:

# Required-Start: mysql
# Required-Stop: mysql

Then ran: insserv -d -vv

Enquiring for the two processes that are failing.

TMR

---

Subject:	No they are not essential!!!!
Author:	[Not Specified]
Posted:	2014-03-12 15:10

No they are not essential!!!!! Please share any config file changes you made so that I can import them into my script!

Hydruid

---

Subject:	Update
Author:	[Not Specified]
Posted:	2014-03-13 10:22

I had a similar issue on Debian 7.4 with another user and have updated the init script and override to resolve it! Pastebin here: http://pastebin.com/QqFWwsxM

Hydruid

---

Subject:	Followup
Author:	[Not Specified]
Posted:	2014-03-13 11:31

Hey, I want to thank you for your help into cornering that issue. I will of course try your latest pastebin to see if it fixes the issue of the 2 missing processes, but here's what I did exactly to solve the issue:

root@sentinel:~# cat /etc/insserv/overrides/zenoss
### BEGIN INIT INFO
# Provides: zenoss-stack
# Required-Start: mysql
# Required-Stop: mysql
# Should-Start:
# Should-Stop:
# Default-Start: 2 3 4 5
# Default-Stop: 0 1 6
# Short-Description: Start/stop Zenoss-stack
# Description: Start/stop Zenoss-stack
### END INIT INFO

So basically, I just edited the zenoss override to add up mysql as a required-start and required-stop process. That pretty much fixed the bootup issue, excluding the two missing processes. I think your script should also include mysql as a pre-requisite to startup zenoss in the override. I will mess up a bit with it and report back atcha!

Thanks again dude, most appreciated

TMR

---

Subject:	You're welcome but honestly
Author:	[Not Specified]
Posted:	2014-03-13 11:46

You're welcome but honestly THANK YOU for the feedback, it's issues like this that make things better!

Hydruid

---

Subject:	Again, followup
Author:	[Not Specified]
Posted:	2014-03-14 10:52

Greetings Hydruid,

So, finally i made the same modifications that you did to the zenoss override. It didn't make zensyslog and zentrap work any better, sadly, but I will keep the modifications this way. I would suggest that you add up mysql dependency on your override because I'm pretty sure it will occur to other squeeze users. I may open another thread with the zentrap and zensyslog issue because i'm just too damn curious about why those doesn't startup properly.

Thanks again for all your help and if i can help you test anything about your scripts, just let me know!

TMR

---

Subject:	Have you tried...
Author:	[Not Specified]
Posted:	2014-03-17 09:36

Guys, I see a ton of troubleshooting in here but nothing to indicate that you tried simply connecting to your MySQL server from the command line. Review the username/password settings for zodb and zep in $ZENHOME/etc/global.conf and attempt using those usernames/passwords, and the port number specified in there to connect to mysql from the commandline via the mysql command. If you can't get connected with those settings then there's your problem.

---

Subject:	Rmatte, it works just not
Author:	[Not Specified]
Posted:	2014-03-18 08:02

Rmatte, it works just not after a reboot (manually doing a "zenoss restart" works with no issues).

Hydruid