![]() |
![]() |
Subject: | Rabbitmq Blocking Zenoss Connections |
Author: | Benjamin Dronen |
Posted: | 2015-01-07 14:27 |
In my Zenoss dashboard, the Daemon Processes down portlet shows these daemons as being down: zenactiond, zencommand, zeneventd, zenhub, zenmodeler, zenperfsnmp, zenping, zenprocess, zenpython, zenstatus, zensyslog, zentrap, and zenwinsrvsnmp
This is similar to the problem described by QnA here: http://www.zenoss.org/forum/171
His solution ended up being that RabbitMQ was blocking his connections due to a low memory threshold described here: http://community.zenoss.org/thread/18511start=0&tstart=0
I checked rabbitmq connections using rabbitmqctl list_connections, and it gave me this output:
Listing connections ...
zenoss 127.0.0.1 43886 blocked
zenoss 127.0.0.1 43870 blocked
zenoss 127.0.0.1 52474 blocking
zenoss 127.0.0.1 43877 blocking
zenoss 127.0.0.1 52685 blocked
zenoss 127.0.0.1 52687 blocked
zenoss 127.0.0.1 43895 blocking
zenoss 127.0.0.1 43902 blocking
zenoss 127.0.0.1 43876 blocked
zenoss 127.0.0.1 52686 blocked
zenoss 127.0.0.1 43896 blocking
zenoss 127.0.0.1 43903 blocked
...done.
I increased my memory threshold to 0.7 and restarted both rabbitmq, zenoss, and the server itself to no avail.
Any suggestions
Zenoss 4.2.5
Centos 5
Subject: | Disregard |
Author: | Benjamin Dronen |
Posted: | 2015-01-07 14:46 |
It seems as though the daemons are running properly now and rabbitmq is no longer blocking any connections.
Subject: | yeah rabbit often takes a few |
Author: | Andrew Kirch |
Posted: | 2015-01-08 10:20 |
yeah rabbit often takes a few to resolve when it breaks, reply if there are further issues.
Andrew Kirch
akirch@gvit.com
Need Zenoss support, consulting or custom development Look no further. Email or PM me!
Ready for Distributed Topology (collectors) for Zenoss 5 Coming May 1st from GoVanguard
Subject: | Started Blocking Connections |
Author: | Benjamin Dronen |
Posted: | 2015-01-14 15:11 |
Not sure why this has happened again, but it seems to be blocking all the daemons once again.
Running list-connections gives me this again:
Listing connections ...
zenoss 127.0.0.1 43146 blocking
zenoss 127.0.0.1 43144 blocking
zenoss 127.0.0.1 43138 blocked
zenoss 127.0.0.1 43145 blocked
zenoss 127.0.0.1 43137 blocking
zenoss 127.0.0.1 43147 blocked
zenoss 127.0.0.1 43143 blocking
zenoss 127.0.0.1 43134 blocked
...done.
Any further suggestions
Subject: | RabbitMQ can block |
Author: | Jan Garaj |
Posted: | 2015-01-14 17:00 |
RabbitMQ can block connections because:
1.) Memory
https://www.rabbitmq.com/memory.html
2.) Disk
https://www.rabbitmq.com/disk-alarms.html
So check your RabbitMQ settings and current values for memory + disk also.
Devops Monitoring Expert advice:
Dockerize/automate/monitor all the things.
DevOps stack:
Docker / Kubernetes / Mesos / Zabbix / Zenoss / Grafana / Puppet / Ansible / Vagrant / Terraform /
Elasticsearch
Subject: | you may also want to install |
Author: | Andrew Kirch |
Posted: | 2015-01-19 08:24 |
you may also want to install the RabbitMQ ZenPack
Andrew Kirch
akirch@gvit.com
Need Zenoss support, consulting or custom development Look no further. Email or PM me!
Ready for Distributed Topology (collectors) for Zenoss 5 Coming May 1st from GoVanguard
Subject: | same issue here |
Author: | [Not Specified] |
Posted: | 2015-01-26 22:54 |
I face the same issue here as well. The rabbitmq connections are blocked even I have reset them.
Any idea how to solve this issue
Currently, all the events are not coming in and the GUI prompts unable to connect to the server message.
How to change the rabbitmq.config I can't find this find in the whole server.
Zenoss 4.2.5
RH Linux 6.5
I am not using auto-deploy for installation, but manual installation. The system has been running for 2 months without problem before the issue happen.
Subject: | It depends on your config, |
Author: | Jan Garaj |
Posted: | 2015-01-27 00:57 |
It depends on your config, but generally it'll be /etc/rabbitmq/rabbitmq.config (If it doesn't exist, then default values are used). Use example config https://github.com/rabbitmq/rabbitmq-server/blob/master/docs/rabbitmq.co... and uncomment what do you need ({disk_free_limit, {mem_relative, 1.0}}, ....) save and restart rabbitmq-server. Check rabbitmq logs if new settings have been applied.
Devops Monitoring Expert advice:
Dockerize/automate/monitor all the things.
DevOps stack:
Docker / Kubernetes / Mesos / Zabbix / Zenoss / Grafana / Puppet / Ansible / Vagrant / Terraform /
Elasticsearch
Subject: | Is there any possible |
Author: | [Not Specified] |
Posted: | 2015-01-27 06:23 |
Is there any possible solution for the blocked rabbitmq connections
Besides, is the error message below a symptom of this problem I get this error message.
Type:
Value: NULL object passed to Py_BuildValue
Traceback (most recent call last):
File "/opt/zenoss/lib/python/ZPublisher/Publish.py", line 134, in
publish
transactions_manager.commit()
File "/opt/zenoss/lib/python/Zope2/App/startup.py", line 301, in commit
transaction.commit()
File "/opt/zenoss/lib/python/transaction/_manager.py", line 89, in
commit
return self.get().commit()
File "/opt/zenoss/lib/python/transaction/_transaction.py", line 342, in
commit
reraise(t, v, tb)
File "/opt/zenoss/lib/python/transaction/_transaction.py", line 333, in
commit
self._commitResources()
File "/opt/zenoss/lib/python/transaction/_transaction.py", line 473, in
_commitResources
reraise(t, v, tb)
File "/opt/zenoss/lib/python/transaction/_transaction.py", line 447, in
_commitResources
rm.commit(self)
File "/opt/zenoss/lib/python/ZODB/Connection.py", line 567, in commit
self._commit(transaction)
File "/opt/zenoss/lib/python/ZODB/Connection.py", line 623, in _commit
self._store_objects(ObjectWriter(obj), transaction)
File "/opt/zenoss/lib/python/ZODB/Connection.py", line 658, in
_store_objects
p = writer.serialize(obj) # This calls __getstate__ of obj
File "/opt/zenoss/lib/python/ZODB/serialize.py", line 422, in serialize
return self._dump(meta, obj.__getstate__())
SystemError: NULL object passed to Py_BuildValue
Subject: | Managed to solve the rabbitmq |
Author: | [Not Specified] |
Posted: | 2015-02-04 00:36 |
Managed to solve the rabbitmq blocked connection issue by modifying the threshold in rabbitmq-config file.
However, still have the error message about SystemError: NULL object passed to Py_BuildValue
Subject: | zenoss 4 rabbitmq showing blocking connections |
Author: | [Not Specified] |
Posted: | 2017-01-29 05:54 |
I am posting this reply as it is maybe of use to others who find zenoss4 sortof unable to function and blocking messages are appearing.
Step1( dont miss it) check the diskspace on all partitions, if this is below 1% left , then this can cause an issue.
Clearing enough space for messaging basically.
rabbitmqctl list_connections
Listing connections ...
zenoss ::1 55476 running
zenoss 127.0.0.1 45400 running
if the above says blocking or blocked then, this will cause zenoss to fail to work and modeling will fail.
I also found issues around default graphs giving a python error about threaholds. I found reset bindings in the console cleared this.
< |
Previous JMX monitoring through SSH tunnel |
Next Triggers With Graphs Data |
> |