TECHZEN Zenoss User Community ARCHIVE  

Rabbitmq Blocking Zenoss Connections

Subject: Rabbitmq Blocking Zenoss Connections
Author: Benjamin Dronen
Posted: 2015-01-07 14:27

In my Zenoss dashboard, the Daemon Processes down portlet shows these daemons as being down: zenactiond, zencommand, zeneventd, zenhub, zenmodeler, zenperfsnmp, zenping, zenprocess, zenpython, zenstatus, zensyslog, zentrap, and zenwinsrvsnmp

This is similar to the problem described by QnA here: http://www.zenoss.org/forum/171

His solution ended up being that RabbitMQ was blocking his connections due to a low memory threshold described here: http://community.zenoss.org/thread/18511start=0&tstart=0

I checked rabbitmq connections using rabbitmqctl list_connections, and it gave me this output:

Listing connections ...
zenoss 127.0.0.1 43886 blocked
zenoss 127.0.0.1 43870 blocked
zenoss 127.0.0.1 52474 blocking
zenoss 127.0.0.1 43877 blocking
zenoss 127.0.0.1 52685 blocked
zenoss 127.0.0.1 52687 blocked
zenoss 127.0.0.1 43895 blocking
zenoss 127.0.0.1 43902 blocking
zenoss 127.0.0.1 43876 blocked
zenoss 127.0.0.1 52686 blocked
zenoss 127.0.0.1 43896 blocking
zenoss 127.0.0.1 43903 blocked
...done.

I increased my memory threshold to 0.7 and restarted both rabbitmq, zenoss, and the server itself to no avail.

Any suggestions

Zenoss 4.2.5
Centos 5



Subject: Disregard
Author: Benjamin Dronen
Posted: 2015-01-07 14:46

It seems as though the daemons are running properly now and rabbitmq is no longer blocking any connections.



Subject: yeah rabbit often takes a few
Author: Andrew Kirch
Posted: 2015-01-08 10:20

yeah rabbit often takes a few to resolve when it breaks, reply if there are further issues.

Andrew Kirch

akirch@gvit.com

Need Zenoss support, consulting or custom development Look no further. Email or PM me!

Ready for Distributed Topology (collectors) for Zenoss 5 Coming May 1st from GoVanguard



Subject: Started Blocking Connections
Author: Benjamin Dronen
Posted: 2015-01-14 15:11

Not sure why this has happened again, but it seems to be blocking all the daemons once again.

Running list-connections gives me this again:

Listing connections ...
zenoss 127.0.0.1 43146 blocking
zenoss 127.0.0.1 43144 blocking
zenoss 127.0.0.1 43138 blocked
zenoss 127.0.0.1 43145 blocked
zenoss 127.0.0.1 43137 blocking
zenoss 127.0.0.1 43147 blocked
zenoss 127.0.0.1 43143 blocking
zenoss 127.0.0.1 43134 blocked
...done.

Any further suggestions



Subject: RabbitMQ can block
Author: Jan Garaj
Posted: 2015-01-14 17:00

RabbitMQ can block connections because:
1.) Memory
https://www.rabbitmq.com/memory.html
2.) Disk
https://www.rabbitmq.com/disk-alarms.html

So check your RabbitMQ settings and current values for memory + disk also.

Devops Monitoring Expert advice: Dockerize/automate/monitor all the things.

DevOps stack: Docker / Kubernetes / Mesos / Zabbix / Zenoss / Grafana / Puppet / Ansible / Vagrant / Terraform / Elasticsearch



Subject: you may also want to install
Author: Andrew Kirch
Posted: 2015-01-19 08:24

you may also want to install the RabbitMQ ZenPack

Andrew Kirch

akirch@gvit.com

Need Zenoss support, consulting or custom development Look no further. Email or PM me!

Ready for Distributed Topology (collectors) for Zenoss 5 Coming May 1st from GoVanguard



Subject: same issue here
Author: [Not Specified]
Posted: 2015-01-26 22:54

I face the same issue here as well. The rabbitmq connections are blocked even I have reset them.
Any idea how to solve this issue

Currently, all the events are not coming in and the GUI prompts unable to connect to the server message.

How to change the rabbitmq.config I can't find this find in the whole server.

Zenoss 4.2.5
RH Linux 6.5

I am not using auto-deploy for installation, but manual installation. The system has been running for 2 months without problem before the issue happen.



Subject: It depends on your config,
Author: Jan Garaj
Posted: 2015-01-27 00:57

It depends on your config, but generally it'll be /etc/rabbitmq/rabbitmq.config (If it doesn't exist, then default values are used). Use example config https://github.com/rabbitmq/rabbitmq-server/blob/master/docs/rabbitmq.co... and uncomment what do you need ({disk_free_limit, {mem_relative, 1.0}}, ....) save and restart rabbitmq-server. Check rabbitmq logs if new settings have been applied.

Devops Monitoring Expert advice: Dockerize/automate/monitor all the things.

DevOps stack: Docker / Kubernetes / Mesos / Zabbix / Zenoss / Grafana / Puppet / Ansible / Vagrant / Terraform / Elasticsearch



Subject: Is there any possible
Author: [Not Specified]
Posted: 2015-01-27 06:23

Is there any possible solution for the blocked rabbitmq connections

Besides, is the error message below a symptom of this problem I get this error message.

Type:
Value: NULL object passed to Py_BuildValue

Traceback (most recent call last):
File "/opt/zenoss/lib/python/ZPublisher/Publish.py", line 134, in
publish
transactions_manager.commit()
File "/opt/zenoss/lib/python/Zope2/App/startup.py", line 301, in commit
transaction.commit()
File "/opt/zenoss/lib/python/transaction/_manager.py", line 89, in
commit
return self.get().commit()
File "/opt/zenoss/lib/python/transaction/_transaction.py", line 342, in
commit
reraise(t, v, tb)
File "/opt/zenoss/lib/python/transaction/_transaction.py", line 333, in
commit
self._commitResources()
File "/opt/zenoss/lib/python/transaction/_transaction.py", line 473, in
_commitResources
reraise(t, v, tb)
File "/opt/zenoss/lib/python/transaction/_transaction.py", line 447, in
_commitResources
rm.commit(self)
File "/opt/zenoss/lib/python/ZODB/Connection.py", line 567, in commit
self._commit(transaction)
File "/opt/zenoss/lib/python/ZODB/Connection.py", line 623, in _commit
self._store_objects(ObjectWriter(obj), transaction)
File "/opt/zenoss/lib/python/ZODB/Connection.py", line 658, in
_store_objects
p = writer.serialize(obj) # This calls __getstate__ of obj
File "/opt/zenoss/lib/python/ZODB/serialize.py", line 422, in serialize
return self._dump(meta, obj.__getstate__())
SystemError: NULL object passed to Py_BuildValue



Subject: Managed to solve the rabbitmq
Author: [Not Specified]
Posted: 2015-02-04 00:36

Managed to solve the rabbitmq blocked connection issue by modifying the threshold in rabbitmq-config file.

However, still have the error message about SystemError: NULL object passed to Py_BuildValue



Subject: zenoss 4 rabbitmq showing blocking connections
Author: [Not Specified]
Posted: 2017-01-29 05:54

I am posting this reply as it is maybe of use to others who find zenoss4 sortof unable to function and blocking messages are appearing.

Step1( dont miss it) check the diskspace on all partitions, if this is below 1% left , then this can cause an issue.

Clearing enough space for messaging basically.

rabbitmqctl list_connections
Listing connections ...
zenoss ::1 55476 running
zenoss 127.0.0.1 45400 running

if the above says blocking or blocked then, this will cause zenoss to fail to work and modeling will fail.

I also found issues around default graphs giving a python error about threaholds. I found reset bindings in the console cleared this.



< Previous
JMX monitoring through SSH tunnel
  Next
Triggers With Graphs Data
>