TECHZEN Zenoss User Community ARCHIVE  
FORUMS / Legacy Discussions / DISCUSSION

A Zenoss error has occurred-PLEASE HELP

Subject: A Zenoss error has occurred-PLEASE HELP
Author: [Not Specified]
Posted: 2014-01-15 02:52

Hi Guys,

I recently restarted my Zenoss server and after the reboot the following error came up:

A Zenoss error has occurred:

This is above our paygrade... can somebody please help us to get this server up and running again


Details:



Type:
Value: Service Unavailable 

Traceback (most recent call last):
File "/opt/zenoss/lib/python/ZPublisher/Publish.py", line 126, in publish
request, bind=1)
File "/opt/zenoss/lib/python/ZPublisher/mapply.py", line 77, in mapply
if debug is not None: return debug(object,args,context)
File "/opt/zenoss/lib/python/ZPublisher/Publish.py", line 46, in call_object
result=apply(object,args) # Type s to step into published object.
File "/opt/zenoss/lib/python/Products/Five/browser/metaconfigure.py", line 476, in __call__
return self.index(*args, **kw)
File "/opt/zenoss/lib/python/Products/Five/browser/pagetemplatefile.py", line 125, in __call__
return self.im_func(im_self, *args, **kw)
File "/opt/zenoss/Products/ZenWidgets/FileGzipper.py", line 82, in __call__
return original(self, __instance, *args, **keywords)
File "/opt/zenoss/lib/python/Products/Five/browser/pagetemplatefile.py", line 59, in __call__
sourceAnnotations=getattr(debug_flags, 'sourceAnnotations', 0),
File "/opt/zenoss/lib/python/zope/pagetemplate/pagetemplate.py", line 132, in pt_render
strictinsert=0, sourceAnnotations=sourceAnnotations
File "/opt/zenoss/lib/python/zope/pagetemplate/pagetemplate.py", line 240, in __call__
interpreter()
File "/opt/zenoss/lib/python/zope/tal/talinterpreter.py", line 266, in __call__
self.interpret(self.program)
File "/opt/zenoss/lib/python/zope/tal/talinterpreter.py", line 338, in interpret
handlers[opcode](self, args)
File "/opt/zenoss/lib/python/zope/tal/talinterpreter.py", line 883, in do_useMacro
self.interpret(macro)
File "/opt/zenoss/lib/python/zope/tal/talinterpreter.py", line 338, in interpret
handlers[opcode](self, args)
File "/opt/zenoss/lib/python/zope/tal/talinterpreter.py", line 528, in do_optTag_tal
self.do_optTag(stuff)
File "/opt/zenoss/lib/python/zope/tal/talinterpreter.py", line 513, in do_optTag
return self.no_tag(start, program)
File "/opt/zenoss/lib/python/zope/tal/talinterpreter.py", line 508, in no_tag
self.interpret(program)
File "/opt/zenoss/lib/python/zope/tal/talinterpreter.py", line 338, in interpret
handlers[opcode](self, args)
File "/opt/zenoss/lib/python/zope/tal/talinterpreter.py", line 883, in do_useMacro
self.interpret(macro)
File "/opt/zenoss/lib/python/zope/tal/talinterpreter.py", line 338, in interpret
handlers[opcode](self, args)
File "/opt/zenoss/lib/python/zope/tal/talinterpreter.py", line 528, in do_optTag_tal
self.do_optTag(stuff)
File "/opt/zenoss/lib/python/zope/tal/talinterpreter.py", line 513, in do_optTag
return self.no_tag(start, program)
File "/opt/zenoss/lib/python/zope/tal/talinterpreter.py", line 508, in no_tag
self.interpret(program)
File "/opt/zenoss/lib/python/zope/tal/talinterpreter.py", line 338, in interpret
handlers[opcode](self, args)
File "/opt/zenoss/lib/python/zope/tal/talinterpreter.py", line 883, in do_useMacro
self.interpret(macro)
File "/opt/zenoss/lib/python/zope/tal/talinterpreter.py", line 338, in interpret
handlers[opcode](self, args)
File "/opt/zenoss/lib/python/zope/tal/talinterpreter.py", line 528, in do_optTag_tal
self.do_optTag(stuff)
File "/opt/zenoss/lib/python/zope/tal/talinterpreter.py", line 513, in do_optTag
return self.no_tag(start, program)
File "/opt/zenoss/lib/python/zope/tal/talinterpreter.py", line 508, in no_tag
self.interpret(program)
File "/opt/zenoss/lib/python/zope/tal/talinterpreter.py", line 338, in interpret
handlers[opcode](self, args)
File "/opt/zenoss/lib/python/zope/tal/talinterpreter.py", line 737, in do_insertStructure_tal
structure = self.engine.evaluateStructure(expr)
File "/opt/zenoss/lib/python/Products/PageTemplates/Expressions.py", line 218, in evaluateStructure
text = super(ZopeContext, self).evaluateStructure(expr)
File "/opt/zenoss/lib/python/zope/tales/tales.py", line 696, in evaluate
return expression(self)
File "/opt/zenoss/lib/python/zope/contentprovider/tales.py", line 80, in __call__
return provider.render()
File "/opt/zenoss/Products/ZenUI3/utils/javascript.py", line 28, in render
raw_js = '\n'.join(v.render() for v in self.viewlets)
File "/opt/zenoss/Products/ZenUI3/utils/javascript.py", line 28, in 
raw_js = '\n'.join(v.render() for v in self.viewlets)
File "/opt/zenoss/Products/ZenUI3/utils/javascript.py", line 40, in render
return self.snippet()
File "/opt/zenoss/Products/ZenUI3/browser/eventconsole/grid.py", line 159, in snippet
defs = column_config(self.request, archive=archive)
File "/opt/zenoss/Products/ZenUI3/browser/eventconsole/grid.py", line 115, in column_config
columns = _find_column_definitions(archive)
File "/opt/zenoss/Products/ZenUI3/browser/eventconsole/grid.py", line 71, in _find_column_definitions
details = getFacade('zep').getUnmappedDetails()
File "/opt/zenoss/Products/Zuul/facades/zepfacade.py", line 656, in getUnmappedDetails
return getDetailsInfo().getUnmappedDetails()
File "/opt/zenoss/Products/Zuul/facades/zepfacade.py", line 811, in getUnmappedDetails
self._initDetails()
File "/opt/zenoss/Products/Zuul/facades/zepfacade.py", line 769, in _initDetails
response, content = self._configClient.getDetails()
File "/opt/zenoss/lib/python/zenoss/protocols/services/zep.py", line 257, in getDetails
return self.client.get('index_details')
File "/opt/zenoss/lib/python/zenoss/protocols/services/__init__.py", line 181, in get
return self._request(path, method='GET', params=params, headers=headers)
File "/opt/zenoss/lib/python/zenoss/protocols/services/__init__.py", line 160, in _request
response, content = self._executeRequest(request)
File "/opt/zenoss/lib/python/zenoss/protocols/services/__init__.py", line 154, in _executeRequest
raise ServiceResponseError(response.reason, response.status, request, response, content)
ServiceResponseError: Service Unavailable

Subject: Hi Guys, anybody please?
Author: [Not Specified]
Posted: 2014-01-15 13:42

Hi Guys, anybody please
Subject: Tips
Author: [Not Specified]
Posted: 2014-01-15 14:42

Try these commands/steps:

zenoss stop
zenchkrels -r -x1
zenoss start
# If that doesn't work, try the repair script at this link: http://wiki.zenoss.org/Fixing_ZODB_Integrity_Issues

Also pastebin the output of the following:
df -h
service rabbitmq-server restart
rabbitmqctl -p /zenoss list_queues

Also what version of Zenoss are you running on what OS

Hydruid


Subject: Hi Hydruid,
Author: [Not Specified]
Posted: 2014-01-16 01:08

Hi Hydruid,

Thank you very much for your responce!

When trying the first commands that you gave we get "command not found".

zenoss stop
zenchkrels -r -x1
zenoss start

Same story with zendmd....

Here is the info you were asking for:---------------------------------

root@localhost ~]# df -h
Filesystem Size Used Avail Use% Mounted on
/dev/mapper/VolGroup-lv_root
50G 4.6G 43G 10% /
tmpfs 3.9G 200K 3.9G 1% /dev/shm
/dev/sda1 485M 32M 429M 7% /boot
/dev/mapper/VolGroup-lv_home
407G 199M 386G 1% /home

---------------------------------------------

Restarting rabbitmq-server: SUCCESS
rabbitmq-server.
[root@localhost ~]# rabbitmqctl -p /zenoss list_queues
Listing queues ...
celery 0
zenoss.queues.zep.signal 0
zenoss.queues.zep.rawevents 0
localhost.localdomain.celeryd.pidbox 0
...done.

OS- CENTOS 6.4

ZENOS-4.2.3
Subject: You have to run those
Author: [Not Specified]
Posted: 2014-01-16 08:39

You have to run those commands as the zenoss user!

Hydruid


Subject: It looks like your queue for
Author: [Not Specified]
Posted: 2014-01-16 09:55

It looks like your queue for zenoss doesn't exist in rabbit....try this:

su zenoss # become zenoss user
zenoss stop
sudo su - # become root user
export VHOST="/zenoss"
export USER="zenoss"
export PASS="zenoss" # replace it with $amqppassword from $ZENHOME/etc/global.conf, if needed
rabbitmqctl stop_app
rabbitmqctl reset
rabbitmqctl start_app
rabbitmqctl add_vhost $VHOST
rabbitmqctl add_user "$USER" "$PASS"
rabbitmqctl set_permissions -p "$VHOST" "$USER" '.*' '.*' '.*'
rabbitmqctl -p $VHOST list_queues # check if all queues exist
exit # back to zenoss user
zenoss start

Hydruid


Subject: I'm having a similar/the same issue.
Author: [Not Specified]
Posted: 2014-04-27 18:42

Ubuntu 12.04
Installed July 2013 using Hydruid's scripts.
Worked fine until I don't know when (haven't had time to troubleshoot).

After logging in to the GUI, every tab/screen results in the following error:
http://pastebin.com/w341T5y7

Searched through the community, found two posts, followed instructions in them, still busted.

-- zenoss status - results in all daemons "running".
-- zenchkrels -r -x1 returns "INFO:zen.CheckRelations:Checking relations..." and nothing else.

How do I go about even beginning to troubleshoot :)

Thanks for the help!

Trey


Subject: Errors
Author: [Not Specified]
Posted: 2014-04-27 20:53

In looking through logs, so far I found this in zenstatus.log

2014-04-27 20:48:32,647 INFO zen.maintenance: Performing periodic maintenance
2014-04-27 20:48:32,647 INFO zen.zenstatus: Counter eventCount, value 12
2014-04-27 20:48:32,679 INFO zen.zenstatus: 0 devices processed (0 datapoints)
2014-04-27 20:48:32,679 INFO zen.collector.scheduler: Tasks: 1 Successful_Runs: 0 Failed_Runs: 0 Missed_Runs: 0 Queued_Tasks: 0 Running_Tasks: 0
2014-04-27 20:48:32,689 ERROR zen.maintenance: Maintenance failed. Message from hub: ServiceResponseError: Service Unavailable


Subject: More errors
Author: [Not Specified]
Posted: 2014-04-27 20:55

This is from zenhub.log
2014-04-27 20:50:32,745 INFO zen.ZenHub: Worker (25344) reports Unhandled Error
Traceback (most recent call last):
File "/usr/local/zenoss/lib/python/twisted/spread/pb.py", line 841, in _recvMessage
netResult = object.remoteMessageReceived(self, message, netArgs, netKw)
File "/usr/local/zenoss/lib/python/twisted/spread/flavors.py", line 114, in remoteMessageReceived
state = method(*args, **kw)
File "/usr/local/zenoss/Products/ZenHub/PBDaemon.py", line 84, in inner
return callable(*args, **kw)
File "/usr/local/zenoss/lib/python/twisted/internet/defer.py", line 1141, in unwindGenerator
return _inlineCallbacks(None, f(*args, **kwargs), Deferred())
--- ---
File "/usr/local/zenoss/lib/python/twisted/internet/defer.py", line 1020, in _inlineCallbacks
result = g.send(result)
File "/usr/local/zenoss/Products/ZenHub/zenhubworker.py", line 204, in remote_execute
result = runOnce()
File "/usr/local/zenoss/Products/ZenHub/zenhubworker.py", line 190, in runOnce
res = m(*args, **kw)
File "/usr/local/zenoss/Products/ZenHub/PBDaemon.py", line 92, in inner
traceback.format_exc())
Products.ZenHub.PBDaemon.RemoteException: Remote exception: : Service Unavailable: Traceback (most recent call last):
File "/usr/local/zenoss/Products/ZenHub/PBDaemon.py", line 84, in inner
return callable(*args, **kw)
File "/usr/local/zenoss/Products/ZenHub/services/EventService.py", line 52, in remote_getDevicePingIssues
return zep.getDevicePingIssues()
File "/usr/local/zenoss/Products/Zuul/facades/zepfacade.py", line 617, in getDevicePingIssues
status=[STATUS_NEW,STATUS_ACKNOWLEDGED,STATUS_SUPPRESSED])
File "/usr/local/zenoss/Products/Zuul/facades/zepfacade.py", line 627, in getDeviceIssues
tagSeverities = self._getEventTagSeverities(eventClass, severity, status)
File "/usr/local/zenoss/Products/Zuul/facades/zepfacade.py", line 611, in _getEventTagSeverities
response, content = self.client.getEventTagSeverities(from_dict(EventFilter, eventFilter))
File "/usr/local/zenoss/lib/python/zenoss/protocols/services/zep.py", line 174, in getEventTagSeverities
return self.client.post('tag_severities', body=event_filter)
File "/usr/local/zenoss/lib/python/zenoss/protocols/services/__init__.py", line 175, in post
return self._request_with_body(path, method='POST', body=body, params=params, headers=headers)
File "/usr/local/zenoss/lib/python/zenoss/protocols/services/__init__.py", line 170, in _request_with_body
response, content = self._executeRequest(request)
File "/usr/local/zenoss/lib/python/zenoss/protocols/services/__init__.py", line 154, in _executeRequest
raise ServiceResponseError(response.reason, response.status, request, response, content)
ServiceResponseError: Service Unavailable


Subject: zeneventserver-stdio.2014_04_27.log
Author: [Not Specified]
Posted: 2014-04-27 21:04

http://pastebin.com/QLV2RyQv

This one was a little long.


Subject: Same issue
Author: [Not Specified]
Posted: 2014-04-28 11:00

I'm sorry, I thought it was appropriate since it appears to be the same, or at least a very similar issue, and it didn't appear that there was a specific resolution on this one, and the thread wasn't very old.

But, I understand now, so I'll go post as a new thread. Thanks for the guidance!

Trey



< Previous
Do you have good suggestion for Windows Event filtering 		  Next
all daemon showed as down in porlet but running in Daemon page, error ZepConnect ... 		>