|
|
| Subject: | Error running ZenUp |
| Author: | [Not Specified] |
| Posted: | 2014-04-30 14:58 |
While running an install with ZenUp I received the following error:
Copying JS files into place...
/tmp/zenoss-core-4.2.4_GeChDv/new_zup/custom/blobs/el6/zenoss-compiled-debug.js
/tmp/zenoss-core-4.2.4_GeChDv/new_zup/custom/blobs/el6/zenoss-compiled.js
Traceback (most recent call last):
File "/opt/zenoss/bin/easy_install", line 5, in
from pkg_resources import load_entry_point
ImportError: No module named pkg_resources
ERROR: /tmp/zenoss-core-4.2.4_GeChDv/new_zup/post returned a non-zero exit code
When I executed it with --dry-run it came out saying successful. Anyway to resolve this At this point my system is down and I'm looking at going back to the backup.
Thanks
| Subject: | Did you not backup your |
| Author: | [Not Specified] |
| Posted: | 2014-04-30 15:08 |
Did you not backup your system like the ZenUp Wiki Article suggests to
Hydruid
| Subject: | No I did backup, but I'd like |
| Author: | [Not Specified] |
| Posted: | 2014-04-30 15:11 |
No I did backup, but I'd like to get the update installed.
| Subject: | Since it errored out you |
| Author: | [Not Specified] |
| Posted: | 2014-04-30 15:25 |
Since it errored out you could also try it again and use the --force option but you will most likely have to restore the backup and investigate what went wrong!
Hydruid
| Subject: | I'll try with the force |
| Author: | [Not Specified] |
| Posted: | 2014-04-30 15:26 |
I'll try with the force option, I'm recovering the back up now.
Thanks
| Subject: | I've completed the restore |
| Author: | [Not Specified] |
| Posted: | 2014-04-30 16:30 |
I've completed the restore but when I pull up the dashboard I get the following error:
Type:
Value: Service Unavailable
Traceback (most recent call last): File "/opt/zenoss/lib/python/ZPublisher/Publish.py", line 126, in publish request, bind=1) File "/opt/zenoss/lib/python/ZPublisher/mapply.py", line 77, in mapply if debug is not None: return debug(object,args,context) File "/opt/zenoss/lib/python/ZPublisher/Publish.py", line 46, in call_object result=apply(object,args) # Type s
I can see the top with the "tabs" but the message is where the body would be.
| Subject: | What kind of backup did you |
| Author: | [Not Specified] |
| Posted: | 2014-05-01 09:49 |
What kind of backup did you do Did you make sure that zenoss wasn't running when you restored
Hydruid
| Subject: | I created a tar file as the |
| Author: | [Not Specified] |
| Posted: | 2014-05-01 10:16 |
I created a tar file as the directions instructed "tar --exclude backups --exclude perf --exclude log -czf zenoss_core-4.2.X-SPXXX_backup.[TIMESTAMP].tgz /opt/zenoss".
The service should have been stopped while I moved the files back. I can try again if that might be work, force the service to stop and re-untar. Should I do that
| Subject: | Yes and make sure to |
| Author: | [Not Specified] |
| Posted: | 2014-05-01 11:20 |
Yes and make sure to overwrite any files!
Hydruid
| Subject: | what I normally do is untar |
| Author: | [Not Specified] |
| Posted: | 2014-05-01 11:21 |
what I normally do is untar it to say /home/user/temp and then rsync it to $ZENHOME
rsync -zav /home/user/temp/opt/zenoss /opt/zenoss
or
rsync -zav /home/user/temp/opt/zenoss /opt/
Something like that
Hydruid
| Subject: | I've untar'd the files into |
| Author: | [Not Specified] |
| Posted: | 2014-05-01 17:19 |
I've untar'd the files into the $ZENHOME path and restored the 3 databases to the point they were at prior to running ZenUp and I'm still getting the following error. I made sure that the service was stopped. I can login to Zenoss fine but only the top tabs appear. The body of the site has the error. For different tabs it changes slightly. Is there a way to tell what the response is or what service is unavailable
Type:
Value: Service Unavailable
Traceback (most recent call last): File "/opt/zenoss/lib/python/ZPublisher/Publish.py", line 126, in publish request, bind=1) File "/opt/zenoss/lib/python/ZPublisher/mapply.py", line 77, in mapply if debug is not None: return debug(object,args,context) File "/opt/zenoss/lib/python/ZPublisher/Publish.py", line 46, in call_object result=apply(object,args) # Type s
| Subject: | Did you untar to a temp |
| Author: | [Not Specified] |
| Posted: | 2014-05-02 08:33 |
Did you untar to a temp directory first and then rsync them like I sugested or did you just untar it directly into $ZENHOME Did you triple check to make sure they went to the right place
If so, make sure that all your daemons are running and check the zenhub.log!
Hydruid
| Subject: | I used rsync still with no |
| Author: | [Not Specified] |
| Posted: | 2014-05-02 09:59 |
I used rsync still with no luck. It appears that all the daemons are running. I'd not looked at zenhub.log before and it's repeating an error that looks to be something with rabbitMQ. Any thoughts
2014-05-02 10:55:25,543 INFO zen.ZenHub: Worker (32584) reports 2014-05-02 10:55:25,542 CRITICAL zen.zenoss.protocols.amqp: Could not use exchange $Heartbeats: [Errno 104] Connection reset by peer
2014-05-02 10:55:25,542 CRITICAL zen.Events: Unable to publish event to
2014-05-02 10:55:28,555 INFO zen.ZenHub: Worker (32584) reports 2014-05-02 10:55:28,554 CRITICAL zen.zenoss.protocols.amqp: Could not use exchange $RawZenEvents: [Errno 104] Connection reset by peer
2014-05-02 10:55:28,554 CRITICAL zen.Events: Unable to publish event to
2014-05-02 10:55:31,574 INFO zen.ZenHub: Worker (32584) reports 2014-05-02 10:55:31,573 INFO zen.zenoss.protocols.amqp: RabbitMQ connection was closed: [Errno 104] Connection reset by peer
2014-05-02 10:55:32,613 INFO zen.ZenHub: Worker (32585) reports 2014-05-02 10:55:32,611 CRITICAL zen.zenoss.protocols.amqp: Could not use exchange $RawZenEvents: [Errno 104] Connection reset by peer
2014-05-02 10:55:32,611 CRITICAL zen.Events: Unable to publish event to
2014-05-02 10:55:32,679 INFO zen.ZenHub: Worker (32585) reports Exception
2014-05-02 10:55:32,688 INFO zen.ZenHub: Worker (32585) reports socket
2014-05-02 10:55:32,688 INFO zen.ZenHub: Worker (32585) reports .
2014-05-02 10:55:32,689 INFO zen.ZenHub: Worker (32585) reports error
2014-05-02 10:55:32,689 INFO zen.ZenHub: Worker (32585) reports :
2014-05-02 10:55:32,690 INFO zen.ZenHub: Worker (32585) reports (
2014-05-02 10:55:32,690 INFO zen.ZenHub: Worker (32585) reports 107
2014-05-02 10:55:32,690 INFO zen.ZenHub: Worker (32585) reports ,
2014-05-02 10:55:32,691 INFO zen.ZenHub: Worker (32585) reports '
2014-05-02 10:55:32,694 INFO zen.ZenHub: Worker (32585) reports Transport endpoint is not connected') in
2014-05-02 10:55:32,709 INFO zen.ZenHub: Worker (32585) reports Unhandled Error
Traceback (most recent call last):
File "/opt/zenoss/lib/python/twisted/spread/pb.py", line 841, in _recvMessage
netResult = object.remoteMessageReceived(self, message, netArgs, netKw)
File "/opt/zenoss/lib/python/twisted/spread/flavors.py", line 114, in remoteMessageReceived
state = method(*args, **kw)
File "/opt/zenoss/Products/ZenHub/PBDaemon.py", line 85, in inner
return callable(*args, **kw)
File "/opt/zenoss/lib/python/twisted/internet/defer.py", line 1141, in unwindGenerator
return _inlineCallbacks(None, f(*args, **kwargs), Deferred())
---
File "/opt/zenoss/lib/python/twisted/internet/defer.py", line 1020, in _inlineCallbacks
result = g.send(result)
File "/opt/zenoss/Products/ZenHub/zenhubworker.py", line 204, in remote_execute
result = runOnce()
File "/opt/zenoss/Products/ZenHub/zenhubworker.py", line 190, in runOnce
res = m(*args, **kw)
File "/opt/zenoss/Products/ZenHub/PBDaemon.py", line 93, in inner
traceback.format_exc())
Products.ZenHub.PBDaemon.RemoteException: Remote exception:
File "/opt/zenoss/Products/ZenHub/PBDaemon.py", line 85, in inner
return callable(*args, **kw)
File "/opt/zenoss/Products/ZenHub/services/EventService.py", line 52, in remote_getDevicePingIssues
return zep.getDevicePingIssues()
File "/opt/zenoss/Products/Zuul/facades/zepfacade.py", line 644, in getDevicePingIssues
status=[STATUS_NEW,STATUS_ACKNOWLEDGED,STATUS_SUPPRESSED])
File "/opt/zenoss/Products/Zuul/facades/zepfacade.py", line 654, in getDeviceIssues
tagSeverities = self._getEventTagSeverities(eventClass, severity, status)
File "/opt/zenoss/Products/Zuul/facades/zepfacade.py", line 638, in _getEventTagSeverities
response, content = self.client.getEventTagSeverities(from_dict(EventFilter, eventFilter))
File "/opt/zenoss/lib/python/zenoss/protocols/services/zep.py", line 174, in getEventTagSeverities
return self.client.post('tag_severities', body=event_filter)
File "/opt/zenoss/lib/python/zenoss/protocols/services/__init__.py", line 175, in post
return self._request_with_body(path, method='POST', body=body, params=params, headers=headers)
File "/opt/zenoss/lib/python/zenoss/protocols/services/__init__.py", line 170, in _request_with_body
response, content = self._executeRequest(request)
File "/opt/zenoss/lib/python/zenoss/protocols/services/__init__.py", line 154, in _executeRequest
raise ServiceResponseError(response.reason, response.status, request, response, content)
ServiceResponseError: Service Unavailable
2014-05-02 10:55:34,577 INFO zen.ZenHub: Worker (32584) reports 2014-05-02 10:55:34,576 INFO zen.zenoss.protocols.amqp: RabbitMQ connection was closed: [Errno 104] Connection reset by peer
2014-05-02 10:55:34,577 INFO zen.ZenHub: Worker (32584) reports 2014-05-02 10:55:34,576 ERROR zen.Events: Unable to publish event to
Traceback (most recent call last):
File "/opt/zenoss/Products/ZenEvents/MySqlSendEvent.py", line 43, in sendEvents
self._publishEvent(event, publisher)
File "/opt/zenoss/Products/ZenEvents/MySqlSendEvent.py", line 89, in _publishEvent
return self._sendHeartbeat(event)
File "/opt/zenoss/Products/ZenEvents/MySqlSendEvent.py", line 107, in _sendHeartbeat
publisher.publish('$Heartbeats', 'zenoss.heartbeat.%s' % heartbeat.monitor, heartbeat)
File "/opt/zenoss/Products/ZenMessaging/queuemessaging/publisher.py", line 425, in publish
headers=headers, declareExchange=declareExchange)
File "/opt/zenoss/lib/python/zenoss/protocols/amqp.py", line 177, in publish
raise Exception("Could not publish message to RabbitMQ: %s" % lastexc)
| Subject: | I believe we've resolved some |
| Author: | [Not Specified] |
| Posted: | 2014-05-02 13:18 |
I believe we've resolved some of the rabbitMQ issue by resetting it. However on restarting rabbitMQ and Zenoss not all of the queues are recreated. When I run "/usr/sbin/rabbitmqctl -p /zenoss list_queues"
Here's what I get:
Listing queues ...
celery 0
zenoss.queues.zep.signal 0
zenoss.queues.zep.rawevents 0
dbsmon01.celeryd.pidbox 0
...done.
| Subject: | Recreate your rabbit queue's |
| Author: | [Not Specified] |
| Posted: | 2014-05-02 14:40 |
Recreate your rabbit queue's http://wiki.zenoss.org/Working_with_Queues
Hydruid
| Subject: | That's exactly what I did and |
| Author: | [Not Specified] |
| Posted: | 2014-05-02 15:36 |
That's exactly what I did and I'm only getting those queues to be recreated.
| Subject: | Are you still getting the |
| Author: | [Not Specified] |
| Posted: | 2014-05-02 16:07 |
Are you still getting the same errors in zenhub.log Check the credentials in $ZENHOME/etc/global.conf for the rabbit credentials to make sure they're zenoss for both username/password
Hydruid
| Subject: | The credentials are correct, |
| Author: | [Not Specified] |
| Posted: | 2014-05-05 09:39 |
The credentials are correct, it's using them to remake some of the queues, just not all of them.
| Subject: | You could try this to fix the |
| Author: | [Not Specified] |
| Posted: | 2014-05-06 15:05 |
You could try this to fix the rabbit queue's
rabbitmqctl stop_app
rabbitmqctl reset
rabbitmqctl start_app
Make sure to Stop zenoss first, do the above comands, then start zenoss. If the queue's still aren't there, use the wiki page previously linked to rebuild them.
Hydruid
| Subject: | That's exactly what I've done |
| Author: | [Not Specified] |
| Posted: | 2014-05-07 13:32 |
That's exactly what I've done as it's listed in the wiki article. The wiki mentions that after resetting everything the processes should recreate the queues on startup, which they're not.
| Subject: | Did you double check the |
| Author: | [Not Specified] |
| Posted: | 2014-05-07 13:49 |
Did you double check the credentials
Hydruid
| Subject: | Yes, I'm positive the |
| Author: | [Not Specified] |
| Posted: | 2014-05-08 12:54 |
Yes, I'm positive the credentials are correct. It's been using them to successfully remake some of the queues. I'm also not seeing any errors in the log regarding a credential issue.
| Subject: | Thanks to your help I was |
| Author: | [Not Specified] |
| Posted: | 2014-05-12 13:14 |
Thanks to your help I was able to get it fixed! Over the weekend we had /opt restored from a backup of the server and it still was no working. We started it up and it was still producing the same errors. It turns out there was something wrong with the Zeneventserver daemon. Casting a bit more of a wildcard out to Google I found another of your posts that solved the issue. The last post on http://www.zenoss.org/forum/971, what exactly does "zeneventserver stop && cd $ZENHOME/var/zeneventserver/index && rm -rf summary && rm -rf archive && zeneventserver start" do
Obviously it restarts the daemon but what's the usage of the files it deletes Either way it solved my problem going but now we're going to need to figure out how to update to 4.2.5.
| Subject: | Glad you got it worked out, |
| Author: | [Not Specified] |
| Posted: | 2014-05-13 13:11 |
Glad you got it worked out, ping rmatte on IRC sometime and ask him. He said something about deleting zope indexes but don't quote me on that.
Hydruid
| < |
Previous Zenoss 4.2.5 stops graphing some time after startup |
Next zenoss.protocols.services.ServiceResponseError - Value: Service Unavailable |
> |