Type: <class 'zenoss.protocols.services.ServiceResponseError'> 
Value: Service Unavailable 

Traceback (most recent call last):
File "/opt/zenoss/lib/python/ZPublisher/Publish.py", line 126, in publish
request, bind=1)
File "/opt/zenoss/lib/python/ZPublisher/mapply.py", line 77, in mapply
if debug is not None: return debug(object,args,context)
File "/opt/zenoss/lib/python/ZPublisher/Publish.py", line 46, in call_object
result=apply(object,args) # Type s<cr> to step into published object.
File "/opt/zenoss/lib/python/Products/Five/browser/metaconfigure.py", line 476, in __call__
return self.index(*args, **kw)
File "/opt/zenoss/lib/python/Products/Five/browser/pagetemplatefile.py", line 125, in __call__
return self.im_func(im_self, *args, **kw)
File "/opt/zenoss/Products/ZenWidgets/FileGzipper.py", line 82, in __call__
return original(self, __instance, *args, **keywords)
File "/opt/zenoss/lib/python/Products/Five/browser/pagetemplatefile.py", line 59, in __call__
sourceAnnotations=getattr(debug_flags, 'sourceAnnotations', 0),
File "/opt/zenoss/lib/python/zope/pagetemplate/pagetemplate.py", line 132, in pt_render
strictinsert=0, sourceAnnotations=sourceAnnotations
File "/opt/zenoss/lib/python/zope/pagetemplate/pagetemplate.py", line 240, in __call__
interpreter()
File "/opt/zenoss/lib/python/zope/tal/talinterpreter.py", line 266, in __call__
self.interpret(self.program)
File "/opt/zenoss/lib/python/zope/tal/talinterpreter.py", line 338, in interpret
handlers[opcode](self, args)
File "/opt/zenoss/lib/python/zope/tal/talinterpreter.py", line 883, in do_useMacro
self.interpret(macro)
File "/opt/zenoss/lib/python/zope/tal/talinterpreter.py", line 338, in interpret
handlers[opcode](self, args)
File "/opt/zenoss/lib/python/zope/tal/talinterpreter.py", line 528, in do_optTag_tal
self.do_optTag(stuff)
File "/opt/zenoss/lib/python/zope/tal/talinterpreter.py", line 513, in do_optTag
return self.no_tag(start, program)
File "/opt/zenoss/lib/python/zope/tal/talinterpreter.py", line 508, in no_tag
self.interpret(program)
File "/opt/zenoss/lib/python/zope/tal/talinterpreter.py", line 338, in interpret
handlers[opcode](self, args)
File "/opt/zenoss/lib/python/zope/tal/talinterpreter.py", line 883, in do_useMacro
self.interpret(macro)
File "/opt/zenoss/lib/python/zope/tal/talinterpreter.py", line 338, in interpret
handlers[opcode](self, args)
File "/opt/zenoss/lib/python/zope/tal/talinterpreter.py", line 528, in do_optTag_tal
self.do_optTag(stuff)
File "/opt/zenoss/lib/python/zope/tal/talinterpreter.py", line 513, in do_optTag
return self.no_tag(start, program)
File "/opt/zenoss/lib/python/zope/tal/talinterpreter.py", line 508, in no_tag
self.interpret(program)
File "/opt/zenoss/lib/python/zope/tal/talinterpreter.py", line 338, in interpret
handlers[opcode](self, args)
File "/opt/zenoss/lib/python/zope/tal/talinterpreter.py", line 737, in do_insertStructure_tal
structure = self.engine.evaluateStructure(expr)
File "/opt/zenoss/lib/python/Products/PageTemplates/Expressions.py", line 218, in evaluateStructure
text = super(ZopeContext, self).evaluateStructure(expr)
File "/opt/zenoss/lib/python/zope/tales/tales.py", line 696, in evaluate
return expression(self)
File "/opt/zenoss/lib/python/zope/contentprovider/tales.py", line 80, in __call__
return provider.render()
File "/opt/zenoss/Products/ZenUI3/utils/javascript.py", line 28, in render
raw_js = '\n'.join(v.render() for v in self.viewlets)
File "/opt/zenoss/Products/ZenUI3/utils/javascript.py", line 28, in <genexpr>
raw_js = '\n'.join(v.render() for v in self.viewlets)
File "/opt/zenoss/Products/ZenUI3/utils/javascript.py", line 40, in render
return self.snippet()
File "/opt/zenoss/Products/ZenUI3/browser/eventconsole/grid.py", line 221, in snippet
defs = column_config(self.request, archive=archive)
File "/opt/zenoss/Products/ZenUI3/browser/eventconsole/grid.py", line 176, in column_config
columns = _find_column_definitions(archive)
File "/opt/zenoss/Products/ZenUI3/browser/eventconsole/grid.py", line 132, in _find_column_definitions
details = getFacade('zep').getUnmappedDetails()
File "/opt/zenoss/Products/Zuul/facades/zepfacade.py", line 692, in getUnmappedDetails
return getDetailsInfo().getUnmappedDetails()
File "/opt/zenoss/Products/Zuul/facades/zepfacade.py", line 862, in getUnmappedDetails
self._initDetails()
File "/opt/zenoss/Products/Zuul/facades/zepfacade.py", line 820, in _initDetails
response, content = self._configClient.getDetails()
File "/opt/zenoss/lib/python/zenoss/protocols/services/zep.py", line 261, in getDetails
return self.client.get('index_details')
File "/opt/zenoss/lib/python/zenoss/protocols/services/__init__.py", line 181, in get
return self._request(path, method='GET', params=params, headers=headers)
File "/opt/zenoss/lib/python/zenoss/protocols/services/__init__.py", line 160, in _request
response, content = self._executeRequest(request)
File "/opt/zenoss/lib/python/zenoss/protocols/services/__init__.py", line 154, in _executeRequest
raise ServiceResponseError(response.reason, response.status, request, response, content)
ServiceResponseError: Service Unavailable

---

Subject:	RE: Zenoss Core URL login
Author:	Ted Mottinger
Posted:	2017-09-26 11:18

BTW, here is the zenoss status......



------------------------------
Ted Mottinger
Network Engineer
Guy M Turner Inc.
Greensboro NC
3368099233
------------------------------

---

Subject:	RE: Zenoss Core URL login
Author:	Jane Curry
Posted:	2017-09-26 11:39

I have seen this happen when the indexes for the events databases are messed up.  You can throw these indexes away and they will be automatically recreated:

zenoss stop
cd $ZENHOME/var/zeneventserver/index
ls  -l          (You should see an archive and a summary subdirectory)
rm -rf *           (Removes both directories and their contents)
zenoss start

Cheers,
Jane

------------------------------
Jane Curry
Skills 1st United Kingdom
jane.curry@skills-1st.co.uk
------------------------------

---

Subject:	RE: Zenoss Core URL login
Author:	Ted Mottinger
Posted:	2017-09-27 09:41

Thanks so much Jane for your help.  I did as you instructed and deleted the indexes for the events database but I still have the same issue.  Looks like it is the same error as before.  

Type: <class 'zenoss.protocols.services.ServiceResponseError'> 
Value: Service Unavailable 

------------------------------
Ted Mottinger
Network Engineer
Guy M Turner Inc.
Greensboro NC
3368099233
------------------------------

---

Subject:	RE: Zenoss Core URL login
Author:	Jane Curry
Posted:	2017-09-27 10:11

Maybe it is the RabbitMQ that has broken, especially if you have changed users / passwords?  Your log file looks like the issue is somewhere around the events subsystem (which relies on RabbitMQ).

As the root user, try
rabbitmqctl -p /zenoss list_queues

You should see about 9 queues.  If you don't then you need to rebuild the queues.  It is documented in the Core Admin guide in section 14.7 (get the Admin guide from here -  https://www.zenohttps://www.zenoss.com/sites/default/files/Zenoss_Core_Administration_02-022014-4.2-v08.pdfss.com/sites/default/files/Zenoss_Core_Administration_02-022014-4.2-v08.pdf  , if you don't already have it.  Then restart your Zenoss again.

BTW - where did you get the log file you posted?
Does any part of the Zenoss GUI work?

Cheers,
Jane

------------------------------
Jane Curry
Skills 1st United Kingdom
jane.curry@skills-1st.co.uk
------------------------------

---

Subject:	RE: Zenoss Core URL login
Author:	Ted Mottinger
Posted:	2017-09-29 15:09

Hi, I did not have 9 queues so I ran the following with no errors....

export VHOST="/zenoss"
export USER="zenoss"
export PASS="zenoss"
rabbitmqctl stop_app
rabbitmqctl reset
rabbitmqctl start_app
rabbitmqctl add_vhost "$VHOST"
rabbitmqctl add_user "$USER" "$PASS"
rabbitmqctl set_permissions -p "$VHOST" "$USER" '.*' '.*' '.*'

I restarted and re-ran the command rabbitmqctl -p /zenoss list_queues and still have 0 queues.  Did I not reset the queues correctly?

------------------------------
Ted Mottinger
Network Engineer
Guy M Turner Inc.
Greensboro NC
3368099233
------------------------------

---

Subject:	RE: Zenoss Core URL login
Author:	Jane Curry
Posted:	2017-09-30 09:06

OK. We've found the basic problem - no queues -> no events subsystem and you get the error messages you are seeing.

Two possibilities on the fix process not working.  The first is typos in your process - check you really do have the dollar symbols...

The other possibility is that the password isn't zenoss. There were some 4.2.x installs that automatically applied a "secure" patch that generated random passwords for various aspects of zenoss.  Have a look in $ZENHOME/etc  at global.conf.  There are usernames and passwords for various aspects of zenoss.  Amongst other entries, I have the following for the RabbitMQ subsystem:

amqphost localhost
amqpport 5672
amqpvhost /zenoss
amqpuser zenoss
amqppassword zenoss
amqpusessl 0
amqpadminport 55672
amqpadminusessl 0

I think this is the default. If you have anything different as amqpvhost, amqpuser or amqppassword, then substitute your values in the fix process.

Cheers,
Jane

------------------------------
Jane Curry
Skills 1st United Kingdom
jane.curry@skills-1st.co.uk
------------------------------

---

Subject:	RE: Zenoss Core URL login
Author:	Ted Mottinger
Posted:	2017-10-02 09:19

Sorry, I'm not sure what you mean as "fix process"?

------------------------------
Ted Mottinger
Network Engineer
Guy M Turner Inc.
Greensboro NC
3368099233
------------------------------

---

Subject:	RE: Zenoss Core URL login
Author:	Ted Mottinger
Posted:	2017-10-02 09:24

Here is what I have in the global.conf file.....

amqphost localhost
amqpport 5672
amqpvhost /zenoss
amqpuser zenoss
amqppassword "user-password"
amqpusessl 0
amqpadminport 55672
amqpadminusessl 0

------------------------------
Ted Mottinger
Network Engineer
Guy M Turner Inc.
Greensboro NC
3368099233
------------------------------

---

Subject:	RE: Zenoss Core URL login
Author:	Jane Curry
Posted:	2017-10-03 02:57

So the "fix process" should be:

export VHOST="/zenoss"
export USER="zenoss"
export PASS="user-password"
rabbitmqctl stop_app
rabbitmqctl reset
rabbitmqctl start_app
rabbitmqctl add_vhost "$VHOST"
rabbitmqctl add_user "$USER" "$PASS"
rabbitmqctl set_permissions -p "$VHOST" "$USER" '.*' '.*' '.*'

Cheers,
Jane

------------------------------
Jane Curry
Skills 1st United Kingdom
jane.curry@skills-1st.co.uk
------------------------------

---

Subject:	RE: Zenoss Core URL login
Author:	Jane Curry
Posted:	2017-10-02 09:27

By "fix process" I meant your:

export VHOST="/zenoss"
export USER="zenoss"
export PASS="zenoss"
rabbitmqctl stop_app
rabbitmqctl reset
rabbitmqctl start_app
rabbitmqctl add_vhost "$VHOST"
rabbitmqctl add_user "$USER" "$PASS"
rabbitmqctl set_permissions -p "$VHOST" "$USER" '.*' '.*' '.*'

So it is particularly the
export PASS = "zenoss"
line that you need to check against what you have for amqppassword in global.conf.

Does that make more sense?
Cheers,
Jane

------------------------------
Jane Curry
Skills 1st United Kingdom
jane.curry@skills-1st.co.uk
------------------------------

---

Subject:	RE: Zenoss Core URL login
Author:	Ted Mottinger
Posted:	2017-10-03 15:15

So, after I run the exports, do I login with username:zenoss and password:zenoss?  I did that and it gave an error "Your session has expired,
or the entered password, or the username is incorrect."  When I use a different username and password that has been letting me go through after successfully logging in I still get the same error 

A Zenoss error has occurred

------------------------------
Ted Mottinger
Network Engineer
Guy M Turner Inc.
Greensboro NC
3368099233
------------------------------

---

Subject:	RE: Zenoss Core URL login
Author:	Jane Curry
Posted:	2017-10-04 14:34

The amqppassword which appears to be user-password in your environment, should only be used when running the script to fix the rabbit queues.  Having done that, what does the:
rabbitmqctl -p /zenoss list_queues
command give? Do you see the queues?  Until you do, you are not going to get past the Service Unavailable error.

You should login to the GUI with your usual GUI userid and password.

Cheers,
Jane

------------------------------
Jane Curry
Skills 1st United Kingdom
jane.curry@skills-1st.co.uk
------------------------------

---

Subject:	RE: Zenoss Core URL login
Author:	Ted Mottinger
Posted:	2017-10-05 08:40

I guess I have no queues.  Do I need to reinstall?

[root@zenoss ~]# rabbitmqctl -p /zenoss list_queues
Listing queues ...
...done.
[root@zenoss ~]#

------------------------------
Ted Mottinger
Network Engineer
Guy M Turner Inc.
Greensboro NC
3368099233
------------------------------

---

Subject:	RE: Zenoss Core URL login
Author:	Ted Mottinger
Posted:	2017-10-05 08:59

So I went through the steps starting from the beginning of this thread and now have some queues but still get the same error.  The log is coming from the GUI but doesn't go any further.  See below.  Thanks.

[root@zenoss ~]# rabbitmqctl -p /zenoss list_queues
Listing queues ...
celery 0
zenoss.queues.zep.signal 0
zenoss.queues.zep.rawevents 0
zenoss.gmturner.com.celeryd.pidbox 0
...done.
[root@zenoss ~]#

------------------------------
Ted Mottinger
Network Engineer
Guy M Turner Inc.
Greensboro NC
3368099233
------------------------------

---

Subject:	RE: Zenoss Core URL login
Author:	Jane Curry
Posted:	2017-10-06 05:21

Are those all your queues?  Not nearly enough. I have:
[root@zen42 jane]# rabbitmqctl -p /zenoss list_queues
Listing queues ...
celery 0
zenoss.queues.zep.signal 0
zenoss.queues.zep.modelchange 0
zenoss.queues.zep.migrated.summary 0
zenoss.queues.zep.rawevents 0
zenoss.queues.zep.heartbeats 0
eventForwarder 0
zenoss.queues.zep.zenevents 0
zen42.class.example.org.celeryd.pidbox 0
zenoss.queues.zep.migrated.archive 0
...done.

Any idea why that amqppassword entry in global.conf was changed???  I might be tempted to change it back to zenoss for now.  Also check whether other entries in global.conf have changed.  Here is mine from my test system , which I believe is completely standard:

[root@zen42 jane]# cat /opt/zenoss/etc/global.conf
zodb-db-type mysql
zodb-host localhost
zodb-port 3306
zodb-db zodb
zodb-user zenoss
zodb-password zenoss
zodb-admin-user root
zodb-admin-password
zodb-cacheservers 127.0.0.1:11211
zodb-cachesize 1000

amqphost localhost
amqpport 5672
amqpvhost /zenoss
amqpuser zenoss
amqppassword zenoss
amqpusessl 0
amqpadminport 55672
amqpadminusessl 0

zep-db-type mysql
zep-host localhost
zep-port 3306
zep-db zenoss_zep
zep-user zenoss
zep-password zenoss
zep-admin-user root
zep-admin-password
zep-uri http://localhost:8084

Having checked that lot and changed amqppassword back to zenoss, I would, once more, go through the "fix procedure" to recreate the queues, exactly as given in the admin manual, then stop Zenoss entirely - make sure that all the processes really have stopped - and then restart.

Cheers,
Jane


------------------------------
Jane Curry
Skills 1st United Kingdom
jane.curry@skills-1st.co.uk
------------------------------

---

Subject:	RE: Zenoss Core URL login
Author:	Ted Mottinger
Posted:	2017-10-12 08:35

So, I've run the fix process and been through this a few times and still only have 4 queues.  At this point is this install unfixable?  Do I need to completely start over?

[root@zenoss etc]# rabbitmqctl -p /zenoss list_queues
Listing queues ...
celery 0
zenoss.queues.zep.signal 0
zenoss.queues.zep.rawevents 0
zenoss.gmturner.com.celeryd.pidbox 0
...done.
[root@zenoss etc]#

------------------------------
Ted Mottinger
Network Engineer
Guy M Turner Inc.
Greensboro NC
3368099233
------------------------------

---

Subject:	RE: Zenoss Core URL login
Author:	Jane Curry
Posted:	2017-10-12 16:32

Four queues just ain't enough.  Don't know what else to suggest here.  Weird scenario!

Sorry,
Jane

------------------------------
Jane Curry
Skills 1st United Kingdom
jane.curry@skills-1st.co.uk
------------------------------

---

Subject:	RE: Zenoss Core URL login
Author:	Ted Mottinger
Posted:	2017-10-19 09:19

Hi Jane, I found a previous export of our Zenoss VM and installed it.  I now see all nine queues but I don't know what user to login with.  Again, because I did not set this up, is there a way to tell or to create a user that can login to the Zenoss Core webpage?



------------------------------
Ted Mottinger
Network Engineer
Guy M Turner Inc.
Greensboro NC
3368099233
------------------------------

---

Subject:	RE: Zenoss Core URL login
Author:	Jane Curry
Posted:	2017-10-20 03:31

Hi Ted,
That looks like a MUCH better bunch of queues!  Now you can go back to the start of this forum append and use the zpasswd utility to change passwords for Zenoss GUI users.  Unless someone has done something really weird to your environment, the admin user should exist as a minimum.  Once you can login with that, you can then go ADVANCED -> Settings -> Users and see what other users you have.  You can then use the GUI to change a password for a user.

Cheers,
Jane

------------------------------
Jane Curry
Skills 1st United Kingdom
jane.curry@skills-1st.co.uk
------------------------------

---

Subject:	RE: Zenoss Core URL login
Author:	Ted Mottinger
Posted:	2017-10-20 09:07

That did it!!!!  Thanks so much Jane!

------------------------------
Ted Mottinger
Network Engineer
Guy M Turner Inc.
Greensboro NC
3368099233
------------------------------