TECHZEN Zenoss User Community ARCHIVE  

Zenoss 4.2.5 Zenup 732 - Production State problem

Subject: Zenoss 4.2.5 Zenup 732 - Production State problem
Author: Pheripheral Pheripheral
Posted: 2018-05-09 12:30

A change in production state does not take effect in regards to monitoring until a daemon restart?

I've recently noticed some funny behaviour regarding the production states and how they affect monitoring of devices after moving to Zenoss 4.2.5 Zenup SUP 732

 

Previously, on Zenoss 425 SUP 671  if a device is set to decommissioned (i.e. -1, or any production state value less than its zProdStateThreshold value – which defaults to 300 for devices) it would no longer monitor the device via ping or by snmp and no further events would be raised until the production state was set back to something above the zProdStateThreshold.

 

However, I've noticed on SUP732 that this is no longer the case and setting the production state lower than the zProdStateThreshold does not prevent monitoring or events being raised.

With further experiments I discovered that once prod state is lower than the threshold, restarting the zenping will stop ping monitoring and restarting the zenperfsnmp daemons will stop snmp monitoring.

 

This behaviour is also similar once re-raising the production state whilst monitoring is off. Until the ping and snmp daemons are restarted monitoring for each protocol does not begin again…

 

Is this a bug that has crept in during the 732 update? – I noticed that 732 has added a whole set of things around production state under zenutils and introduced a ProdStateManager.

 

Has anyone else noticed this behaviour or are they able to recreate it to confirm my findings?

 

And any ideas of how to rectify it so that reducing the production state below the zProdStateThreshold once again stops monitoring as it is supposed to?



------------------------------
Thanks
Pheripheral

------------------------------


Subject: RE: Zenoss 4.2.5 Zenup 732 - Production State problem
Author: Jane Curry
Posted: 2018-05-16 10:18

I have certainly seen glitches with SUP732 if a device does not have its productionState attribute set and needed to manually fix a few devices, but I haven't seen the problems you describe. 

You might do a little digging around with zendmd to see what is going on; for example:


>>> for d in dmd.Devices.getSubDevices():
            if d.getProductionState() < 1000:
                    print d.id, d.getProductionState()

There is also a setProdState method, eg:
d.setProdState(300)

Cheers,
Jane

------------------------------
Jane Curry
Skills 1st United Kingdom
jane.curry@skills-1st.co.uk
------------------------------


Subject: RE: Zenoss 4.2.5 Zenup 732 - Production State problem
Author: Pheripheral Pheripheral
Posted: 2018-05-17 11:54

I've taken a look at things in zendmd and this seems to only confirm the behaviour is not working on my zenoss with 732 installed

I can see in zendmd that the production state is set to -1 (Decommisioned) and yet i'm still getting events raised on my device.
Once I restart the zenping / zenperfsnmp daemons the  events stop.

Setting the production state back to 1000 (Production) similarly does not restart events until the daemon(s) are restarted.

I can also produce this behaviour when setting the production states via zendmd rather than in the gui:

for d in dmd.Devices.getSubDevices():
    d.setProdState(-1)
    commit()

again monitoring does not cease until daemon restart unfortunately.

I've compared this with an instance of zenoss I have with SUP 671 installed and the production state changes operates as it should, with monitoring ceasing when state is changed without needing daemon restarts...

I may have to roll back to Sup 671 if i can't seem to shake this strange behaviour in the 732 instance.


------------------------------
Pheripheral Pheripheral
------------------------------


Subject: RE: Zenoss 4.2.5 Zenup 732 - Production State problem
Author: Jane Curry
Posted: 2018-05-18 04:07

Sounds like you have found a real issue. Have you logged a JIRA ticket for this?
Cheers,
Jane

------------------------------
Jane Curry
Skills 1st United Kingdom
jane.curry@skills-1st.co.uk
------------------------------


Subject: RE: Zenoss 4.2.5 Zenup 732 - Production State problem
Author: Pheripheral Pheripheral
Posted: 2018-05-21 06:45

Hi,

Jira ticket raised:
ZEN-30167 - Zenoss Core 4.2.5 with SUP 732 - Production State < zProdThreshold does not prevent monitoring 
(although it doens't seem to be viewable by the public)

I recently create a new build of zenoss but only installed SUP 671 and the production state performs as expected without needing to restart daemons to stop / start monitoring when setting things to decommissioned / Production.
I then upgraded to SUP 732 and the production state ceased to perform as expected, so i'm reasonably convinced that this is a problem introduced in 732...

------------------------------
Pheripheral Pheripheral
------------------------------

A change in production state does not take effect in regards to monitoring until a daemon restart?

I've recently noticed some funny behaviour regarding the production states and how they affect monitoring of devices after moving to Zenoss 4.2.5 Zenup SUP 732

 

Previously, on Zenoss 425 SUP 671  if a device is set to decommissioned (i.e. -1, or any production state value less than its zProdStateThreshold value – which defaults to 300 for devices) it would no longer monitor the device via ping or by snmp and no further events would be raised until the production state was set back to something above the zProdStateThreshold.

 

However, I've noticed on SUP732 that this is no longer the case and setting the production state lower than the zProdStateThreshold does not prevent monitoring or events being raised.

With further experiments I discovered that once prod state is lower than the threshold, restarting the zenping will stop ping monitoring and restarting the zenperfsnmp daemons will stop snmp monitoring.

 

This behaviour is also similar once re-raising the production state whilst monitoring is off. Until the ping and snmp daemons are restarted monitoring for each protocol does not begin again…

 

Is this a bug that has crept in during the 732 update? – I noticed that 732 has added a whole set of things around production state under zenutils and introduced a ProdStateManager.

 

Has anyone else noticed this behaviour or are they able to recreate it to confirm my findings?

 

And any ideas of how to rectify it so that reducing the production state below the zProdStateThreshold once again stops monitoring as it is supposed to?



------------------------------
Thanks
Pheripheral

------------------------------


Subject: RE: Zenoss 4.2.5 Zenup 732 - Production State problem
Author: Jane Curry
Posted: 2018-05-22 03:55

Yup - that is the correct Jira.  Also copy John Boyle -  @John -, the Community manager; otherwise it will end up as a private ticket that no-one else can see.  Please could you post the ticket number here when you have it?

Thanks for this - I will try and find time to confirm I see the same thing.

Cheers,
Jane​

------------------------------
Jane Curry
Skills 1st United Kingdom
jane.curry@skills-1st.co.uk
------------------------------


Subject: RE: Zenoss 4.2.5 Zenup 732 - Production State problem
Author: Pheripheral Pheripheral
Posted: 2018-05-22 06:48

Ah oops , I updated my above post before spotting your reply...

Yes I've raised a ticket - ZEN-30167


------------------------------
Pheripheral Pheripheral
------------------------------


Subject: RE: Zenoss 4.2.5 Zenup 732 - Production State problem
Author: John Boyle
Posted: 2018-05-22 08:50

Thanks for sharing the ticket. I updated the status so it is accessible to you. Cheers! - JB

------------------------------
John Boyle
Technology Partnerships & Community Manager
Zenoss
Austin TX
jboyle@zenoss.com
http://www.zenoss.com
------------------------------

Subject: RE: Zenoss 4.2.5 Zenup 732 - Production State problem
Author: Jane Curry
Posted: 2018-05-23 04:29

The link to this ticket is  https://jira.zenoss.com/browse/ZEN-30167 (and it is public - thanks John).
Cheers,
Jane

------------------------------
Jane Curry
Skills 1st United Kingdom
jane.curry@skills-1st.co.uk
------------------------------


Subject: RE: Zenoss 4.2.5 Zenup 732 - Production State problem
Author: Thomas Luther
Posted: 2018-05-31 02:40

Hi

I have done some testing on this issue (as I needed to upgrade to latest RPS to support OpenSSH 7.x), and it appears that if you change the productionstate on a device, then you need to issue a "Push Changes" before it takes effect. 

So I guess the problem is that the ProductionState change functionality has been changed and now it just doesnt push the changes to collectors

Thomas

------------------------------
Thomas Luther
Senior Consultant
Netic A/S
------------------------------


Subject: RE: Zenoss 4.2.5 Zenup 732 - Production State problem
Author: Willy Hidayat
Posted: 2018-06-05 01:14

Maybe we should create telegram group for more discussing about 
1. How to install zenoss on Ubuntu, CentOS, etc
2. Configuration and the other things that belongs about zenoss core


Subject: RE: Zenoss 4.2.5 Zenup 732 - Production State problem
Author: Jane Curry
Posted: 2018-06-06 05:48

Hi Willy,
Why do you suggest a new forum for discussing installation and configuration of Zenoss?  Isn't that what this forum is for?

Cheers,
Jane

------------------------------
Jane Curry
Skills 1st United Kingdom
jane.curry@skills-1st.co.uk
------------------------------


Subject: RE: Zenoss 4.2.5 Zenup 732 - Production State problem
Author: Jay Stanley
Posted: 2018-07-05 07:12

I created a patch for this, it seems to work in my lab, but if anyone else wants to test I can post the patch file.

------------------------------
jstanley
------------------------------


Subject: RE: Zenoss 4.2.5 Zenup 732 - Production State problem
Author: Pheripheral Pheripheral
Posted: 2018-07-05 12:42

Yes, i'd be very interested in seeing the patch!

I did some quick playing around with this adding an extra line or two to push the changes in my custom bit of code i was using to toggle between 2  production state via one custom button press, and i think it seems to work once i added the line device._p_changed = True which seems to be what pushes it to the collectors. This may be the wrong way to do it though, and ti certianly won't fix the normal prod state change functionality.

        prodState = device.productionState
        if prodState == self.PRODSTATE1:        

device.setProdState(self.PRODSTATE2)    
device._p_changed = True                 
successFlag = True





------------------------------
Pheripheral Pheripheral
------------------------------
Maybe we should create telegram group for more discussing about 
1. How to install zenoss on Ubuntu, CentOS, etc
2. Configuration and the other things that belongs about zenoss core



Original Message------

Hi

I have done some testing on this issue (as I needed to upgrade to latest RPS to support OpenSSH 7.x), and it appears that if you change the productionstate on a device, then you need to issue a "Push Changes" before it takes effect.

So I guess the problem is that the ProductionState change functionality has been changed and now it just doesnt push the changes to collectors

Thomas

------------------------------
Thomas Luther
Senior Consultant
Netic A/S
------------------------------


Subject: RE: Zenoss 4.2.5 Zenup 732 - Production State problem
Author: Jay Stanley
Posted: 2018-07-05 13:15

Yup, you were on the right track. Just a one line edit.

drop this in your $ZENHOME, then
patch -p0 < ZEN-30167.patch

restart zenoss
(push to remote hubs and collectors)

------------------------------
jstanley
------------------------------

Attachments:

ZEN-30167.patch.txt



Subject: RE: Zenoss 4.2.5 Zenup 732 - Production State problem
Author: Thomas Luther
Posted: 2018-07-06 04:52

Fantastic... I have just tested it on our development zenoss 4.2.5 with SP732 applied... and now when I change productionState on a device from Production (1000) to Decomissioned (-1) then after a short period, the checks stops... and when changing back to any higher productionState (300 + ) then checks starts coming in again after a short periode...

So thanks a lot for this very simple fix... I have added a comment to the JIRA https://jira.zenoss.com/browse/ZEN-30167

------------------------------
Thomas Luther
Senior Consultant
Netic A/S
------------------------------

Subject: RE: Zenoss 4.2.5 Zenup 732 - Production State problem
Author: Jane Curry
Posted: 2018-07-06 10:56

You fantastic fellow! Thanks Jay.

------------------------------
Jane Curry
Skills 1st United Kingdom
jane.curry@skills-1st.co.uk
------------------------------


< Previous
Zenoss 4.2.5 job_catalog corrupted and POSKeyError
  Next
Close events at change to decommision status
>