| Subject: |
RE: Zenoss 4.2.5, Notification send three traps instead two traps. |
| Author: |
Jane Curry |
| Posted: |
2017-09-13 04:11 |
Not sure I quit understand your scenario here....
You have 2 Zenoss Servers? An "old" one that works as expected, and a "new one" that doesn't???
Are you then trying to send traps on from one Zenoss to another? Using the Zenoss TRAP Notification method?
You take down your switch and see a Zenoss event for ping down - do you then have a trigger and notification that forwards a TRAP?? And this works OK?
You bring your switch back up, see the Node Up event, and generate another TRAP via Zenoss triggers and notifications, sending a Zenoss TRAP 3?? (why 3?? I thought the Zenoss MIB only has 1 TRAP?)
Your second "Status ping device is down". Is this an event in the Zenoss Event Console? Is the device actually up or down? From your description, I assume it is actually up? From the Zenoss GUI, can you navigate to the Catalyst device and then use the Command menu (bottom-left) to do a ping - and does it work? On that device Overview panel, at the top, is the DEVICE STATUS Up or Down?
When you "disable" your Cat device, do you actually take it down or just disable an interface?
Is your Cat sending TRAPs to the "first" Zenoss or are you just relying on Zenoss's ping-polling to see these issues and then generating TRAPs from there to forward to your second Zenoss?
Bit unclear about your "other system" that also thinks the device is down? Is there any chance that there is something cyclical going on where several manager systems are passing around the "ping down" TRAP??
I guess the key question is whether your "first" Zenoss really receives a second event in the console saying that the Cat is down? If it does, then obviously it will generate a second trap if it obeys the trigger and notification rule.
Cheers,
Jane
------------------------------
Jane Curry
Skills 1st United Kingdom
jane.curry@skills-1st.co.uk
------------------------------
| Subject: |
RE: Zenoss 4.2.5, Notification send three traps instead two traps. |
| Author: |
Marek Milik |
| Posted: |
2017-09-13 09:30 |
Thank you for your answer.
I have one server Zenoss, and one device Catalyst 3550 added (for test).
I configured one triger:
UP/DOWN,
Rule:
Device Production state equals Production
Event class contains /Status/Ping
and one Notification:
Action: SNMP trap,
is enable checkbox: Enabled, Send clear, Send only on initial occurrence,
notification contains trigger: UP/DOWN.
Content:
SNMP Trap destination: 192.168.21.22 (my second server receiving SNMP traps, for test it is my computer with Wireshark).
SNMP community: public
SNMP version: v2c
port 162.
When I turn off the power of the switch, i see in Zenoss Event console "SW_3550 is down", and when I turn on the switch, in Zenoss Event console is "SW_3550 is UP" - there is OK.
But in Wireshark I capture three traps like on previous screenshot:
UDP stream :
first trap: (SW_3550||/Status/Ping|5|SW_3550 is DOWN!0..
second trap: &SW_3550||/Status/Ping|0|SW_3550 is UP!0..
third tarp: (SW_3550||/Status/Ping|5|SW_3550 is DOWN!0..
My question: Is it normal situation when a trap is forwarded?
------------------------------
Marek Milik
Kom
Olsztyn
------------------------------
| Subject: |
RE: Zenoss 4.2.5, Notification send three traps instead two traps. |
| Author: |
Jane Curry |
| Posted: |
2017-09-13 14:36 |
Hmm.
On triggers, I always also include Status = new and usually include a check for severity. but I doubt that is causing your confusion. You are saying that you get 2 Zenoss events and see three traps?
It looks like your traps are including the dedup (de-duplication) field from the event so the 5 in the middle of the first one is severity of Critical and the 0 in the second one is severity of Clear so it looks like the third one is another down / critical event.
You may want to refine your trigger to send different traps for Critical and Clear events? Try adding Status=New to your trigger.
In your event console, set your filters so you can see all event severities, including clear and make sure the status filter shows closed and cleared events in addition to new. Check the counts, especially against the down events. From what you are saying, the counts should be 1 and the ping up event should automatically clear the ping down event.
Cheers,
Jane
------------------------------
Jane Curry
Skills 1st United Kingdom
jane.curry@skills-1st.co.uk
------------------------------
| Subject: |
RE: Zenoss 4.2.5, Notification send three traps instead two traps. |
| Author: |
Jane Curry |
| Posted: |
2017-09-14 08:32 |
OK! The problem is that in your trigger, your are just triggering on the Prod State of the device and the event class of /Status/Ping - nothing to do with severity, so notifications are generated for the node down and the node up.
Your notification also has "Send Clear" checked. This means that a clearing trap will be sent automatically, in addition to the trap sent by the Node Up event - so at the trap receiving end, you get one down trap and 2 up traps. If you look closely into the varbinds on the trap, they do differ. For example, check:
1.3.6.1.4.1.14296.1.100.9 for severity (5 = Critical on node down AND on the automatic clearing event; 0 on the node up event)
1.3.6.1.4.1.14296.1.100.10 for state ( 0 = New, 4 = Cleared)
Cheers,
Jane
------------------------------
Jane Curry
Skills 1st United Kingdom
jane.curry@skills-1st.co.uk
------------------------------
| Subject: |
RE: Zenoss 4.2.5, Notification send three traps instead two traps. |
| Author: |
Marek Milik |
| Posted: |
2017-09-15 02:41 |
Thank you for information Jane, but if I don't mark checkbox "Send Clear" in my notification "Send to Wireshark", I don't receive trap with info "Device_UP". Then I receive only one trap "Device_Down".
How I can create trigger and notification that will only send trap "Device_Down" and "Device_Up" without third trap "Device_Down"? Is it possibly?
------------------------------
Marek Milik
Kom
Olsztyn
------------------------------
| Subject: |
RE: Zenoss 4.2.5, Notification send three traps instead two traps. |
| Author: |
Marek Milik |
| Posted: |
2017-09-15 03:59 |
It looks like the notification could not send trap "Device_UP" without "Send Clear".
------------------------------
Marek Milik
Kom
Olsztyn
------------------------------
| Subject: |
RE: Zenoss 4.2.5, Notification send three traps instead two traps. |
| Author: |
Jane Curry |
| Posted: |
2017-09-15 06:11 |
Bah!! You are quite correct - disable the Send Clear on the Notification and it doesn't send either "clearing" event. I can argue that that is a bug and that it should send the Node Up event - but life's too short to argue such bugs with Zenoss.
I would write a transform to test the varbind 9 (severity) and drop one of the clearing events. If you want some help with that, pull my event management paper from here -
https://www.skills-1st.co.uk/papers/jane/zenoss4-events/ .
Cheers,
jane
------------------------------
Jane Curry
Skills 1st United Kingdom
jane.curry@skills-1st.co.uk
------------------------------
