TECHZEN Zenoss User Community ARCHIVE  

Email Alert/Notification not working

Subject: Email Alert/Notification not working
Author: Marius
Posted: 2017-11-01 04:15

Hello Team,

I have installed Fresh Zenoss 4.2.4 version on new Ubuntu 14.04 system. i have copied all the configuration fro my previous zenoss monitoring system which is again the same version.

Everything working smooth except Email alert notification, i have checked everything in configuration point of view but unable to identify the issue.

I have configure SMTP settings using SMTP auth & TLS, while i am checking email server logs i dont see any logs from zenoss server also i am unable to find any errors in zenactiond.log file. below are the logs i am observing in zenactiond.log file.

2017-11-01 09:04:41,188 INFO zen.maintenance: Performing periodic maintenance
2017-11-01 09:05:41,193 INFO zen.maintenance: Performing periodic maintenance
2017-11-01 09:06:41,200 INFO zen.maintenance: Performing periodic maintenance
2017-11-01 09:07:41,204 INFO zen.maintenance: Performing periodic maintenance
2017-11-01 09:08:41,209 INFO zen.maintenance: Performing periodic maintenance
2017-11-01 09:09:41,215 INFO zen.maintenance: Performing periodic maintenance
2017-11-01 09:10:41,222 INFO zen.maintenance: Performing periodic maintenance
2017-11-01 09:11:41,227 INFO zen.maintenance: Performing periodic maintenance
2017-11-01 09:12:41,233 INFO zen.maintenance: Performing periodic maintenance
2017-11-01 09:13:41,240 INFO zen.maintenance: Performing periodic maintenance

In my old zenoss everything working fine with the same configuration which i did in new zenoss.

can you please help me to identify the issue and resolve it.

Regards,
Marius Johansen


Subject: RE: Email Alert/Notification not working
Author: Jane Curry
Posted: 2017-11-01 05:47

Hi Marius,
I would just question why you have built a new Zenoss 4.2.4?  That is very old and there have probably been hundreds of bug fixes if you move up to Zenoss 4.2.5 with SUP 732.  Maybe once you get this stable, it would be a good idea to upgrade before making your new system live - it shouldn't be a huge deal.

A few things you might look at re your email notifications...
Not clear from your post whether you are seeing anything in your EMAIL server logs?  Do you know whether anything is reaching your email server?  One scenario we have is that our email server will not accept unknown / illegal fields in the email "from" field.  In Zenoss ADVANCED -> Settings, there is a field "From Address for emails"; make sure that is filled in and legal - compare it with your old system.

Have you tested the email address for the user specified by the notification (from ADVANCED -Settings -> Users - there should be a test button beside the email.  This does NOT guarantee that an email notification will work, but if the test doesn't then the email won't.

Your notification also relies on a trigger.  Check both to ensure it really does have what you want.  I find the GUI for triggers and notifications a bit perverse.  When you choose a trigger for a notification, you do need to click the Add button, not just select it; similarly with adding users for a notification.

Turn up logging on zenactiond.  By default, you will just see those messages you reported.  As the zenoss user in a command window, type:
zenactiond debug

That should set the logging level to debug (value 10) on the running zenactiond.  The debug flag is a toggle - another "zenactiond debug" will set the log level back to normal.  You should then see LOTS of chat analysing triggers and notifications for suitability for any event.  This will show whether your trigger is actually being activated.

Cheers,
Jane

------------------------------
Jane Curry
Skills 1st United Kingdom
jane.curry@skills-1st.co.uk
------------------------------


Subject: RE: Email Alert/Notification not working
Author: Marius
Posted: 2017-11-03 06:22

Hi Jane,

Thanks for replying on my query. please find my answer on each thing you asked me to check highlighted in blue.

I would just question why you have built a new Zenoss 4.2.4?  That is very old and there have probably been hundreds of bug fixes if you move up to Zenoss 4.2.5 with SUP 732.  Maybe once you get this stable, it would be a good idea to upgrade before making your new system live - it shouldn't be a huge deal
Comment : Can you help me to upgrade version from 4.2.4 to 4.2.5. i unable to get that version to install.


A few things you might look at re your email notifications...
Not clear from your post whether you are seeing anything in your EMAIL server logs?  Do you know whether anything is reaching your email server?  One scenario we have is that our email server will not accept unknown / illegal fields in the email "from" field.  In Zenoss ADVANCED -> Settings, there is a field "From Address for emails"; make sure that is filled in and legal - compare it with your old system.
Comment : Yes Form address configured in the zenoss is valid address with Authentication configured for relay on our mail server.

Have you tested the email address for the user specified by the notification (from ADVANCED -Settings -> Users - there should be a test button beside the email.  This does NOT guarantee that an email notification will work, but if the test doesn't then the email won't.
Comment : Yes, I tested and when i click on test mail i am able to receive mail from zenoss.

Your notification also relies on a trigger.  Check both to ensure it really does have what you want.  I find the GUI for triggers and notifications a bit perverse.  When you choose a trigger for a notification, you do need to click the Add button, not just select it; similarly with adding users for a notification.
Comment : I have OLD zenoss server which is on the same version, i configured triggers with same configuration which i have on OLD zenoss so there is no issue with the trigger configuration. i checked it so many times but i dont see any issue in triggers configuration. can you let me know if you can me to check some advanced parameters which i might not aware of.

Turn up logging on zenactiond.  By default, you will just see those messages you reported.  As the zenoss user in a command window, type:
zenactiond debug
Comment : I turned on and i dont see any issue in the logs. even logs dont help me with any suspicious logs. please find logs below for your reference.
LOG:
2017-11-03 11:18:01,336 DEBUG zen.zenoss.protocols.amqp: Publishing with routing key zenoss.heartbeat.localhost to exchange zenoss.heartbeats
2017-11-03 11:18:01,336 DEBUG zen.maintenance: Rescheduling maintenance in 60s
2017-11-03 11:19:01,342 INFO zen.maintenance: Performing periodic maintenance
2017-11-03 11:19:01,342 DEBUG zen.maintenance: calling hearbeat sender
2017-11-03 11:19:01,343 DEBUG zen.maintenance: sending heartbeat monitor: "localhost"
daemon: "zenactiond"
timeout_seconds: 180


That should set the logging level to debug (value 10) on the running zenactiond.  The debug flag is a toggle - another "zenactiond debug" will set the log level back to normal.  You should then see LOTS of chat analysing triggers and notifications for suitability for any event.  This will show whether your trigger is actually being activated.
Comment: below are the logs,  i dont find anything which help me to troubleshoot the issue.
2017-11-03 11:18:01,336 DEBUG zen.zenoss.protocols.amqp: Publishing with routing key zenoss.heartbeat.localhost to exchange zenoss.heartbeats
2017-11-03 11:18:01,336 DEBUG zen.maintenance: Rescheduling maintenance in 60s
2017-11-03 11:19:01,342 INFO zen.maintenance: Performing periodic maintenance
2017-11-03 11:19:01,342 DEBUG zen.maintenance: calling hearbeat sender
2017-11-03 11:19:01,343 DEBUG zen.maintenance: sending heartbeat monitor: "localhost"
daemon: "zenactiond"
timeout_seconds: 180

Can you please help me what other thing i can check to correct the issue.

Regards,
Marius



------------------------------
Marius
------------------------------


Subject: RE: Email Alert/Notification not working
Author: Jane Curry
Posted: 2017-11-03 07:39

We need to look more closely at the zenactiond logs.  The "zenactiond debug" command is a toggle - the first one turns logging to debug, a second "zenactiond debug" sets logging back to normal.  This is on the running zenactiond daemon.  The other alternative is to to set debug-level logging in the zenactiond.conf file in $ZENHOME/etc :

logseverity 10

With this, you then need to restart zenactiond.

You should see LOTS of chat with logseverity at 10.  Each notification you have should be tested to see whether it is valid for this event.  Here is a snippet from my zenactiond.log when Zenoss could not access our mailserver:


2017-11-03 11:26:55,016 DEBUG zen.notificationdao: Notification "<NotificationSubscription at Important_File_Dir_Email>" does not subscribe to this signal.
2017-11-03 11:26:55,016 DEBUG zen.notifications: Notification is enabled: ping_trap
2017-11-03 11:26:55,016 DEBUG zen.notifications: Notification is enabled, but has no windows, it is active.
2017-11-03 11:26:55,017 DEBUG zen.notificationdao: Notification "<NotificationSubscription at ping_trap>" does not subscribe to this signal.
2017-11-03 11:26:55,017 DEBUG zen.zenactiond: Found these matching notifications: [<NotificationSubscription at /zport/dmd/NotificationSubscriptions/zen42_email_traps_su>]
2017-11-03 11:26:55,021 DEBUG zen.actions: Executing batch action for targets.
2017-11-03 11:26:55,021 DEBUG zen.actions: Executing Email action for targets: set(['jane.curry@skills-1st.co.uk'])
2017-11-03 11:26:55,025 DEBUG zen.actions: Sending this subject: [zenoss] zen42.class.example.org pam_unix(su:auth): authentication failure; logname=jane uid=1337 euid=0 tty=pts/12 ruser=zenoss rhost= user=root reported by zensyslog
2017-11-03 11:26:55,026 DEBUG zen.actions: Sending this body:
Device: zen42.class.example.org
Component: su
Severity: 5
Time: 2017/11/03 11:26:50.000
Message: This is a test -----------
pam_unix(su:auth): authentication failure; logname=jane uid=1337 euid=0 tty=pts/12 ruser=zenoss rhost= user=root
<a tal:attributes="href urls/eventUrl">Event Detail</a>
<a href="http://zen42.class.example.org:8080/zport/dmd/Events/viewDetail?evid=000c29b5-8a24-b840-11e7-c089e55624d7">Event Detail</a>
<a href="http://zen42.class.example.org:8080/zport/dmd/Events/manage_ackEvents?evids=000c29b5-8a24-b840-11e7-c089e55624d7&zenScreenName=viewEvents">Acknowledge</a>
<a href="http://zen42.class.example.org:8080/zport/dmd/Events/manage_deleteEvents?evids=000c29b5-8a24-b840-11e7-c089e55624d7&zenScreenName=viewHistoryEvents">Close</a>
<a href="http://zen42.class.example.org:8080/zport/dmd/Devices/Server/Linux/devices/zen42.class.example.org/devicedetail#deviceDetailNav:device_events">Device Events</a>

2017-11-03 11:26:55,926 ERROR zen.actions: Notification 'zen42_email_traps_su' FAILED to send emails to set(['jane.curry@skills-1st.co.uk']): <class 'socket.gaierror'> - [Errno -2] Name or service not known
2017-11-03 11:26:55,926 ERROR zen.actions: Error executing action zen42_email_traps_su on set(['jane.curry@skills-1st.co.uk'])
2017-11-03 11:26:55,927 DEBUG zen.Events: =============== incoming event ===============
2017-11-03 11:26:55,928 DEBUG zen.queuepublisher: About to publish this event to the raw event queue:uuid: "e5f66ddc-c089-11e7-8c58-000c29b58a24"
created_time: 1509708415926
event_class: "/App/Failed"
actor {
element_type_id: DEVICE
element_identifier: "localhost"
element_sub_type_id: COMPONENT
element_sub_identifier: "zenactiond"
}
summary: "Error executing action zen42_email_traps_su on set([\'jane.curry@skills-1st.co.uk\'])"

You can see it deciding that zen42_email_traps_su is a valid notification (after discarding a couple of other notifications), finding the targets ( ['jane.curry@skills-1st.co.uk'] ), constructing the email subject and body, and then the error ( [Errno -2] Name or service not known.. ), when it couldn't find our mailserver.

The little bit of log you showed didn't look like you were really in debug mode.  When you are drowned in logging stuff, try searching for ERROR for a start.

Cheers,
Jane

------------------------------
Jane Curry
Skills 1st United Kingdom
jane.curry@skills-1st.co.uk
------------------------------


Subject: RE: Email Alert/Notification not working
Author: Marius
Posted: 2017-11-03 08:23

Hi Jane,

I changed severity of zenactiond logs to 10 but i still dont see any logs except below.

2017-11-03 13:20:44,914 DEBUG zen.zenoss.protocols.amqp: Publishing with routing key zenoss.heartbeat.localhost to exchange zenoss.heartbeats
2017-11-03 13:20:44,915 DEBUG zen.maintenance: Rescheduling maintenance in 60s
2017-11-03 13:21:44,915 INFO zen.maintenance: Performing periodic maintenance
2017-11-03 13:21:44,915 DEBUG zen.maintenance: calling hearbeat sender
2017-11-03 13:21:44,916 DEBUG zen.maintenance: sending heartbeat monitor: "localhost"
daemon: "zenactiond"
timeout_seconds: 180

Any suggestions.

NOTE: i previously had severity set to 20 in zenactiond.conf file.

Regards,
Marius

------------------------------
Marius
------------------------------


Subject: RE: Email Alert/Notification not working
Author: Jane Curry
Posted: 2017-11-07 06:04

Hmm - that's weird. I am beginning to wonder if you have a bug that is part of 4.2.4 (I am running 4.2.5 with SUP 671.

Are you running Core or Enterprise?
How many notifications do you have?

When I have normal debugging level (logseverity = 20) and then do a:
zenactiond debug

I get lines in zenactiond.log like:

2017-11-07 10:21:30,955 INFO zen.maintenance: Performing periodic maintenance
2017-11-07 10:22:29,810 INFO zen: Setting logging level to DEBUG
2017-11-07 10:22:29,811 WARNING SignalHandler: Threads traceback dump at 2017-11-07 10:22:29

Thread 140182879917824:
File "/opt/zenoss/Products/ZenEvents/zenactiond.py", line 263, in <module>
zad.run()
File "/opt/zenoss/Products/ZenEvents/zenactiond.py", line 236, in run
reactor.run()
File "/opt/zenoss/lib/python/twisted/internet/base.py", line 1162, in run
self.mainLoop()
File "/opt/zenoss/lib/python/twisted/internet/base.py", line 1174, in mainLoop
self.doIteration(t)
File "/opt/zenoss/lib/python/twisted/internet/selectreactor.py", line 104, in doSelect
[], timeout)

End of dump
2017-11-07 10:22:30,960 DEBUG zen.Schedule: Never going to run Maintenance Window test6 for zen42.class.example.org again
2017-11-07 10:22:30,960 DEBUG zen.Schedule: Never going to run Maintenance Window Off_net for Off_Internet again
2017-11-07 10:22:30,960 DEBUG zen.Schedule: Never going to run Maintenance Window Scheduled_Maintenance for VMs again
2017-11-07 10:22:30,961 DEBUG zen.Schedule: Waiting 31049.039962 seconds
2017-11-07 10:22:30,961 INFO zen.maintenance: Performing periodic maintenance
2017-11-07 10:22:30,961 DEBUG zen.maintenance: calling hearbeat sender
2017-11-07 10:22:30,961 DEBUG zen.maintenance: sending heartbeat monitor: "localhost"
daemon: "zenactiond"
timeout_seconds: 180

If you are not seeing any of this then I suspect it is an artifact of your old Zenoss.  Not quite sure what to suggest if you cannot turn up the debugging by any method.

Cheers,
Jane

------------------------------
Jane Curry
Skills 1st United Kingdom
jane.curry@skills-1st.co.uk
------------------------------


Subject: RE: Email Alert/Notification not working
Author: Marius
Posted: 2017-11-07 06:52

Hi Jane,

Can you please help me to know how can i upgrade my zenoss core version (not enterprise) to latest version.

so i can check by upgrading version if that resolve my issue.

Regards,
Marius

------------------------------
Marius
------------------------------


Subject: RE: Email Alert/Notification not working
Author: Jane Curry
Posted: 2017-11-08 05:21

You said you had trouble doing the upgrade to 4.2.5??  What was the problem?

Do you have the Zenoss 4.2.5 Installation & Upgrade Guide - it does document upgrade from 4.2.4 as supported (but it only documents RedHat / CentOS and I think you are Ubuntu?  If you can't find this manual, let me know and I can send it to you if you provide email.

The other reference you might use is the auto-deploy work that I did 12 months ago which did do a build for Ubuntu - see https://github.com/jcurry/Zenoss_4.2.5_core-autodeploy/tree/ubuntu  (make sure you are on the Ubuntu branch, not the master branch).  This work is for a new build but might help you do an upgrade.

Cheers,
Jane

------------------------------
Jane Curry
Skills 1st United Kingdom
jane.curry@skills-1st.co.uk
------------------------------


Subject: RE: Email Alert/Notification not working
Author: Marius
Posted: 2017-11-14 08:28

Hi Jane,

I tried to update Zenoss to version 4.2.5 but i am unable to do that as multiple download links given in the script not working.

Can you help me with the other version script which can help me to upgrade zenoss on some other version from 4.2.4 to latest one on Ubuntu 14.04. I tried to find but i am unable to get it.

Regards,
Marius

------------------------------
Marius
------------------------------


Subject: RE: Email Alert/Notification not working
Author: Marius
Posted: 2017-11-21 12:19

Hi Jane,

Can you please help. I am still unable to get zenoss alert.

Regards,
Marius

------------------------------
Marius
------------------------------


< Previous
Rabbitmq queues backingup
  Next
Python eror
>