TECHZEN Zenoss User Community ARCHIVE  

Zenoss Shows Everything Down

Subject: Zenoss Shows Everything Down
Author: Tim Meads
Posted: 2018-11-21 18:02

So yesterday my zenoss decided to go nutty. EVERYTHING shows down. I can't ping anything from the GUI but if I go into the CLI and ping the devices, everything works fine. 

Where would a good place to start looking be?

------------------------------
Tim Meads
NOC Supervisor
Mountain West Technologies Network
------------------------------


Subject: RE: Zenoss Shows Everything Down
Author: nandha K
Posted: 2018-11-22 03:37

Hi,

Share your version and share the error in the system log or the error in the screen when you try to login


------------------------------
nandha K
------------------------------


Subject: RE: Zenoss Shows Everything Down
Author: Tim Meads
Posted: 2018-11-23 01:16


Hi,

Share your version and share the error in the system log or the error in the screen when you try to login
nandha K,  11-22-2018 03:37 AM
I just upgraded to the latest version, 6.1.2, and I'm not sure which error log you want to see.....I'm not even sure where the CLI logs for zenoss are at. 

I can sign in fine without an issue...I don't have any problems talking to anything from the CLI but I cannot get anything in Zenoss to ping using the WebGUI, so likely there is something broken between the softeware and the networking on the server.

------------------------------
Tim Meads
NOC Supervisor
Mountain West Technologies Network
------------------------------


Subject: RE: Zenoss Shows Everything Down
Author: Tim Meads
Posted: 2018-11-23 03:10

Yes I can ping itself and in zenoss it was showing the server itself being the only device up. Out of all of the services, only the zenmail / zenpop was stopped. Zenmail i figured would have died as it couldn't talk to the mail server.

I noticed that when I attacked to the zope service, I couldn't ping anything but when I was on the command line before attaching to the service I could. I hadn't a clue about attaching to the service container before running an upgrade to the system yesterday. So there's that for learning something new. After finding out that I couldn't ping inside the container, I restarted docker from the command line and it started working. I could then ping my main switch gateways. Which is shdown below. Before it would fail the pings. 

Now I rebooted the server after I started having this problem and it didn't correct the issue. So I'm not sure how restarting docker would have fixed it....


root@zenoss [~] : serviced service attach zope/0

[root@639fccad075a /]# ping 10.58.0.1

PING 10.58.0.1 (10.58.0.1) 56(84) bytes of data.

64 bytes from 10.58.0.1: icmp_seq=1 ttl=254 time=0.900 ms

64 bytes from 10.58.0.1: icmp_seq=2 ttl=254 time=0.775 ms

64 bytes from 10.58.0.1: icmp_seq=3 ttl=254 time=0.734 ms

^C

--- 10.58.0.1 ping statistics ---

3 packets transmitted, 3 received, 0% packet loss, time 2001ms

rtt min/avg/max/mdev = 0.734/0.803/0.900/0.070 ms

[root@639fccad075a /]# ping 10.32.0.2

PING 10.32.0.2 (10.32.0.2) 56(84) bytes of data.

64 bytes from 10.32.0.2: icmp_seq=1 ttl=63 time=0.732 ms

64 bytes from 10.32.0.2: icmp_seq=2 ttl=63 time=0.429 ms

^C

--- 10.32.0.2 ping statistics ---

2 packets transmitted, 2 received, 0% packet loss, time 1000ms

rtt min/avg/max/mdev = 0.429/0.580/0.732/0.153 ms

[root@639fccad075a /]#



------------------------------
Tim Meads
NOC Supervisor
Mountain West Technologies Network
------------------------------


Subject: RE: Zenoss Shows Everything Down
Author: Tim Meads
Posted: 2019-02-01 00:21

Same thing happened again tonight...Zenoss Crashed...I had hell getting it running again and when I did, nothing would be able to ping. It seems that when iptables was running and docker started, I couldn't get zenoss to talk to anything. Shutting down iptables then allows me to get into zenoss, but it wouldn't ping anything. After restarting docker (With iptables already shut down), everything comes back up. This is bizare.

------------------------------
Tim Meads
NOC Supervisor
Mountain West Technologies Network
------------------------------


Subject: RE: Zenoss Shows Everything Down
Author: Jane Curry
Posted: 2018-11-22 04:04

Can you logon to the GUI?

Have you run out of disk space?

Run the toolbox tools - https://support.zenoss.com/hc/en-us/articles/203117595-How-To-Install-And-Use-the-zenoss-toolbox  

Is your Zenoss server a monitored device?  Can it ping itself?

What version are you using?  Are all the daemons running?

Check log files - I'd start with zenhub.log.

Cheers,
Jane

------------------------------
Jane Curry
Skills 1st United Kingdom
jane.curry@skills-1st.co.uk
------------------------------

Subject: RE: Zenoss Shows Everything Down
Author: Jason Olson
Posted: 2018-11-22 09:52

Check if any serviced or java processes are using 100% of a core on the host. if so, you'll need to log into Control Center to see which process is pinned and subsequently failing. Also, do any services show as failed in Control Center?

------------------------------
Jason Olson
------------------------------


Subject: RE: Zenoss Shows Everything Down
Author: Arthur
Posted: 2019-02-02 02:16

Consider this about firewall settings: Chapter 3 Networking
https://www.zenoss.com/services-support/documentation/planning-guide

------------------------------
Arthur
------------------------------


Subject: RE: Zenoss Shows Everything Down
Author: Ryan Matte
Posted: 2019-02-21 10:57

Hey Tim,

When you say "everything shows as down" do you mean you actually have device down events under /Status/Ping for every device in your system, or simply that all of the devices show a status of DOWN if you navigate to them?  I ask because there is currently a known issue which hasn't been addressed yet where the presence of a Critical level event for a device anywhere under the /Status event class will cause that device to be marked as down.  That should hopefully be fixed soon in an upcoming release, but for the time being you probably want to add an event transform to /Status to downgrade the severity of all events from Critical to Error level except for the ones in /Status/Ping.



------------------------------
Ryan Matte
------------------------------


Subject: RE: Zenoss Shows Everything Down
Author: Ryan Matte
Posted: 2019-02-21 11:00

Just to add, if your issue is related to what I described in my last note then clearing any Critical events from /Status by hand (closing them) will get the devices marked as Up again until any further events like that come in.  The transform I mentioned would be to prevent further events coming in as Critical under /Status except for /Status/Ping (which is required for normal ping monitoring to operate properly).

------------------------------
Ryan Matte
------------------------------


< Previous
Import event classes, mappings and transforms via zendmd
  Next
Zenoss 6 zendmd adding device takes 40 seconds
>