TECHZEN Zenoss User Community ARCHIVE  

Multiple Issues with Zenpython on Zenoss 6.2.1

Subject: Multiple Issues with Zenpython on Zenoss 6.2.1
Author: Larry
Posted: 2020-01-31 02:26

We are running pythoncollector ZenPack v1.11.0 on Zenoss 6.2.1 Community Edition. Zenpython is acting strangely as viewed from CC.

Issues:
1. "Missed Runs" keeps growing to a large value (then drops to 0).
2. Spikes in "Event Queue Length" graph to 2K+.
3. It appears that no data is collected for several hours after the "reset" (missed runs dropping to 0).

See the attached graphs for details. OS (CPU and memory) metrics look fine.

Any ideas?

Thanks in advance,


------------------------------
Larry
------------------------------

Attachments:

Zenpython_2020.01.31_24hr.jpg



Subject: RE: Multiple Issues with Zenpython on Zenoss 6.2.1
Author: Michael Rogers
Posted: 2020-02-03 11:13

Larry,

The best way to find out what zenpython is up to will be to check its logs directly.  You can attach to the container and tail its log with the following:

serviced service attach zenpython
su - zenoss
cd /opt/zenoss/log/
tail -f zenpython.log​

If a collection daemon is slow to return to work after a restart, it could come down to an issue with zenhub.  There are numerous factors that can affect performance (number of collection daemons you're running, number of devices they're monitoring, CPU cores in the pool, available zenhub workers, etc.), so checking logs is a decent place to start.  If the zenpython.log points to zenhub as a culprit, you can get to its log by altering the commands above.

Check the zenhub.log for messages like "all workers busy" or "available workers []".  If you see messages along those lines, you may need to scale up your available workers.  Please note that prior to 6.3, zenhub workers are controlled via the zenhub.cnf file; after 6.3, zenhub workers were spun out into their own service which can be scaled directly in Control Center.

Let us know what you find?

------------------------------
Michael Rogers
Senior Instructor
Zenoss
Austin TX
------------------------------


Subject: RE: Multiple Issues with Zenpython on Zenoss 6.2.1
Author: Larry
Posted: 2020-02-04 00:42

Hi Michael,
    Thanks for the response. I checked the zenpython and zenhub logs but didn't find any "worker" messages.

I did find this though:
2020-02-04 01:59:04,414 INFO zen.zenpython: Counter discardedEvents, value 3629
2020-02-04 01:59:04,416 INFO zen.zenpython: Counter eventCount, value 19224856
2020-02-04 01:59:04,417 INFO zen.zenpython: 123 devices processed (8250973 datapoints)
2020-02-04 01:59:04,508 INFO zen.collector.scheduler: Tasks: 1383 Successful_Runs: 459587 Failed_Runs: 12505 Missed_Runs: 6254 Queued_Tasks: 25 Running_Tasks: 152

Even though it didn't indicate a worker problem, I decided to go ahead and increase the workers. The server has 8 CPUs and 32 GB memory, so this shouldn't be an issue. I found out from experience that the variables need to be changed vi the CC edit window. I changed the following:

#workers 4
#workers-reserved-for-events 2
Thanks,
Larry




------------------------------
Larry
------------------------------


< Previous
Not Able to Change Number of Zenpython Instances in Control Center
  Next
Getting "Improper format of Kerberos configuration file" for WinRM Device
>