| Subject: |
RE: zencommand suddenly using high amounts of memory |
| Author: |
Jane Curry |
| Posted: |
2018-10-11 05:49 |
A few thoughts...
Doesn't look like the number of devices polled has changed over the last few hours but has it grown substantially over the last month?
Check zencommand .log to see if there are significant timeouts going on which might result in lots more processes actually running at one time and in zencommand checking threads to see if an answer has been received.
Do you have "top" records for your Zenoss server to see if anything else has changed?
What happened in your Zenoss world "about a month ago" that might have caused this? Extra devices added? "Big" devices added eg. big routers with lots of interfaces, VMware devices added? New ZenPack or version of ZenPack installed eg. updated Linux ZenPack?
Cheers,
Jane
------------------------------
Jane Curry
Skills 1st United Kingdom
jane.curry@skills-1st.co.uk
------------------------------
| Subject: |
RE: zencommand suddenly using high amounts of memory |
| Author: |
Paul Giordano |
| Posted: |
2018-11-26 10:28 |
Apologies for taking forever to respond. There has been no significant increase in the number or types of servers. The zencommand continues to use more and more memory, now requesting 16GB. Just seems to go up and up. I cannot find the zencommand.log file, it's not in /opt/zenoss/log - is there something I need to turn on somewhere? I'm running with all of the defaults.
------------------------------
Paul Giordano
Senior Systems Engineer
Zethcon Corporation
------------------------------
| Subject: |
RE: zencommand suddenly using high amounts of memory |
| Author: |
Jane Curry |
| Posted: |
2018-11-26 11:07 |
The log file ought to be in /opt/zenoss/log - in the zencommand container. Do you have separate collectors? If so, the logfile will be under a subdirectory with the collector's name.
You might have something going on whereby LOTS of commands are starting and either never timing out or take a long time to time out. What is your zCommandCommandTimeout zProperty typically set to?
Have a look with top and see if there are lots of processes that have been started by zencommand.
Cheers,
Jane
------------------------------
Jane Curry
Skills 1st United Kingdom
jane.curry@skills-1st.co.uk
------------------------------
| Subject: |
RE: zencommand suddenly using high amounts of memory |
| Author: |
Paul Giordano |
| Posted: |
2018-11-26 14:07 |
Thanks Jane - checking the logs. The zCommandTimeout is set to 15 seconds. Top, though, yields over 3000 tasks, almost all of them "Hiding the command line arguments" and "tail -n1". So I think we're onto the problem - but how do I figure out what all of those tasks are? Some of them have been around for days, back to the last time I restarted Zenoss.
------------------------------
Paul Giordano
Senior Systems Engineer
Zethcon Corporation
------------------------------
| Subject: |
RE: zencommand suddenly using high amounts of memory |
| Author: |
Paul Giordano |
| Posted: |
2018-11-27 09:23 |
The logs indicated some issues, which I corrected - but every 5 minutes one of the "Hiding" and "tail -n1" are hanging out there. I came in to several hundred this AM. How do I figure out what they are? Is there some debug I can issue that will generate the command rather than hiding it?
------------------------------
Paul Giordano
Senior Systems Engineer
Zethcon Corporation
------------------------------
| Subject: |
RE: zencommand suddenly using high amounts of memory |
| Author: |
Paul Giordano |
| Posted: |
2018-11-27 10:47 |
So I restarted Zenoss this morning, and lo and behold the problem is not occurring any more. Sheesh. Thanks for all the help.
------------------------------
Paul Giordano
Senior Systems Engineer
Zethcon Corporation
------------------------------
| Subject: |
RE: zencommand suddenly using high amounts of memory |
| Author: |
Jane Curry |
| Posted: |
2018-11-27 11:24 |
I suspect that the problem has "gone away" because of the restart. I am afraid itr may come back over time :)
You end up with lots of processes started by zencommand and then they get orphaned - for some reason, they do not get properly killed off after the zCommandTimeout time. Hope I am wrong on this but keep yours eyes on it ;)
Look at the scripts that are being run and consider whether they might be re-written to ensure they get properly killed rather than becoming zombies. You might look at running a script from cron that kills any such processes that are more than a few minutes old.
Cheers,
Jane
------------------------------
Jane Curry
Skills 1st United Kingdom
jane.curry@skills-1st.co.uk
------------------------------
