TECHZEN Zenoss User Community ARCHIVE  

"Unknown connection problem to zenhub" messages in zenhub log

Subject: "Unknown connection problem to zenhub" messages in zenhub log
Author: Larry
Posted: 2020-05-22 00:43

We are running Zenoss 6.2.1 (Community). Recently we started getting a lot of these messages in the zenhub service log:

CRITICAL zen.pbclientfactory: Unknown connection problem to zenhub <class 'twisted.spread.pb.PBConnectionLost'>

We are getting 2-3K such messages a day. This started a few days ago. We recently made some configuration changes to resolve some performance issues:

1. Increased zenhub instances: 1 -> 2
2. Increased zenhub workers. Currently at 26.

I know that the number of zenhub workers is WAY too high, but we're trying to resolve problems with missed runs and discarded events.

Any ideas on how to resolve this error message?

We are also seeing a lot of this message:

INFO zen.ZenHub: Worker 11 (5394) reports 2020-05-22 04:09:28,693 INFO zen.pbclientfactory: Initial connect timed out after 30 seconds

Thanks in advance,




------------------------------
Larry
------------------------------


Subject: RE: "Unknown connection problem to zenhub" messages in zenhub log
Author: Ryan Matte
Posted: 2020-06-05 17:11

Hi, unfortunately you're likely running in to some known performance issues with the hub on older versions of Resource Manager.  The best I can recommend is to update to the latest version of Core that is available (6.3.2 I believe) which contains some fixes and then to upgrade to the next release (probaby 6.5.0 I think, but possibly  6.4.x, not sure how they plan to do the core release) once it comes out as it will contain further improvements to hub performance.  Having 26 workers is honestly likely counter-productive and you may want to reduce that number a bit.  We do not recommend having multiple zenhub instances as that causes problems of it's own.  In fact in newer versions of RM we don't even make it possible to increase the number of zenhub instances and we break out the workers in to their own services so that they can be increased individually as separate instances.  In any case, upgrades are going to be the way forward for you in terms of improving the performance of this.



------------------------------
Ryan Matte
------------------------------


Subject: RE: "Unknown connection problem to zenhub" messages in zenhub log
Author: Larry
Posted: 2020-06-05 22:30

Hi Ryan,
   Thanks for the responses. I will try your recommendations. As for upgrading Core, will 6.5.0 be available for Community users or only paying users? Any estimate on when the next Community release will be out?

Thanks!

------------------------------
Larry
------------------------------


Subject: RE: "Unknown connection problem to zenhub" messages in zenhub log
Author: Ryan Matte
Posted: 2020-06-09 17:31

Generally all Zenoss releases eventually become Core releases, however there is generally a significant delay for that.  The Core version generally trails behind by one version, so the next core release will probably be 6.4.1 and then 6.5.0 would be released sometime after that.  I however can't make any guarantees on that as I'm not involved in that decision making.

------------------------------
Ryan Matte
------------------------------

Hi, unfortunately you're likely running in to some known performance issues with the hub on older versions of Resource Manager.  The best I can recommend is to update to the latest version of Core that is available (6.3.2 I believe) which contains some fixes and then to upgrade to the next release (probaby 6.5.0 I think, but possibly  6.4.x, not sure how they plan to do the core release) once it comes out as it will contain further improvements to hub performance.  Having 26 workers is honestly likely counter-productive and you may want to reduce that number a bit.  We do not recommend having multiple zenhub instances as that causes problems of it's own.  In fact in newer versions of RM we don't even make it possible to increase the number of zenhub instances and we break out the workers in to their own services so that they can be increased individually as separate instances.  In any case, upgrades are going to be the way forward for you in terms of improving the performance of this.



------------------------------
Ryan Matte


Subject: RE: "Unknown connection problem to zenhub" messages in zenhub log
Author: Larry
Posted: 2020-06-22 15:45

Thanks again Ryan. We tried upgrading our DEV installation from 6.2.1 -> 6.3.2, but the upgrade-core.sh script fails:

ImportError: No module named jira.client

ERROR:zen.ZenPackCmd:Command '['zenpack', '--install', '/tmp/ZenPacks.zenoss.Dashboard-1.3.3-py2.7.egg']' returned non-zero exit status 3

We have a custom zenpack that uses jira. No ideas about dashboard. Any ideas?

Thanks,

------------------------------
Larry
------------------------------


Subject: RE: "Unknown connection problem to zenhub" messages in zenhub log
Author: Ryan Matte
Posted: 2020-06-05 17:17

For the time being I would recommend that you move back to a single instance of zenhub and then increase the workers on that single instance as needed.  There is no benefit to having multiple master zenhub instances.  Only the number of workers should be increased as load increases.

------------------------------
Ryan Matte
------------------------------


< Previous
"Discarded event - queue overflow" messages in zenpython log
  Next
how to install zenos 5 community or zenoss 6 community
>