TECHZEN Zenoss User Community ARCHIVE  

UI Hangs, Oversubscription, General Instability

Subject: UI Hangs, Oversubscription, General Instability
Author: Paul Manno
Posted: 2018-09-24 09:01

Hi all,

Previously, I have run Zenoss v4 on a bare metal machine for years without issue.  Running Zenoss v6 on a VSphere cluster recently and it's been nothing less than an incredible headache.  I'll log in first thing when I get to the office and the UI just spins and spins.  I'll pull up Control Center to see a number of services having issues (!Answering typically), usually HBase, ZenHub, Central Query are the culprits.  Seems like I have to log in to CC and increase the RAM on several services since there are always one or two of them complaining about being oversubscribed.  I jacked the CPU up to 12 and ram up to 64GB and performance is better, but it will still completely hang periodically.  I set up a cron job to do a nightly bounce of the Zenoss.core service and, again, that seems to help but from time to time the UI will still hang and I'll have to bounce it manually.  Now I've tried everything from the appliance to the ISO, to Yum repositories and it's the same situation every time.  I'm only monitoring about 350 devices.  Some of those devices are using Zenoss as a syslog target.

I can't help but think I'm doing something wrong because I don't know how anyone could live with this.  Are any other users out there running Zenoss on VSphere without issue?  If so, could you please let me in on how you have your VM and storage configured?

Thanks,
Paul

------------------------------
Paul Manno
ShowingTime, Inc.
------------------------------


Subject: RE: UI Hangs, Oversubscription, General Instability
Author: Serge VS
Posted: 2018-09-25 18:36

Hi Paul,

Dont't blame yourself.

Zenoss core does lost stability at 6.x version.

And  Zenoss.IInc stopped to publish sources for components introduced in 6.x version&

Serge.


------------------------------
Serge
------------------------------


Subject: RE: UI Hangs, Oversubscription, General Instability
Author: Paul Manno
Posted: 2018-09-26 08:51

Thanks Serge.  I did a parallel install of the OVA appliance and tweaked some of the hardware parameters.  Specifically, (I am running a single server instance) I set CPU shares to High, fully reserved guest memory (64GB), and set every hard disk shares to High.  Additionally, instead of starting Zenoss.core from Control Center by clicking the top level and starting "all services", I followed the directions in this post:

Zenoss Master - Staged Startup and Shutdown Best Practices for Maintenance
Zenoss remove preview
Zenoss Master - Staged Startup and Shutdown Best Practices for Maintenance
When it becomes necessary to perform maintenance on Zenoss Resource Manager (RM), it is useful to perform controlled, incremental shutdowns and startups for Control Center (CC) and Resource Manager. This KB discusses the best practices for performing these tasks. Stop RM and top-level applications. The RM application should be stopped before stopping Control Center.
View this on Zenoss >


Allowing me to ensure that every dependent service starts before the next.  So far, the system has been quite stable.  I have found the need to restart the zenperfsnmp since, for some reason, it seems to lose connection to zenhub (which is reporting OK health).  Don't know what the deal is there, but fingers crossed that I've gotten this thing to behave.  Oh... I have also had to increase RAM for many of the services which get oversubscribed pretty quickly.  They start out with something like 256MB... I've increased those to 2GB and some even to 10GB.

Paul

------------------------------
Paul Manno
------------------------------


Subject: RE: UI Hangs, Oversubscription, General Instability
Author: Jane Curry
Posted: 2018-09-26 09:46

Moral of the story just might be to return to Zenoss 4.2.5.  It will work fine on vSphere with a fraction of the resources.

Cheers,
Jane

------------------------------
Jane Curry
Skills 1st United Kingdom
jane.curry@skills-1st.co.uk
------------------------------


Subject: RE: UI Hangs, Oversubscription, General Instability
Author: Paul Manno
Posted: 2018-09-26 19:18

I think compliance might be an issue there though?  What kernel versions will v4 run on?  What ssh versions/crypto's are supported with v4?

------------------------------
Paul Manno
------------------------------


Subject: RE: UI Hangs, Oversubscription, General Instability
Author: Jane Curry
Posted: 2018-09-27 07:08

What versions do you need??
Cheers,
Jane

------------------------------
Jane Curry
Skills 1st United Kingdom
jane.curry@skills-1st.co.uk
------------------------------


< Previous
Key questions to consider upgrading from Zenoss 4 to 6
  Next
Use Zenoss to run SQL Query
>