TECHZEN Zenoss User Community ARCHIVE  

runzope high cpu

Subject: runzope high cpu
Author: [Not Specified]
Posted: 2015-01-26 07:23

Our Zenoss is running really slow.
Top shows runzope running at 100+ % all the time.

The only suspicious log message would be:

tail -100 zeneventserver.log
2015-01-26T15:20:25.885 [qtp1709834834-4221 - http://localhost:8084/zeneventserver/api/1.0/events/] WARN org.zenoss.zep.index.impl.QueryBuilder - Invalid IP address:
2015-01-26T15:20:35.622 [qtp1709834834-4289 - http://localhost:8084/zeneventserver/api/1.0/events/] WARN org.zenoss.zep.index.impl.QueryBuilder - Invalid IP address:
2015-01-26T15:20:36.145 [qtp1709834834-4289 - http://localhost:8084/zeneventserver/api/1.0/events/] WARN org.zenoss.zep.index.impl.QueryBuilder - Invalid IP address:
2015-01-26T15:20:55.020 [qtp1709834834-4221 - http://localhost:8084/zeneventserver/api/1.0/events/] WARN org.zenoss.zep.index.impl.QueryBuilder - Invalid IP address:
2015-01-26T15:20:55.610 [qtp1709834834-4221 - http://localhost:8084/zeneventserver/api/1.0/events/] WARN org.zenoss.zep.index.impl.QueryBuilder - Invalid IP address:
2015-01-26T15:20:57.506 [qtp1709834834-4318 - http://localhost:8084/zeneventserver/api/1.0/events/] WARN org.zenoss.zep.index.impl.QueryBuilder - Invalid IP address:
2015-01-26T15:20:58.320 [qtp1709834834-4221 - http://localhost:8084/zeneventserver/api/1.0/events/] WARN org.zenoss.zep.index.impl.QueryBuilder - Invalid IP address:
2015-01-26T15:21:07.525 [qtp1709834834-4317 - http://localhost:8084/zeneventserver/api/1.0/events/] WARN org.zenoss.zep.index.impl.QueryBuilder - Invalid IP address:
.
.
.

Any ideas what to look for



Subject: it does look like something
Author: Andrew Kirch
Posted: 2015-01-26 13:06

it does look like something amiss there but I doubt that that is your problem. Could you provide a bit more information:
1. how many devices are you monitoring
2. what hardware are you using
3. when did this problem start
4. how many events are in your event history
5. have you seen (now or recently) a PosKey Error in the UI or in your logs

Andrew Kirch

akirch@gvit.com

Need Zenoss support, consulting or custom development Look no further. Email or PM me!

Ready for Distributed Topology (collectors) for Zenoss 5 Coming May 1st from GoVanguard



Subject: 1. how many devices are you
Author: [Not Specified]
Posted: 2015-01-27 02:48

1. how many devices are you monitoring

~1500

2. what hardware are you using

Dell M600. CPU 2x Xeon L5420, 32GB RAM.

3. when did this problem start

About one day after install. Imported devices from previous 3.2.1. Current ver is 4.2.5.

4. how many events are in your event history

mysql> select * from event_archive
8696 rows in set (5.68 sec)

5. have you seen (now or recently) a PosKey Error in the UI or in your logs

No.

Misc.
#################################

zendmd
>>> len(dmd.global_catalog)
93911

>>> import math; from test import pystone; int(math.ceil(sum(s[1] for s in (pystone.pystones() for i in range(3)))/150.0))
1416

#################################

cat /etc/my.cnf | grep -v ^#
[mysqld]
skip_external_locking

innodb_buffer_pool_size =256M

innodb_log_file_size = 1024M

innodb_additional_mem_pool_size = 32M
innodb_log_buffer_size = 8M
innodb_flush_method = O_DIRECT
innodb_flush_log_at_trx_commit = 2

innodb_thread_concurrency = 2

innodb_file_format = Barracuda

query_cache_size = 0
query_cache_type = OFF

max_allowed_packet = 64M
wait_timeout = 86400
interactive_timeout = 86400
max_connections = 1500
max_user_connections = 1500

innodb_purge_threads = 1
innodb_max_purge_lag = 100000

[client]
user = zenoss

[mysql]
max_allowed_packet = 64M

[mysqldump]
max_allowed_packet = 64M

#################################

[root@nms log]# rabbitmqctl -p /zenoss list_queues
Listing queues ...
celery 0
nms.alcom.ax.celeryd.pidbox 0
zenoss.queues.zep.signal 0
zenoss.queues.zep.modelchange 0
zenoss.queues.zep.migrated.summary 0
zenoss.queues.zep.rawevents 0
zenoss.queues.zep.heartbeats 0
zenoss.queues.zep.zenevents 0
zenoss.queues.zep.migrated.archive 0



Subject: it's going to generate some
Author: Andrew Kirch
Posted: 2015-01-27 13:46

it's going to generate some load, but as this is a migration, I'd suggest remodeling all the devices, and making sure nothing is "stuck"

Andrew Kirch

akirch@gvit.com

Need Zenoss support, consulting or custom development Look no further. Email or PM me!

Ready for Distributed Topology (collectors) for Zenoss 5 Coming May 1st from GoVanguard



< Previous
Status WMI error event Error collecting performance data: NT_STATUS_CONNECTION_R ...
  Next
Debugging ZenWebserver
>