TECHZEN Zenoss User Community ARCHIVE  
FORUMS / ZenPacks, JSON API, and Integrations / DISCUSSION

Community ESXi monitor Zenpack times out

Subject: Community ESXi monitor Zenpack times out
Author: [Not Specified]
Posted: 2015-01-23 04:37

I am currently evaluating Zenoss core as a solution, but I am unable to monitor our ESXi hosts as the commands time out and receive no data when modelling the hosts. Hoping someone else has encountered similar and can shed some light.
Being able to monitor our virtual environment is key for us.

the text returned by the modeller is below

2015-01-23 09:55:07,786 INFO zen.ZenModeler: Connecting to localhost:8789
2015-01-23 09:55:07,792 INFO zen.ZenModeler: Connected to ZenHub
2015-01-23 09:55:08,002 INFO zen.ZenModeler: Collecting for device server.domain
2015-01-23 09:55:08,379 INFO zen.ZenModeler: skipping WMI-based collection, PySamba zenpack not installed
2015-01-23 09:55:08,388 INFO zen.ZenModeler: Python collection device server.domain
2015-01-23 09:55:08,388 INFO zen.ZenModeler: plugins: community.cmd.VMwareESXiHostMap, community.cmd.VMwareESXiInterfaceMap, community.cmd.VMwareESXiDatastoreMap, community.cmd.VMwareESXiGuestMap
2015-01-23 09:55:08,388 INFO zen.PythonClient: Getting VMware ESXi host info for device server.domain
2015-01-23 09:58:57,787 INFO zen.PythonClient: Getting VMware ESXi interface info for device server.domain
2015-01-23 10:02:49,212 INFO zen.PythonClient: Getting VMware ESXi datastore info for device server.domain
2015-01-23 10:06:37,683 INFO zen.PythonClient: Getting VMware ESXi guest info for device server.domain
2015-01-23 10:10:27,675 INFO zen.PythonClient: Python client finished collection for server.domain
2015-01-23 10:10:27,675 WARNING zen.ZenModeler: The plugin community.cmd.VMwareESXiHostMap returned no results.
2015-01-23 10:10:27,675 WARNING zen.ZenModeler: The plugin community.cmd.VMwareESXiInterfaceMap returned no results.
2015-01-23 10:10:27,676 WARNING zen.ZenModeler: The plugin community.cmd.VMwareESXiDatastoreMap returned no results.
2015-01-23 10:10:27,676 WARNING zen.ZenModeler: The plugin community.cmd.VMwareESXiGuestMap returned no results.
2015-01-23 10:10:27,676 INFO zen.ZenModeler: No change in configuration detected
2015-01-23 10:10:27,676 INFO zen.ZenModeler: No command plugins found for server.domain
2015-01-23 10:10:27,676 INFO zen.ZenModeler: SNMP monitoring off for server.domain
2015-01-23 10:10:27,677 INFO zen.ZenModeler: No portscan plugins found for server.domain
2015-01-23 10:10:27,677 INFO zen.ZenModeler: Scan time: 919.67 seconds
2015-01-23 10:10:27,922 INFO zen.ZenModeler: Daemon ZenModeler shutting down

I have also ran the raw command in an ssh shell on the box hosting Zenoss and the data returned is as below. Obviously server name and passwords have changed

zencommand run -v10 --device=server.domain --showfullcommand --showrawresults
2015-01-23 10:20:26,219 DEBUG zen.zencommand: Starting PBDaemon initialization
2015-01-23 10:20:26,220 INFO zen.zencommand: Connecting to localhost:8789
2015-01-23 10:20:26,221 DEBUG zen.pbclientfactory: Starting connection...
2015-01-23 10:20:26,221 DEBUG zen.zencommand: Logging in as admin
2015-01-23 10:20:26,222 DEBUG zen.pbclientfactory: Connected
2015-01-23 10:20:26,222 DEBUG zen.pbclientfactory: Cancelling connect timeout
2015-01-23 10:20:26,223 DEBUG zen.pbclientfactory: Sending credentials
2015-01-23 10:20:26,225 DEBUG zen.pbclientfactory: Cancelling connect timeout
2015-01-23 10:20:26,225 INFO zen.zencommand: Connected to ZenHub
2015-01-23 10:20:26,225 DEBUG zen.zencommand: Setting up initial services: EventService, Products.ZenHub.services.CommandPerformanceConfig
2015-01-23 10:20:26,226 DEBUG zen.zencommand: Chaining getInitialServices with d2
2015-01-23 10:20:26,227 DEBUG zen.zencommand: Loaded service EventService from zenhub
2015-01-23 10:20:26,227 DEBUG zen.zencommand: Loaded service Products.ZenHub.services.CommandPerformanceConfig from zenhub
2015-01-23 10:20:26,228 DEBUG zen.zencommand: Queued event (total of 1) {'rcvtime': 1422008426.228187, 'severity': 0, 'component': 'zencommand', 'agent': 'zencommand', 'summary': 'started', 'manager': 'svr-zenoss.xcddahn.nhs.uk', 'device': 'localhost', 'eventClass': '/App/Start', 'monitor': 'localhost'}
2015-01-23 10:20:26,228 DEBUG zen.zencommand: Sending 1 events, 0 perf events, 0 heartbeats
2015-01-23 10:20:26,229 DEBUG zen.zencommand: Calling connected.
2015-01-23 10:20:26,229 DEBUG zen.collector.config: Heartbeat timeout set to 900s
2015-01-23 10:20:26,229 DEBUG zen.collector.scheduler: add task configLoader, using 1200 second interval
2015-01-23 10:20:26,230 DEBUG zen.zencommand: Performing periodic maintenance
2015-01-23 10:20:26,230 DEBUG zen.collector.scheduler: Task configLoader starting (waited 0 seconds) on 1200 second intervals
2015-01-23 10:20:26,230 DEBUG zen.collector.scheduler: Task configLoader changing state from IDLE to QUEUED
2015-01-23 10:20:26,230 DEBUG zen.collector.scheduler: Task configLoader changing state from QUEUED to RUNNING
2015-01-23 10:20:26,230 DEBUG zen.collector.config: configLoader gathering configuration
2015-01-23 10:20:26,231 DEBUG zen.collector.config: Fetching daemon configuration properties
2015-01-23 10:20:26,252 DEBUG zen.collector.scheduler: Task configLoader changing state from RUNNING to FETCHING_MISC_CONFIG
2015-01-23 10:20:26,253 DEBUG zen.zencommand: Updated configCycleInterval preference to 360
2015-01-23 10:20:26,253 DEBUG zen.zencommand: Changing config task interval from 20 to 360 minutes
2015-01-23 10:20:26,253 DEBUG zen.collector.scheduler: Stopping task configLoader,
2015-01-23 10:20:26,253 DEBUG zen.collector.scheduler: call finished LoopingCall<1200>(CallableTask: configLoader, *(), **{}) : LoopingCall<1200>(CallableTask: configLoader, *(), **{})
2015-01-23 10:20:26,253 INFO zen.collector.scheduler: Detailed Task Statistics:
configLoader Current State: FETCHING_MISC_CONFIG Successful_Runs: 1 Failed_Runs: 0 Missed_Runs: 0

Detailed Task States:
configLoader State: RUNNING Total: 1 Total Elapsed: 0.0221 Min: 0.0221 Max: 0.0221 Mean: 0.0221 StdDev: 0.0000
configLoader State: QUEUED Total: 1 Total Elapsed: 0.0004 Min: 0.0004 Max: 0.0004 Mean: 0.0004 StdDev: 0.0000

2015-01-23 10:20:26,253 DEBUG zen.collector.config: Heartbeat timeout set to 900s
2015-01-23 10:20:26,254 DEBUG zen.collector.scheduler: add task configLoader, using 21600 second interval
2015-01-23 10:20:26,254 DEBUG zen.zencommand: Updated defaultRRDCreateCommand preference to ('RRA:AVERAGE:0.5:1:600', 'RRA:AVERAGE:0.5:6:600', 'RRA:AVERAGE:0.5:24:600', 'RRA:AVERAGE:0.5:288:600', 'RRA:MAX:0.5:6:600', 'RRA:MAX:0.5:24:600', 'RRA:MAX:0.5:288:600')
2015-01-23 10:20:26,254 DEBUG zen.collector.config: Fetching threshold classes
2015-01-23 10:20:26,259 DEBUG zen.zencommand: Loading classes ['Products.ZenModel.MinMaxThreshold', 'Products.ZenModel.ValueChangeThreshold']
2015-01-23 10:20:26,260 DEBUG zen.collector.config: Fetching collector thresholds
2015-01-23 10:20:26,273 DEBUG zen.thresholds: Updating threshold ('high event queue', ('localhost collector', ''))
2015-01-23 10:20:26,274 DEBUG zen.thresholds: Updating threshold ('zenmodeler cycle time', ('localhost collector', ''))
2015-01-23 10:20:26,274 DEBUG zen.collector.config: Fetching configurations
2015-01-23 10:20:26,298 DEBUG zen.zencommand: updateDeviceConfigs: updatedConfigs=['server.domain']
2015-01-23 10:20:26,298 DEBUG zen.zencommand: Processing configuration for server.domain
2015-01-23 10:20:26,299 DEBUG zen.daemon: DummyListener: configuration server.domain added
2015-01-23 10:20:26,299 DEBUG zen.collector.tasks: Splitting config server.domain
2015-01-23 10:20:26,299 DEBUG zen.zencommand: Tasks for config server.domain: {'server.domain 300 Local': <__main__.SshPerformanceCollectionTask object at 0x5725a10>}
2015-01-23 10:20:26,299 DEBUG zen.collector.scheduler: add task server.domain 300 Local, COMMAND schedule Name: server.domain 300 Local configId: server.domain Datasources: 1 using 300 second interval
2015-01-23 10:20:26,300 DEBUG zen.collector.scheduler: Task server.domain 300 Local starting (waited 0 seconds) on 300 second intervals
2015-01-23 10:20:26,300 DEBUG zen.collector.scheduler: Task server.domain 300 Local changing state from IDLE to QUEUED
2015-01-23 10:20:26,300 DEBUG zen.zencommand: purgeOmittedDevices: deletedConfigs=
2015-01-23 10:20:26,300 DEBUG zen.collector.scheduler: Task configLoader finished, result: 'Configuration loaded'
2015-01-23 10:20:26,301 DEBUG zen.collector.scheduler: Task server.domain 300 Local changing state from QUEUED to RUNNING
2015-01-23 10:20:26,301 DEBUG zen.zencommand: Running command(s) locally
2015-01-23 10:20:26,301 DEBUG zen.collector.scheduler: Task server.domain 300 Local changing state from RUNNING to FETCH_DATA
2015-01-23 10:20:26,301 INFO zen.zencommand: Datasource ESXiHost/VMwareHost command: /usr/bin/perl /opt/zenoss/ZenPacks/ZenPacks.community.VMwareESXiMonitor-2.0.1.egg/ZenPacks/community/VMwareESXiMonitor/libexec/esxi_monitor.pl --server 10.117.145.97 --username root --password 'xxxxxxxxxxx' --options 'hostperf:None' | tail -n1
2015-01-23 10:20:26,301 DEBUG zen.zencommand: Running /usr/bin/perl
2015-01-23 10:20:27,534 DEBUG zen.collector.scheduler: tasks to clean KeyedSet([])
2015-01-23 10:20:27,534 DEBUG zen.collector.scheduler: Cleanup on task configLoader
2015-01-23 10:20:27,535 DEBUG zen.collector.scheduler: Scheduler._cleanupTaskComplete: result=None task.name=configLoader
2015-01-23 10:22:02,613 DEBUG zen.zencommand: Datasource: VMwareHost Received exit code: 0 Output:
''
2015-01-23 10:22:02,613 DEBUG zen.zencommand: Process ESXiHost/VMwareHost stopped (0), 96.31 seconds elapsed
2015-01-23 10:22:02,613 DEBUG zen.collector.scheduler: Task server.domain 300 Local changing state from FETCH_DATA to PARSING_DATA
2015-01-23 10:22:02,613 WARNING zen.zencommand: server.domain 300 Local ESXiHost/VMwareHost No data returned for command: /usr/bin/perl /opt/zenoss/ZenPacks/ZenPacks.community.VMwareESXiMonitor-2.0.1.egg/ZenPacks/community/VMwareESXiMonitor/libexec/esxi_monitor.pl --server 10.117.145.97 --username root --password 'xxxxxxxxxxx' --options 'hostperf:None' | tail -n1
2015-01-23 10:22:02,613 DEBUG zen.collector.scheduler: Task server.domain 300 Local changing state from PARSING_DATA to STORE_PERF_DATA
2015-01-23 10:22:02,614 DEBUG zen.zencommand: Queued event (total of 1) {'rcvtime': 1422008522.614153, 'device_guid': '5974427e-cbba-4327-9993-d82a81b16965', 'component': '', 'agent': 'zencommand', 'manager': 'svr-zenoss.xcddahn.nhs.uk', 'device': 'server.domain', 'eventClass': '/Cmd/Fail', 'severity': 0, 'monitor': 'localhost', 'summary': 'Datasource ESXiHost/VMwareHost command timed out', 'eventKey': 'Timeout'}
2015-01-23 10:22:02,614 DEBUG zen.zencommand: Queued event (total of 2) {'rcvtime': 1422008522.614443, 'monitor': 'localhost', 'component': '', 'agent': 'zencommand', 'summary': "No data returned for command: /usr/bin/perl /opt/zenoss/ZenPacks/ZenPacks.community.VMwareESXiMonitor-2.0.1.egg/ZenPacks/community/VMwareESXiMonitor/libexec/esxi_monitor.pl --server 10.117.145.97 --username root --password 'xxxxxxxxxxx' --options 'hostperf:None' | tail -n1", 'manager': 'svr-zenoss.xcddahn.nhs.uk', 'command': "/usr/bin/perl /opt/zenoss/ZenPacks/ZenPacks.community.VMwareESXiMonitor-2.0.1.egg/ZenPacks/community/VMwareESXiMonitor/libexec/esxi_monitor.pl --server 10.117.145.97 --username root --password 'xxxxxxxxxxx' --options 'hostperf:None' | tail -n1", 'eventKey': 'VMwareHost', 'device': 'server.domain', 'eventClass': '/Cmd/Fail', 'device_guid': '5974427e-cbba-4327-9993-d82a81b16965', 'severity': 3}
2015-01-23 10:22:02,614 DEBUG zen.collector.scheduler: Task server.domain 300 Local finished, result: [(, ParsedResults
events: [{'component': '',
'device': 'server.domain',
'eventClass': '/Cmd/Fail',
'eventKey': 'Timeout',
'severity': 0,
'summary': 'Datasource ESXiHost/VMwareHost command timed out'},
{'command': "/usr/bin/perl /opt/zenoss/ZenPacks/ZenPacks.community.VMwareESXiMonitor-2.0.1.egg/ZenPacks/community/VMwareESXiMonitor/libexec/esxi_monitor.pl --server 10.117.145.97 --username root --password 'xxxxxxxxxxx' --options 'hostperf:None' | tail -n1",
'component': '',
'device': 'server.domain',
'eventClass': '/Cmd/Fail',
'eventKey': 'VMwareHost',
'severity': 3,
'summary': "No data returned for command: /usr/bin/perl /opt/zenoss/ZenPacks/ZenPacks.community.VMwareESXiMonitor-2.0.1.egg/ZenPacks/community/VMwareESXiMonitor/libexec/esxi_monitor.pl --server 10.117.145.97 --username root --password 'xxxxxxxxxxx' --options 'hostperf:None' | tail -n1"}]
values: []})]
2015-01-23 10:22:02,616 DEBUG zen.collector.scheduler: Task server.domain 300 Local changing state from STORE_PERF_DATA to IDLE
2015-01-23 10:22:02,616 INFO zen.zencommand: 1 devices processed (0 datapoints)
2015-01-23 10:22:02,617 INFO zen.collector.scheduler: Tasks: 2 Successful_Runs: 1 Failed_Runs: 0 Missed_Runs: 0 Queued_Tasks: 0 Running_Tasks: 1
2015-01-23 10:22:02,617 DEBUG zen.collector.scheduler: In shutdown stage before
2015-01-23 10:22:02,617 DEBUG zen.collector.scheduler: Stopping running task server.domain 300 Local
2015-01-23 10:22:02,617 DEBUG zen.collector.scheduler: call finished LoopingCall<300>(CallableTask: server.domain 300 Local, *(), **{}) : LoopingCall<300>(CallableTask: server.domain 300 Local, *(), **{})
2015-01-23 10:22:02,617 DEBUG zen.collector.scheduler: Removing task server.domain 300 Local
2015-01-23 10:22:02,617 DEBUG zen.collector.scheduler: Removing task configLoader
2015-01-23 10:22:02,618 DEBUG zen.collector.scheduler: tasks to clean KeyedSet([<__main__.SshPerformanceCollectionTask object at 0x5725a10>, ])
2015-01-23 10:22:02,618 DEBUG zen.collector.scheduler: Cleanup on task server.domain 300 Local COMMAND schedule Name: server.domain 300 Local configId: server.domain Datasources: 1
2015-01-23 10:22:02,618 DEBUG zen.collector.scheduler: Scheduler._cleanupTaskComplete: result=None task.name=server.domain 300 Local
2015-01-23 10:22:02,618 DEBUG zen.collector.scheduler: Cleanup on task configLoader
2015-01-23 10:22:02,618 DEBUG zen.collector.scheduler: Scheduler._cleanupTaskComplete: result=None task.name=configLoader
2015-01-23 10:22:02,618 DEBUG zen.zencommand: Tried to stop reactor that was stopped
2015-01-23 10:22:02,618 INFO zen.zencommand: Daemon CollectorDaemon shutting down
2015-01-23 10:22:02,619 DEBUG zen.zencommand: Sending 2 events, 0 perf events, 0 heartbeats
2015-01-23 10:22:02,635 DEBUG zen.zencommand: Removing service EventService
2015-01-23 10:22:02,636 DEBUG zen.zencommand: Removing service Products.ZenHub.services.CommandPerformanceConfig
2015-01-23 10:22:02,636 DEBUG zen.pbclientfactory: Lost connection to ::1:8789 - [Failure instance: Traceback (failure with no frames): : Connection to the other side was lost in a non-clean fashion: Connection lost.
]
2015-01-23 10:22:02,636 DEBUG zen.collector.scheduler: In shutdown stage during
2015-01-23 10:22:02,637 DEBUG zen.collector.scheduler: In shutdown stage after


Subject: In my case I had some issues
Author: Joan
Posted: 2015-01-23 05:52

In my case I had some issues with the guestperf options but no the hostperf, unfortunately I guess that the issue (at least in my case) is because of the perl sdk from vmware, that is not having much attention due to having cooler alternatives (java, ruby, python) released as opensource.
Hope you can get some help


Subject: Fixed!
Author: [Not Specified]
Posted: 2015-01-27 03:17

I've managed to resolve this.

It transpires that the VMware perl and cli commands were spitting out an error about a self signed certificate installed on ESXi as default which didn't match the dns FQDN.

Delving deeper with some hard google fu indicated that it was a problem with the perl-libwww and LWP libraries.

Various suggested fixes about removing proxy environment variables and setting PERL_LWP_SSL_VERIFY_HOSTNAME=0 all failed.

What did work was removing the perl-libwww-perl package from the system and then manually installing version 5.87 using CPAN as below

cpan> install GAAS/libwww-perl-5.837.tar.gz

This has done the trick and is working great.

Hope this helps someone else


Subject: Need help removing the libwww module
Author: [Not Specified]
Posted: 2015-07-30 14:14

anyone have any direction on this
been searching around for a couple of hours

thanks!


Subject: Nevermind I figured out the issue
Author: [Not Specified]
Posted: 2015-08-01 02:31

you have to have a user with SSH access and the ready only one does not do it!



< Previous
ZenPacks.zenoss.Layer2 		  Next
Virtual IP Zenpack 		>