TECHZEN Zenoss User Community ARCHIVE  
FORUMS / ZenPacks, JSON API, and Integrations / DISCUSSION

Windows ZenPack - Kerberos settings config file

Subject: Windows ZenPack - Kerberos settings config file
Author: Jane Curry
Posted: 2017-09-08 05:19

I have ZenPacks.zenoss.Microsoft.Windows V 2.7.7 with Zenoss Core 5.2.1.  Underlying Python ZenPack is version is 1.9.0.

Most of our Windows devices appear in both our forward and reverse Domain Name Server (DNS) lookups. One or two do not appear in the reverse lookup files (for good reasons), so we can resolve name to address but not address to name with DNS.  We use domain names to a trusted server and have followed the various good bits of advice re Kerberos parameters in the readme of the Windows ZenPack at https://www.zenoss.com/product/zenpacks/microsoft-windows    .Everything works fine except for the few devices that do not have a reverse DNS lookup.  So we have the following zProperties set to the correct values for us:
We have zWinRMServerName set to the fully-qualified domain name.

The error messages we get from  the failing devices includes "WinRS: Failed collection Server not found in Kerberos database: HTTP@<FQDN here> on <FQDN here " (obviously with the <FQDN here" as appropriate.  The ZenPack README strongly suggests that this is because we do not have a reverse DNS entry and suggests using the recently added zWinRMKrb5DisableRDNS parameter, setting it from the default of false to true to inhibit the Kerberos reverse DNS lookup.  Tried this and it makes no difference.  Note that zWinRMKrb5DisableRDNS is a global parameter and must be set at the /Server/Microsoft level.

Under the covers, there is a Kerberos configuration file that this parameter should update.  Find it in the zenpython container under /opt/zenoss/var/krb5 - there should be a file called krb5.conf. There is also a directory, /opt/zenoss/var/krbcc which contains cache files for Kerberos.  Both directories are recreated when zenpython (which runs all the Windows RM stuff) is restarted.  Indeed, the cache files and the krb5.conf file can be deleted with everything running, and they will be quickly recreated.

The problem is that, regardless of any changes to zWinRMKrb5DisableRDNS, I never see any rdns entry in the krb.conf file.  The default, if there is no explicit rdns configuration, is:
(bit confusing - the zProp default is False (so rdns is enabled); the krb5 conf file has default rdns = true (so rdns is enabled).  I want this behaviour reversed.

I can delete the conf file and cache files and swap the zProperty - the files are recreated but no rdns line - anywhere, true or false.

The README also says you can add your own config file; must be a legal krb.conf-format file and filename must only contain alphanumerics, underscore and minus (so don't call it xyz.conf). It goes into a specific directory.  The default is /opt/zenoss/var/krb5/config (again in the Python container).  I have tried adding a wee file, rndc, in this directory with:
[libdefaults]
rdns = false

Removed krb5.conf and cache file - still getting the same Kerberos error events.  Note that this file under the config directory is transient. When zenpython is restarted it will be lost as the krb5 and krb5cc directory hierarchies are completely recreated.

There is also a further zProperty, zWinRMKrb5includedir, where you can put your extra config file, somewhere that will persist for the zenpython container, beyond restarts.  Such a directory is /opt/serviced/var/volumes/<tenant-id>/var-zenpacks which appears in containers as /var/zenoss.  I have a scripts subdirectory under here and have put my rndc file in there and set zWinRMKrb5includedir, at the /Server/Microsoft level, to be /var/zenoss/scripts.  Removed krb5.conf and the cache files.  Again, the krb5.conf is quickly recreated but no changes.  No rdns = false and no changes to the includedir stanza which still has:
So, my conclusion is that none of the zWin properties are making it to the config file - though the [realms] and [domain_realm] sections of krb5.conf obviously ARE being created correctly.

I would love some help or insight on this - or any other experiences of working or non-working.
Cheers,
Jane


------------------------------
Jane Curry
Skills 1st United Kingdom
jane.curry@skills-1st.co.uk
------------------------------

Subject: RE: Windows ZenPack - Kerberos settings config file
Author: Jane Curry
Posted: 2017-09-08 14:35

An update on this - the aspect of setting the zWinRMKrb5includedir zProp.  After "a while" - sorry, I don't know how long, been out for 6 hours - I found the krb5.conf had had the new include directory added so I had:

includedir /opt/zenoss/var/krb5/config
includedir /var/zenoss/scripts

Then realised that /var/zenoss/scripts also had other files, in addition to my little krb config file so changed the zProp to be /var/zenoss/scripts/fred.  krb5.conf was updated very quickly but the old file wasn't removed so I now have:

includedir /opt/zenoss/var/krb5/config
includedir /var/zenoss/scripts/fred
includedir /var/zenoss/scripts

Incidentally, the ZenPack readme says that if the zWinRMKrb5includedir directory contains any non-legal krb configuration files then it would be ignored; in this case, /var/zenoss/scripts was in breach of that rule but it did get added to my krb.conf.

Tried pushing configs but still end up with all three includedir lines.  Tried deleting krb5.conf and cache files - all 3 lines are in the re-created file.  So how do I remove the unwanted one?

Still no rdns entry though :(

Cheers,
Jane




------------------------------
Jane Curry
Skills 1st United Kingdom
jane.curry@skills-1st.co.uk
------------------------------ We have zWinRMServerName set to the fully-qualified domain name.

The error messages we get from  the failing devices includes "WinRS: Failed collection Server not found in Kerberos database: HTTP@<FQDN here> on <FQDN here " (obviously with the <FQDN here" as appropriate.  The ZenPack README strongly suggests that this is because we do not have a reverse DNS entry and suggests using the recently added zWinRMKrb5DisableRDNS parameter, setting it from the default of false to true to inhibit the Kerberos reverse DNS lookup.  Tried this and it makes no difference.  Note that zWinRMKrb5DisableRDNS is a global parameter and must be set at the /Server/Microsoft level.

Under the covers, there is a Kerberos configuration file that this parameter should update.  Find it in the zenpython container under /opt/zenoss/var/krb5 - there should be a file called krb5.conf. There is also a directory, /opt/zenoss/var/krbcc which contains cache files for Kerberos.  Both directories are recreated when zenpython (which runs all the Windows RM stuff) is restarted.  Indeed, the cache files and the krb5.conf file can be deleted with everything running, and they will be quickly recreated.

The problem is that, regardless of any changes to zWinRMKrb5DisableRDNS, I never see any rdns entry in the krb.conf file.  The default, if there is no explicit rdns configuration, is:
(bit confusing - the zProp default is False (so rdns is enabled); the krb5 conf file has default rdns = true (so rdns is enabled).  I want this behaviour reversed.

I can delete the conf file and cache files and swap the zProperty - the files are recreated but no rdns line - anywhere, true or false.

The README also says you can add your own config file; must be a legal krb.conf-format file and filename must only contain alphanumerics, underscore and minus (so don't call it xyz.conf). It goes into a specific directory.  The default is /opt/zenoss/var/krb5/config (again in the Python container).  I have tried adding a wee file, rndc, in this directory with:
[libdefaults]
rdns = false

Removed krb5.conf and cache file - still getting the same Kerberos error events.  Note that this file under the config directory is transient. When zenpython is restarted it will be lost as the krb5 and krb5cc directory hierarchies are completely recreated.

There is also a further zProperty, zWinRMKrb5includedir, where you can put your extra config file, somewhere that will persist for the zenpython container, beyond restarts.  Such a directory is /opt/serviced/var/volumes/<tenant-id>/var-zenpacks which appears in containers as /var/zenoss.  I have a scripts subdirectory under here and have put my rndc file in there and set zWinRMKrb5includedir, at the /Server/Microsoft level, to be /var/zenoss/scripts.  Removed krb5.conf and the cache files.  Again, the krb5.conf is quickly recreated but no changes.  No rdns = false and no changes to the includedir stanza which still has:
So, my conclusion is that none of the zWin properties are making it to the config file - though the [realms] and [domain_realm] sections of krb5.conf obviously ARE being created correctly.

I would love some help or insight on this - or any other experiences of working or non-working.
Cheers,
Jane


------------------------------
Jane Curry
Skills 1st United Kingdom
jane.curry@skills-1st.co.uk
------------------------------

Subject: RE: Windows ZenPack - Kerberos settings config file
Author: Jane Curry
Posted: 2017-09-11 05:19

Restarted Zenoss.core and serviced.  My incluedir now DOES reflect the parameter in zWinRMKrb5includedir but still also has the default:
includedir /opt/zenoss/var/krb5/config
includedir /var/zenoss/scripts/fred

Still no rdns though :(

Still getting "Server not found in Kerberos database" events.

Cheers,
Jane

------------------------------
Jane Curry
Skills 1st United Kingdom
jane.curry@skills-1st.co.uk
------------------------------

Subject: RE: Windows ZenPack - Kerberos settings config file
Author: Jane Curry
Posted: 2017-10-10 15:33

Anyone else??  Zenoss?? have some thoughts on this?

Cheers,
Jane

------------------------------
Jane Curry
Skills 1st United Kingdom
jane.curry@skills-1st.co.uk
------------------------------

Subject: RE: Windows ZenPack - Kerberos settings config file
Author: Dave Bouchillon
Posted: 2017-10-10 18:03

Hi Jane, can you check a couple of things in zendmd?

1.  Find a device where you're seeing this problem and see what is returned for kerberos_rdns(). 
     find('mydevice').kerberos_rdns()

2.  check for any other zWinRMKrb5DisableRDNS that could have been set to see if we have a problem.
     for o in org.getOverriddenObjects('zWinRMKrb5DisableRDNS', showDevices=True):
         print o.id

Thanks
Dave

------------------------------
Dave Bouchillon
Zenoss
------------------------------

Subject: RE: Windows ZenPack - Kerberos settings config file
Author: Jane Curry
Posted: 2017-10-11 06:30

Thanks for the input, Dave.

Various devices I have tested with:
   find('<mydevice>').kerberos_rdns()

all respond with True.  This, I believe, is the value of the zProperty zWinRMKrb5DisableRDNS, so True should be correct to disable the reverse lookup? But this is looking up stuff in dmd, not accessing what kerberos uses?

Your second test, you need to define "org", so I used:
In [16]: for org in dmd.Devices.Server.Microsoft.getSubOrganizers():
...:    print org.id
...:    for p in org.getOverriddenObjects('zWinRMKrb5DisableRDNS', showDevices=True):
...:       print 'Overrides on organizer %s are %s ' (org.id, p.id)

The organizers were printed but there were no overrides - again, what I would expect?

On close inspection, I find that /opt/zenoss/var/krb5/krb5.conf is different in the zope container and in the zenpython container!  What is the mechanism that actually propagates Kerberos zProperties to the various containers?  I am guessing that when a config cycle runs (or Push Configs to a device) then zProps are propagated by zenhub to the various daemons, like zenpython?

zenhub doesn't seem to have a /opt/zenoss/var/krb5 directory at all.  I assume what is actually used to communicate with a device is what is in the zenpython container?  And the whole /opt/zenoss/var/krb5 directory hierarchy is recreated when zenpython restarts?

Inspecting the [Realms] section of krb5.conf, many have several lines for kdc, each with one IP or address (not comma-separated on one line).  I believe these multiple lines reflect changes made via zProps and any new change just goes in at the top of the list, but none ever get deleted?  And I think that only the first kdc line is actually used??

Still no entry for rdns = false  :(

Cheers,
Jane


------------------------------
Jane Curry
Skills 1st United Kingdom
jane.curry@skills-1st.co.uk
------------------------------

Subject: RE: Windows ZenPack - Kerberos settings config file
Author: Dave Bouchillon
Posted: 2017-10-13 11:50

Yes, that will return the value it is using for the z property zWinRMKrb5DisableRDNS

the second part was just to see if that property was set anywhere else and to see if we're not handling things correctly there.

i'm going to try and test this to make sure we're able to set that property correctly.

the mechanism for zenpython is whenever a datasource runs against a windows device.  in txwinrm_utils.py, we collect z properties, ip, hostname, etc. so that we can use what we need to build the krb5.conf file and then to connect to a device.  the krb5.conf file will change when a new kdc is added, a trusted realm is added, or when the disable rdns option is enabled/disabled(it's supposed to anyway).

whichever service is going to be connecting to a device will be the container that creates the config file.  so zenpython datasources will create it in the zenpython container, automatic zenmodeler runs will create it in zenmodeler, manual models will create it in zminion, etc.

if there are multiple kdcs defined for a realm, then, from what i understand and have tested, kerberos will go down the list until it successfully connects to a valid kdc for the realm.  if you have a kdc defined you no longer want to show up, you can add a '-' in front of the offending address in the zWinKDC field and we'll remove it from the list.  or, you can go through your locally defined configs and make sure that the offending address is removed, then restart zenoss.

Thanks
Dave

------------------------------
Dave Bouchillon Human
Zenoss
------------------------------

Subject: RE: Windows ZenPack - Kerberos settings config file
Author: Doug Syer
Posted: 2018-02-20 17:38

on my Z4 instance i just flagged krb5.conf read only its way to hard to manage the krb5.conf from the indivdual device settings if you have multiple domains and collectors at any scale.  for one or two domains setting it at the device class level is no big deal but for us its more complicated...once i flag the file read only it doesnt matter what is set in the kdcs...

and we just use saltstack to mange those files and the host file on the collector  planning on doing something similar in Z5/6 probably just change the config to point to a krb5.conf file in the shared z5 scripts directory that isnt supposted to change during upgrades or do on include file something like that.

i also have a challenge where i cant rely on customers dns to be correct or working so we manage things via host file although some day it would be nice to use srv records to be able to lookup the kdcs.  also in my case just because active directory sites and services will advice workstations to authenticate in certain orders to certain dcs in certain sites...doest mean i necessarily want the monitoring to work like that.

for example if i have two sites each with domain controllers and a collector in each site i'll probably set collector a to auth to the dcs in site a and collector b to authenticate to the dcs in site b rather than have each site try to authenticate across wan links etc.  in general i find that the small risk you will have of both dcs being down and you not having cached kerberos keys isnt worth the extra polling failure noise you will see with that extra redundancy.

------------------------------
Doug Syer
NWN Corporation
Waltham MA
------------------------------


< Previous
Zenoss 5 ZenPack compatibility 		  Next
Extracting Performance Data 		>