Hate to ask, but *does* the PTR record for the server exist in DNS? Your post doesn't say, nor do you give the version of Zenoss in use. There were a fair number of bugs in earlier versions, a lot of which have been dealt with in version 6.2.x+
| Subject: |
RE: Intermittent - Server not found in Kerberos database |
| Author: |
Eric Ward |
| Posted: |
2018-11-08 15:47 |
I have worked through this one as well. I found the below configuration property in the guide and just upped it to 2 from the default of 1.
zWinRMKRBErrorThreshold
Having a poor network connection can cause erroneous kerberos error events to be sent which could cause confusion or false alarms. The default value is 1, which will always send an event on the first occurrence of an error. You can increase this value to send an event only when there have been x amount of occurrences of an error during collection, where x denotes the threshold number.
------------------------------
Eric Ward
Sys Admin
Restaurant Technologies
mendota heights MN
------------------------------
| Subject: |
RE: Intermittent - Server not found in Kerberos database |
| Author: |
Julius Beard |
| Posted: |
2018-11-09 08:26 |
I thought zWinRMKRBErrorThreshold might be involved as well. I've bumped that all the way to 10 on a few of the affected machines and haven't seen it make a difference. I tried increasing the
zWinRMConnectTimeout too, but that didn't work either.
| Subject: |
RE: Intermittent - Server not found in Kerberos database |
| Author: |
Julius Beard |
| Posted: |
2018-11-09 08:25 |
Unfortunately the PTR records do exist.
We are running the latest version of Core (6.2.1 r218) and I did update the Microsoft.Windows ZenPack to the latest (2.9.2).
I've also tried every combination of zWinRMKrb5DisableRDNS (at the /Server/Microsoft level), manually defining the
zWinRMServerName (both FQDN, none, and ${here/titleOrId}), and checking/adding SPNs.
I'd expect that if something was configured incorrectly that it would either work all of the time or none of the time. I don't understand why it's sporadic. Some servers model cleanly every time. Others are very intermittent.
| Subject: |
RE: Intermittent - Server not found in Kerberos database |
| Author: |
Julius Beard |
| Posted: |
2018-11-09 08:38 |
One other note...restarting the Zenoss server seems to clear a lot of these. Once the server restarts, it typically models most of the servers successfully for a short time.
| Subject: |
RE: Intermittent - Server not found in Kerberos database |
| Author: |
Jason Olson |
| Posted: |
2018-11-09 09:47 |
Another question; do you see any messages in the Event console saying something like "Missing counters in collection for xxx"? If so, that may be why you're missing data. I've found that if Zenoss trips over a failed collection, rather than handling it and carrying on, it throws an exception and halts collection completely and silently.....for a few hours. Then it does the periodic remodel of the servers and graphing and event log collection begins again.
Are any messages like that seen? As well, has the krb5-workstation packages been installed on the host? While it shouldn't be needed for proper operation as that should be handled by the docker images.....I find that it's required for consistent operation for Windows monitoring.
------------------------------
Jason Olson
------------------------------
Server not found in Kerberos database: Attempted to get ticket for HTTP@SERVERNAME. Ensure reverse DNS is correct.
We're monitoring 17 Windows Servers right now and seeing this error intermittently. I've stepped through all of the troubleshooting docs and posts I could find, but nothing seems to work. That same server will show that error, but periodically through the day info and events will come up. So I know it's working, it's just not consistent. At any give time I'll see the same error on 3-4 servers, but the others are all reporting fine.
Also seeing random occurences of Windows Event Log collection failing. That also is working, because we see events for the server, but a lot of this error as well.
WindowsEventLog: failed collection SERVERNAME
We are running the latest version of Zenoss Core on a dedicated machine that meets the hardware reqs.
Any ideas on what we can do to troubleshoot? If it was consistently not working, I'd imagine the config wasn't sound...but the fact that everything works (some of the time) seems to indicate some other type of issue.
| Subject: |
RE: Intermittent - Server not found in Kerberos database |
| Author: |
Julius Beard |
| Posted: |
2018-11-13 08:03 |
Nothing in the event console for missing counters. Typically I'll see the "server not found in Kerberos..." error along with a handful of actual events and sometimes the EventLog failed collection.
I didn't have
krb5-workstation loaded. I've just added that. I'll see if it helps.
It's also confusing to me that some servers never have the issue. A few of my Windows servers have perfect monitoring and never miss a model.
| Subject: |
RE: Intermittent - Server not found in Kerberos database |
| Author: |
Jason Olson |
| Posted: |
2018-11-13 09:39 |
Once installed, you'll need to restart the Zenoss application for any Kerberos changes to take effect. If that doesn't help, can you post the Configuration Properties Windows section (with any IPs, hostnames and user IDs changed to similar but invalid values)?
------------------------------
Jason Olson
------------------------------
| Subject: |
RE: Intermittent - Server not found in Kerberos database |
| Author: |
Julius Beard |
| Posted: |
2018-11-14 11:37 |
Same behavior. I restarted the whole server.
Here's the Windows section. I've tried different things in zWinRMServerName to no avail as well.
| Subject: |
RE: Intermittent - Server not found in Kerberos database |
| Author: |
Jason Olson |
| Posted: |
2018-11-14 12:27 |
Try undefining
zWinRMKrb5DisableRDNS,
zWinRMServerName (which I think is what's causing the issue; that should be defined with a string at the server level, not a variable at the container level),
zWinTrustedKDC, and
zWinTrustedRealm. Restart Zenoss within Control Centre, then give it an hour and see how it behaves?
If you want leave the variables set as they are, though, try undefining only
zWinRMServerName at the /Server/Microsoft level, then going to one of the servers causing grief and setting that variable with the server's fully-qualified domain name in the Configuration properties of that server, and see what happens after an hour or so?
------------------------------
Jason Olson
------------------------------
| Subject: |
RE: Intermittent - Server not found in Kerberos database |
| Author: |
Julius Beard |
| Posted: |
2018-11-19 08:08 |
I tried it both ways and got roughly the same results.
I did notice that on most of them when I blanked the
zWinTrustedKDC and zWinTrustedRealm values, and immediately ran a model, it would go through...but within 30 minutes or so, the error returned.
Any other ideas?
Very confusing as to why some Win servers on the same network would model without issue nearly 100% of the time with the same config settings.
| Subject: |
RE: Intermittent - Server not found in Kerberos database |
| Author: |
Jason Olson |
| Posted: |
2018-11-19 09:16 |
I'm sure you've checked already, but are there duplicate IP entries for any of the servers in DNS, either in the forward *or* the reverse lookup zones for the domain? I *have* seen this error before when there were duplicate hostnames for a specific IP address....where in AD DNS scavenging is off, and stale records get left behind when new ones are created. Even when in the forward zone. Not generally a problem for server records, but sometimes...
------------------------------
Jason Olson
------------------------------
| Subject: |
RE: Intermittent - Server not found in Kerberos database |
| Author: |
Julius Beard |
| Posted: |
2018-11-26 12:02 |
We do have some servers with IPv6 and IPv4 addresses listed. But that's true for some of the servers that are working as well. One of the servers that will consistently not model only has a single IP listed in forward and reverse zones.
Could it be a resource issue? I've noticed that some services will report an 'answering' status after the Zenoss server has been online for 7 or so days. When I restart the host, it seems like all modeling goes through without issue for a bit.
It's looking more an more like we're going to just stick with the old working 3.2 Zenoss install we have.
| Subject: |
RE: Intermittent - Server not found in Kerberos database |
| Author: |
Jason Olson |
| Posted: |
2018-11-30 09:22 |
I was thinking more multiple IPv4 addresses, rather than both v4 and v6, but that doesn't seem to be the case. Hmm....this is a good one. How does your WinRM configuration look? Are you applying it via Group Policy and if so, have you restricted the IP range that can communicate with the Windows servers and if
so, can you try unrestricting the range? Use the
* wildcard to allow all, just as a test?
------------------------------
Jason Olson
------------------------------
| Subject: |
RE: Intermittent - Server not found in Kerberos database |
| Author: |
Kyle Barstow |
| Posted: |
2019-04-17 16:03 |
Was this ever resolved? We've been experiencing intermittent issues with WinRM since around September. For us it seems to only affect devices trying to use the HTTPS scheme, and anything on HTTP works fine. We want to use HTTPS across the board, and I'm hopeful that if you found a solution to this problem it may be something we hadn't thought of on our end yet.
-Thanks
| Subject: |
RE: Intermittent - Server not found in Kerberos database |
| Author: |
Laurent Hemeryck |
| Posted: |
2019-04-18 10:41 |
Hello,
Are you sure that you have setup the correct SPN on the server ? It can sometimes be tricky to find the correct spelling and correct case.
By default, it exists for HTTP, but not for HTTPS.
To create it, it could something like:
setspn -S HTTPS/server.acme.com SERVER
or
setspn -S HTTPS/server SERVER
Check on the AD for this computer object what stands in the DNS Name attribute. You may have to test several options before it works.
Regs
------------------------------
Laurent Hemeryck
Monitoring Engineer
FedNot
------------------------------
| Subject: |
RE: Intermittent - Server not found in Kerberos database |
| Author: |
Kyle Barstow |
| Posted: |
2019-04-18 16:46 |
We don't have HTTP/HTTPS SPN's, only the WSMAN, and have the zWinUseWsmanSPN flag set to true for the device class. Monitoring via HTTPS worked flawlessly until around September, around the last time we updated the Windows ZenPack, with no other changes to our environment at the time.
Yesterday we did manually make an HTTPS/ SPN record on a test server, we restarted Zenoss today and remodeled and have been watching the device, but have seen no changes or improvements.
The most confusing part is that, like OP here, it works intermittently. It'll poll and graph data every once in a while and then just stop for several hours.
| Subject: |
RE: Intermittent - Server not found in Kerberos database |
| Author: |
Jay Stanley |
| Posted: |
2019-04-24 12:55 |
Microsoft ZP 2.9.3 is out, may want to try upgrading. I know there are a lot of fixes in it.
------------------------------
jstanley
------------------------------
| Subject: |
RE: Intermittent - Server not found in Kerberos database |
| Author: |
Josh Zenker |
| Posted: |
2019-09-26 16:02 |
I'm also experiencing this issue. We are on RM 5.3.3 and Microsoft Windows ZP 2.9.2, but I'll consider upgrading the ZP. We use HTTP because, in RHEL 7, Kerberos encrypts the authentication and the payloads anyway (which I confirmed with a packet capture). The issue occurs seemingly out of the blue and resolves itself without human intervention. However, the gaps in the graphs are often multiple hours long.
------------------------------
Josh Zenker
Linux System Administrator
Temple University
Philadelphia PA
------------------------------
| Subject: |
RE: Intermittent - Server not found in Kerberos database |
| Author: |
Kyle Barstow |
| Posted: |
2019-09-27 07:48 |
I can't speak on behalf of the OP, but we figured out part of our issue. We still get the intermittent kerberos errors similar to the first post, but we did figure out our issue with the intermittent graphing. The latest Windows ZenPack resolved the issue for us - v2.9.4. Give it a shot and see if it helps.
Kyle
------------------------------
Kyle Barstow
Ohio University
------------------------------
