---

Subject:	We have instructions for
Author:	Andrew Kirch
Posted:	2014-10-10 09:33

We have instructions for repairing POSKeyErrors on the wiki http://wiki.zenoss.org/Repairing_Poskey_Errors

Andrew Kirch

akirch@gvit.com

Need Zenoss support, consulting or custom development Look no further. Email or PM me!

Ready for Distributed Topology (collectors) for Zenoss 5 Coming May 1st from GoVanguard

---

Subject:	me too, same exact object types
Author:	[Not Specified]
Posted:	2014-10-15 13:03

I detected a poskeyerror that fixposkeyerrors from the toolkit doesn't fix.. same exact objects missing as the original post (macaddresses and _lastPollSnmpUpTime). Not sure that's relevant, but it sure would be nice if the toolkit could find and fix these.

PATH: zport/dmd/Devices/Server/Windows/devices/MFRKNTMFXVIP03
TYPE:
OID: 0x0002ebd5 '\x00\x00\x00\x00\x00\x02\xeb\xd5' 191445
Refers to a missing object:
NAME: macaddresses
TYPE:
OID: 0x002db8db '\x00\x00\x00\x00\x00-\xb8\xdb' 2996443
2014-10-15 12:07:30,038 CRITICAL zodbscan: DANGLING REFERENCE (POSKeyError) FOUND:
PATH: zport/dmd/Devices/Server/Windows/devices/MFRKNTMFXVIP03
TYPE:
OID: 0x0002ebd5 '\x00\x00\x00\x00\x00\x02\xeb\xd5' 191445
Refers to a missing object:
NAME: _lastPollSnmpUpTime
TYPE:
OID: 0x0002ebdc '\x00\x00\x00\x00\x00\x02\xeb\xdc' 191452
2014-10-15 12:28:30,030 INFO zodbscan: 2 Dangling References were detected

---

Subject:	still dangling..
Author:	[Not Specified]
Posted:	2014-10-22 14:07

Well, I went ahead and "fixed" this problem by copying a zoid of type macaddresses and a zoid of type _lastPollSnmpUpTime to the proper zoid IDs... which then allowed me to delete the broken object without POSKeyErrors and then rediscover it. But it sure would be nice to be able to delete an object with dangling references without having to resolve the dangles first. Or that findposkeyerror could find all the POSKeyErrors that zodbscan can find....

---

Subject:	Warning, this is not something you're going to want to do
Author:	[Not Specified]
Posted:	2014-10-23 08:03

Warning, this is not something you're going to want to do
...unless you really know what you're doing
...and have made a backup before you do it
...and maybe don't do it in production.

I didn't even want to describe the procedure in my post because it's something totally unsupported and probably extremely risky. I don't work for Zenoss. I'm just a user of the core product.

Stop reading here if you'll get fired for screwing up your Zenoss.
-----

So, look through the mysql "zodb" database for another object (select state from object_state where zoid=[an object zoid]). This may take some trial and error till you find an object blob. The first few bytes of the blob contain the object type. In this case "cProducts.ZenModel.Device". Save that blob out to a file.

Use "od -c" to see a readable representation of the blob you saved. If you look thru the output of that, you'll see the object type at the beginning, then just look through until you find the field name you need (macaddresses). It'll look something like this:

q 244 U \f m a c a d d r e s s e s
q 245 ( U \b \0 \0 \0 \0 \0 055 270 222

When you see a \b \0 \0 \0 \0 \0 in the output, the next three bytes are a zoid in octal. Be aware that if your octal byte was a printable ascii character, od -c will have rendered it in printable ascii so it may look like this:
q 245 ( U \b \0 \0 \0 \0 \0 a 270 222
where ASCII a is 141 octal. Use od -b to just dump the blob to octal and compare.

So convert your three byte zoid to decimal:
055 oct = 2D hex
270 oct = B8 hex
333 oct = 92 hex
2DB892 (look familiar from a poskeyerror) = 2996370 decimal
That is the zoid of a macaddress type object that belongs to this example object.

Then I used phpMyAdmin to copy the zodb.object_state.zoid=2996370 row to have zoid 2996443 (see my post above with the output from zodbscan, you'll see that this is the missing macaddress zoid I posted about) The blob,tid,and state_size fields remaining the same.

Repeat for the other missing object type _lastSnmpPollTime. Find a good reference zoid from an object blob and then copy it.

** Again: This is probably not a very good idea.**

I left it that way only long enough to be able to delete the original object that had the two dangling references. After deleting I ran zodbscan, findposkeyerror, and zencatalogscan to ensure nothing else was dangling or broken.

Then I went ahead and rediscovered the original broken Device into zenoss.

Now that your Zenoss is a smoldering ruin, I hope this helps...

---

Subject:	Not sure what that means.
Author:	[Not Specified]
Posted:	2014-10-23 14:24

Maybe you copied a zoid blob of the wrong type. Also double check that the new row(zoid) you put in actually has the blob. If you used phpMyAdmin, I had to save it and reupload it using the file dialog in the blob column. If you used something other than phpMyAdmin to do the work, not sure what to suggest.

---

Subject:	And maybe I wasn't clear. I
Author:	[Not Specified]
Posted:	2014-10-23 14:27

And maybe I wasn't clear. I used the words "look through the mysql "zodb" database for another object "

And by object I meant a zoid that represents a "good" Device in your zenoss.

Then decode that to find THAT device's _lastSnmpPollTime and macaddress zoids.

Then copy THOSE zoids to the missing zoids on your broken Device.

You can probably get back to your former, but still broken state, by deleting the rows you inserted into object_state.

---

Subject:	Possible fix in zendmd for both issues
Author:	[Not Specified]
Posted:	2014-10-29 11:38

f you can get back to the original issue, something involving the following might help to un-block you:

$ zendmd

import sys
from ZODB.POSException import POSKeyError
from BTrees.OOBTree import OOSet

# set dev = the device with the issues in question

# then

dev.getMacAddressCache()
dev.macaddresses = OOSet()
commit()
sync()

# then

dev._delOb("_lastPollSnmpUpTime")
commit()
dev._lastPollSnmpUpTime = ZenStatus(0)
commit()
sync()

---

Subject:	I cannot figure that out, and
Author:	[Not Specified]
Posted:	2014-10-29 11:39

Cannot figure out how to do leading spaces/indentation/preformatted text, so I removed the try/except block - just run this all with no indentation, and make sure you set dev = the device in question before you run what I pasted above.

---

Subject:	test test test
Author:	Andrew Kirch
Posted:	2014-10-29 11:40

test
test
test

Andrew Kirch

akirch@gvit.com

Need Zenoss support, consulting or custom development Look no further. Email or PM me!

Ready for Distributed Topology (collectors) for Zenoss 5 Coming May 1st from GoVanguard

---

Subject:	Prokopowichj,
Author:	Jay Stanley
Posted:	2014-10-29 14:25

Prokopowichj,

dev = find('server')

---

Subject:	Prokopowichj, the issue is
Author:	[Not Specified]
Posted:	2014-10-30 15:44

Prokopowichj, the issue is that you keep posting in the instructions we're giving you for dev = without making the needed changes for your environment.

For instance, you put in, in your last example,
dev = find('server')

You need to replace 'server' with the name of your device. It's the same issue as earlier when you did dev = 'the server name'

It looks like previously your device name that is affected is 'server.ca'

Therefore,
dev = find('server.ca')

What happens with that command

---

Subject:	Found this in the main.log,
Author:	[Not Specified]
Posted:	2014-11-03 11:39

Found this in the main.log,

2014-11-03 11:12:11 ERROR ZODB.Connection: Couldn't load state for 0x14abc1
Traceback (most recent call last):
File "/opt/zenoss/lib/python/ZODB/Connection.py", line 860, in setstate
self._setstate(obj)
File "/opt/zenoss/lib/python/ZODB/Connection.py", line 914, in _setstate
self._reader.setGhostState(obj, p)
File "/opt/zenoss/lib/python/ZODB/serialize.py", line 613, in setGhostState
obj.__setstate__(state)
SystemError: new style getargs format but argument is not a tuple

---

Subject:	Prokopowichj, if you get the
Author:	[Not Specified]
Posted:	2014-11-03 13:59

Prokopowichj, if you get the dev set to your device, what do you get if you just type "dev" into zendmd What does it return You ran the toolbox zodbscan and findposkeyerror tools before, but I believe you did a revert in that process - have you run those since the revert Do either detect this issue

---

Subject:	Prokopowichj, in your zendmd
Author:	[Not Specified]
Posted:	2014-11-04 13:51

Prokopowichj, in your zendmd session, with dev the object reference:

dev.getMacAddressCache()

That still fails

---

Subject:	NOTE: "server" is just a
Author:	[Not Specified]
Posted:	2014-11-05 06:46

NOTE: "server" is just a place holder ... I have the actual hostname in the commands.

Here is the output,

$ zendmd
Welcome to the Zenoss dmd command shell!
'dmd' is bound to the DataRoot. 'zhelp()' to get a list of commands.
Use TAB-TAB to see a list of zendmd related commands.
Tab completion also works for objects -- hit tab after an object name and '.'
(eg dmd. + tab-key).
>>> import sys
>>> from ZODB.POSException import POSKeyError
>>> from BTrees.OOBTree import OOSet
>>> dev = find('server')
>>> dev

>>> dev.getMacAddressCache()
2014-11-05 07:44:40 ERROR ZODB.Connection Couldn't load state for 0x14abc1
Traceback (most recent call last):
File "/opt/zenoss/lib/python/ZODB/Connection.py", line 860, in setstate
self._setstate(obj)
File "/opt/zenoss/lib/python/ZODB/Connection.py", line 914, in _setstate
self._reader.setGhostState(obj, p)
File "/opt/zenoss/lib/python/ZODB/serialize.py", line 613, in setGhostState
obj.__setstate__(state)
SystemError: new style getargs format but argument is not a tuple
Traceback (most recent call last):
File "", line 1, in
File "/opt/zenoss/lib/python/ZODB/Connection.py", line 860, in setstate
self._setstate(obj)
File "/opt/zenoss/lib/python/ZODB/Connection.py", line 914, in _setstate
self._reader.setGhostState(obj, p)
File "/opt/zenoss/lib/python/ZODB/serialize.py", line 613, in setGhostState
obj.__setstate__(state)
SystemError: new style getargs format but argument is not a tuple

---

Subject:	After going through the
Author:	[Not Specified]
Posted:	2014-11-24 10:05

After going through the example ... I finally deleted the "bad" server. And running the zodbscan and findposkeyerror things look okay under /zport/dmd/Devices.

To fix the ZenModeler issue I had I had to delete (without removing stats) and re-add the device.

Thanks all for the help on the this issue.