Zenoss-discovered Networks do not appear in list (4.2.5)

Hello,

I am having a very strange problem in Zenoss 4.2.5 with the Networks list. I went up to the latest zenup SP (671) but it did not fix it.

Basically, everything is there in Zope (confirmed through zport/manage/dmd), but through the UI, the networks do not appear. If I add a network manually, it appears for all users, but it is the only one in the list. If I were to add all my networks manually before Zenoss has created them, they would all show up, but that is not a good option because I have many networks. I cannot add the network after Zenoss has found it through modeling because it is already there and it can't add a duplicate.

I have no POSKeyErrors, zodb integrity is fine. relationscan shows fine, catalogs are all fine, no problems. I have been through every single setting in zope for a network is manually created and appears vs one that does not, and there is not a single difference, so all of the properties are correct. Zenoss appears to be behaving fine otherwise, besides some glitches in the Google Maps where it is using outdated values for zDrawMapLinks that I believe is the result of this issue.

I tried the JSON API and the networks facade in zendmd and was able to replicate this issue there. The getTree call in the JSON API to the network_router and the children call to the network facade both only return the manually-added networks and do not include the zenoss added ones in the list, so the issue is somewhere in the Zuul router/facade.

From going through the Zuul code, I see that it builds a 'cache' of the list of children. I am really suspecting that this 'cache' is somehow messed up. I have tried restarting memcached and all of zenoss and this issue is still here, so either there is some issue with memcached itself (how do I troubleshoot that) or there is some other cache that Zenoss is using that is persistent.

For more background, this server was originally at 2.5.2 and was upgraded to 3.x and 4.x through multiple releases, so there might be something in some config file that is messing things up that was set way back at 2.5.2 or 3.x and never changed.

I could really use the help, this issue is driving me crazy. We want to move to 5.x but are not ready yet.

Thanks

Michael Ducharme