On 3/6/08, Tres Seaver <tseaver@palladion.com> wrote:
Ryan Smith wrote:
Posted this to the Plone Mailing-list, but this is more of a problem I am having with Zope. There will be additional information here -
http://www.nabble.com/Indexing-Bug---Index-changes-and-is-not-always-correct...
Basically the problem I am having is that I create an index, but sometimes this index changes and is incorrect.
To test this I would constantly click the 'Clear and Rebuild' button under the Advanced Tab. Then I would browse my index under the Indexes tab.
After repeating this many many times, I would get 2 different results for my index (1 correct, the other completely wrong). I didn't change anything else, I just kept clicking rebuild.
We need more information:
- What kind of index?
- What objects are you indexing?
- What kind of query are you making against the index?
- What does "completely wrong" mean?
Here are the 2 word answers, I go into more detail below. - What kind of index? FieldIndex on an attribute I created called 'access'. Access is a list of accesses a user has. - What objects are you indexing? Any object with the 'access' attribute - What kind of query are you making against the index? I noticed the problem when doing a folder listing in my Plone Site. - What does "completely wrong" mean? The index is missing objects that have the 'access' attribute. If an object's 'access' attribute is B it may be mapped in the index to A. Here is some background on what I am doing - For a plone site, I needed a way to allow a user to only see objects if they had a certain set of roles. I can't use the default security policy because we may want an object to require the roles Alpha AND Beta, but by default if a user has Alpha OR Beta they can see the object. When creating an object, it is possible to specify and 'access' attribute that will require special accesses to view the object. I handle if a user can view an object by dynamically adding them to a group, via a Group Plugin. Problem was this didn't effect if the object showed up in a search result, like a folder listing. Plone calls ZCatalog.searchResults() to get the metadata for all the objects in a folder. I grab these results and remove objects that shouldn't be listed to a specific user. To check if a user is allowed to view it, I check in the metadata, if my attribute is there I compare it to the users roles and remove it if needed This all works most of the time. But sometimes the index changes and the results I filter out will be wrong.
I noticed something interesting in my debugging window. All rebuilds that resulted in a correct index had this - DEBUG txn.-1223402608 commit <Connection at 0b5a01ec> All rebuilds that resulted in an incorrect index had this - DEBUG txn.-1244382320 commit <Connection at 0965020c>
Something different is being done, I'm just not sure what or why.
Any suggestions as to why this is happening or where I should look to fix this would be greatly appreciated.
The two log messages you post are effectively identical: some get served from different ZODB connections, but that shouldn't cause any difference in behavior. If you have one connection with "stale" data for some objects (that would be a bug somewhere), then restarting Zope might make the problem disappear, although we still wouldn't know how that connection got into that state.
The point I was trying to make with this was that if I reindex 100 times. I will get "DEBUG txn.-1223402608 commit <Connection at 0b5a01ec>" 80 times and "DEBUG txn.-1244382320 commit <Connection at 0965020c>" 20 times. I realize they are nearly the same, but I get 2 distinct values every time. So if I dont restart Zope, it will always be one of these 2, nothing else. And the re-indexes that are wrong always share the same value.