[Zope] Re: Index Problem - Index randomly breaks/changes

[Ryan Smith]

On 3/6/08, Tres Seaver <tseaver at palladion.com> wrote:
>
>
> Ryan Smith wrote:
> > Posted this to the Plone Mailing-list, but this is more of a problem I
> > am having with Zope. There will be additional information here -
> >
> http://www.nabble.com/Indexing-Bug---Index-changes-and-is-not-always-correct.-td15830136s6742.html
> >
> > Basically the problem I am having is that I create an index, but
> > sometimes this index changes and is incorrect.
> >
> > To test this I would constantly click the 'Clear and Rebuild' button
> > under the Advanced Tab.
> > Then I would browse my index under the Indexes tab.
> >
> > After repeating this many many times, I would get 2 different results
> > for my index (1 correct, the other completely wrong). I didn't change
> > anything else, I just kept clicking rebuild.
>
>
> We need more information:
>
> - What kind of index?
>
> - What objects are you indexing?
>
> - What kind of query are you making against the index?
>
> - What does "completely wrong" mean?


Here are the 2 word answers, I go into more detail below.
 - What kind of index?
FieldIndex on an attribute I created called 'access'.  Access is a
list of accesses a user has.

 - What objects are you indexing?
Any object with the 'access' attribute

 - What kind of query are you making against the index?
I noticed the problem when doing a folder listing in my Plone Site.

 - What does "completely wrong" mean?
The index is missing objects that have the 'access' attribute.  If an
object's 'access' attribute is B it may be mapped in the index to A.

Here is some background on what I am doing -
For a plone site, I needed a way to allow a user to only see objects
if they had a certain set of roles.  I can't use the default security
policy because we may want an object to require the roles Alpha AND
Beta, but by default if a user has Alpha OR Beta they can see the
object.

When creating an object, it is possible to specify and 'access'
attribute that will require special accesses to view the object.

I handle if a user can view an object by dynamically adding them to a
group, via a Group Plugin.

Problem was this didn't effect if the object showed up in a search
result, like a folder listing.  Plone calls ZCatalog.searchResults()
to get the metadata for all the objects in a folder.  I grab these
results and remove objects that shouldn't be listed to a specific
user.   To check if a user is allowed to view it, I check in the
metadata, if my attribute is there I compare it to the users roles and
remove it if needed

This all works most of the time.  But sometimes the index changes and
the results I filter out will be wrong.

>
>
> > I noticed something interesting in my debugging window.
> > All rebuilds that resulted in a correct index had this -
> > DEBUG txn.-1223402608 commit <Connection at 0b5a01ec>
> > All rebuilds that resulted in an incorrect index had this -
> > DEBUG txn.-1244382320 commit <Connection at 0965020c>
> >
> > Something different is being done, I'm just not sure what or why.
> >
> > Any suggestions as to why this is happening or where I should look to
> > fix this would be greatly appreciated.
>
> The two log messages you post are effectively identical: some get
> served from different ZODB connections, but that shouldn't cause any
> difference in behavior. If you have one connection with "stale" data
> for some objects (that would be a bug somewhere), then restarting Zope
> might make the problem disappear, although we still wouldn't know how
> that connection got into that state.
>

The point I was trying to make with this was that if I reindex 100
times.  I will get "DEBUG txn.-1223402608 commit <Connection at
0b5a01ec>" 80 times and "DEBUG txn.-1244382320 commit <Connection at
0965020c>" 20 times.  I realize they are nearly the same, but I get 2
distinct values every time.  So if I dont restart Zope, it will always
be one of these 2, nothing else.  And the re-indexes that are wrong
always share the same value.