[Zope] Re: Index Problem - Index randomly breaks/changes

[Tres Seaver]

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

Ryan Smith wrote:

> Here are the 2 word answers, I go into more detail below.
>  - What kind of index?
> FieldIndex on an attribute I created called 'access'.  Access is a
> list of accesses a user has.
>  - What objects are you indexing?
> Any object with the 'access' attribute
>

Note that a FieldIndex is likely not to do what you want for
multi-valued attributes, such as lists or tuples:  you probably want a
KeywordIndex, which allows you do supply operators such as 'and' and 'or'.

>  - What kind of query are you making against the index?
> I noticed the problem when doing a folder listing in my Plone Site.

That isn't enough information:  we really need to see the arguments
being passed to the catalog's 'search' or 'searchResults' methods.

>  - What does "completely wrong" mean?
> The index is missing objects that have the 'access' attribute.  If an
> object's 'access' attribute is B it may be mapped in the index to A.

I think you need to give us a script to reproduce the behavior you are
seeing, e.g.:

1. In a folder, create a ZCatalog instance, 'catalog'.

2. On the catalog's "Indexes" tab, add a FieldIndex named 'access'
   (or try it with a Keyword index instead).

3. Create the following DTMLDocuments in the folder, each with the
   specified value for a lines property called 'access'::

   Document ID      access list
   -----------      ------------
   'foo'            ('baz',)
   'qux'            ('bam', 'baz')
   'splift'         ('bar', 'baz')

4. On the catalog's "Find Objects" tab, index the documents.

5. Add a PythonScript, 'show_me' to the folder, with the following
   body::

   catalog = context.catalog
   for access_value in ('baz',
                        ('baz',),
                        ('bam', 'baz'),
                        ('bar', 'baz')):
       print 'Querying value for access:', access_value
       brains = catalog.searchResults(access={'query': access_value,
                                              'operator': 'and'})
       for brain in brains:
           print brain.getPath()
       print
   return printed

6. Run the script, then tell us what you see, versus what you expected
   to see, when you run the script.  When I run it using a Keyword index
   and that query, I get the following, which is exactly what I expect::

   Querying value for access: baz
   /scratch/foo
   /scratch/qux
   /scratch/splift

   Querying value for access: ('baz',)
   /scratch/foo
   /scratch/qux
   /scratch/splift

   Querying value for access: ('bam', 'baz')
   /scratch/qux

   Querying value for access: ('bar', 'baz')
   /scratch/splift


> Here is some background on what I am doing -
> For a plone site, I needed a way to allow a user to only see objects
> if they had a certain set of roles.  I can't use the default security
> policy because we may want an object to require the roles Alpha AND
> Beta, but by default if a user has Alpha OR Beta they can see the
> object.
> 
> When creating an object, it is possible to specify and 'access'
> attribute that will require special accesses to view the object.
> 
> I handle if a user can view an object by dynamically adding them to a
> group, via a Group Plugin.
> 
> Problem was this didn't effect if the object showed up in a search
> result, like a folder listing.  Plone calls ZCatalog.searchResults()
> to get the metadata for all the objects in a folder.  I grab these
> results and remove objects that shouldn't be listed to a specific
> user.   To check if a user is allowed to view it, I check in the
> metadata, if my attribute is there I compare it to the users roles and
> remove it if needed
> 
> This all works most of the time.  But sometimes the index changes and
> the results I filter out will be wrong.

You need to show the filter you are using, because nothing in stock Zope
/ CMF / Plone works the way you describe.

>> The two log messages you post are effectively identical: some get
>> served from different ZODB connections, but that shouldn't cause any
>> difference in behavior. If you have one connection with "stale" data
>> for some objects (that would be a bug somewhere), then restarting Zope
>> might make the problem disappear, although we still wouldn't know how
>> that connection got into that state.
>>
> 
> The point I was trying to make with this was that if I reindex 100
> times.  I will get "DEBUG txn.-1223402608 commit <Connection at
> 0b5a01ec>" 80 times and "DEBUG txn.-1244382320 commit <Connection at
> 0965020c>" 20 times.  I realize they are nearly the same, but I get 2
> distinct values every time.  So if I dont restart Zope, it will always
> be one of these 2, nothing else.  And the re-indexes that are wrong
> always share the same value.

The two "values" you are seeing (I'm willing to bet money that the
transaction ID isn't the same) are two different ZODB connections:  some
requests get one connection from the pool, while others get another.


- --
===================================================================
Tres Seaver          +1 540-429-0999          tseaver at palladion.com
Palladion Software   "Excellence by Design"    http://palladion.com
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.6 (GNU/Linux)
Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org

iD8DBQFH0C2G+gerLs4ltQ4RAt/kAKCuo2wL6U+gKUutIvKAOMlUrPIhRACeKSRw
DbhimjaxI60zvyvazbjvTXY=
=RQj8
-----END PGP SIGNATURE-----