[ZODB-Dev] Re: [Bug] ZODB invalidation processing

[Jürgen Kartnaller]

We also run into this kind of problems.

The only save way to solve it was to serialize our index updates.

We do this by using one zope instance which runs only a 
lovely.remotetask task service. This service contains an indexing task. 
When indexing is needed we just create a new indexing job which is 
stored in the job list of the remote task service.

This solves two problems :

- conflicts
- low speed because of complex index value calculations

Btw the setup for this application contains 18 zope's each zope running 
one thread. 17 zope's are used to handle browser requests and one is 
used to handle the remote tasks (not only indexing tasks).

Jürgen

Tres Seaver wrote:
> -----BEGIN PGP SIGNED MESSAGE-----
> Hash: SHA1
> 
> Joachim Schmitz wrote:
>>>> Any suggestion for a temporary fix would be very welcome, since we get 
>>>> about 6000 conflict errors per day now about 15 % unresolved, and they 
>>>> are really killing our portal.
>>> A workaround might be to replace the DateIndexes for 'created' and
>>> 'modified' with FieldIndexes:  you will see a big jump in the number of
>>> keys in the index, but (perhaps) a reduction in conflicts (altheough
>>> there will be more bucket splits, which can also cause conflicts).
>>>
>> Unfortunately that does not work the FieldIndex also gives an conflict-error
>> <FieldIndex at Date>
>> <Products.ZCatalog.Catalog.Catalog object at 0xb250a6ec>
>> <CatalogTool at portal_catalog>
>> <CPSDefaultSite at uniben>
>> <Application at >
>> {'Application': <Application at >, 'ZGlobals': <BTrees._OOBTree.OOBTree 
>> object at 0xb242989c>}
>> 2007-06-01 20:23:41 INFO ZPublisher.Conflict ConflictError at 
>> /uniben/campus/students/A923157/study_course/create_level: database 
>> conflict error (oid 0x36c429, class BTrees._OOBTree.OOBTree, serial this 
>> txn started with 0x036e0590d58661ee 2007-06-01 17:52:50.044906, serial 
>> currently committed 0x036e059346ef95dd 2007-06-01 17:55:16.625597) (1 
>> conflicts (0 unresolved) since startup at Fri Jun  1 19:54:43 2007)
>>
>>
>>> A longer term solution might be to come up with a derived index type
>>> which used non-persistent objects (rather than the IITreeSet used by
>>> UnIndex) for the set of RIDs stored under each 'minute':  you could then
>>> inspect the state and do more aggressive conflict resolution than the
>>> stock implementation allows (see my other post).
>> any other shortterm suggestions.
> 
> QueueCatalog?
> 
> 
> Tres.
> - --
> ===================================================================
> Tres Seaver          +1 540-429-0999          tseaver at palladion.com
> Palladion Software   "Excellence by Design"    http://palladion.com
> -----BEGIN PGP SIGNATURE-----
> Version: GnuPG v1.4.6 (GNU/Linux)
> Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org
> 
> iD8DBQFGYGZU+gerLs4ltQ4RAk4BAJ46DKm1vLlygIqee1OxUjSPYF40pwCfSLvy
> mS+9UyTtv0OuNWuotzqk5Tg=
> =Fevy
> -----END PGP SIGNATURE-----
> 
> _______________________________________________
> Zope-Dev maillist  -  Zope-Dev at zope.org
> http://mail.zope.org/mailman/listinfo/zope-dev
> **  No cross posts or HTML encoding!  **
> (Related lists - 
>  http://mail.zope.org/mailman/listinfo/zope-announce
>  http://mail.zope.org/mailman/listinfo/zope )
>