zope 2.6.x, zcatalog and keyerrors
hi to all i have sites that's running on zope2.6 (cvs 2.6 branch) on linux and py2.1.3 and we have these errors[1] many times in the day. sometimes, the error is caused by adding stuff, sometimes while searching the catalog. updating the catalog clears the problem that arose from the searching of ZCatalog, but not while adding or modifying objects (that are CatalogAware. this is a ZClass that's been working from 2.1.6 up till 2.5.1) the only difference is that on previous incarnation, we did not use ZCTextIndex. is that where the problem may be? or is it Catalog? or me? thanks for any help --error Exception Type KeyError Exception Value -318038923 Traceback (innermost last): * Module ZPublisher.Publish, line 98, in publish * Module ZPublisher.mapply, line 88, in mapply * Module ZPublisher.Publish, line 39, in call_object * Module OFS.DTMLMethod, line 126, in __call__ * Module DocumentTemplate.DT_String, line 474, in __call__ * Module Products.ZCatalog.CatalogAwareness, line 125, in reindex_object * Module Products.ZCatalog.CatalogAwareness, line 121, in unindex_object * Module Products.ZCatalog.ZCatalog, line 530, in uncatalog_object * Module Products.ZCatalog.Catalog, line 395, in uncatalogObject * Module Products.ZCTextIndex.ZCTextIndex, line 166, in unindex_object * Module Products.ZCTextIndex.OkapiIndex, line 66, in unindex_doc * Module Products.ZCTextIndex.BaseIndex, line 165, in unindex_doc * Module Products.ZCTextIndex.BaseIndex, line 288, in _del_wordinfo
On Thursday 17 April 2003 08:28, Bakhtiar A Hamid wrote:
the only difference is that on previous incarnation, we did not use ZCTextIndex. is that where the problem may be? or is it Catalog? or me?
After add/delete objects you should do (disk-based code example): self.your_catalog.refreshCatalog(clear = 1) -- HTH, Bogdan "Our products just aren't engineered for security." -- Brian Valentine, senior vice-president of Windows development ... what exactly are Microsoft products engineered for? The garbage!
On Thursday 17 April 2003 06:03 pm, Bo M. Maryniuck wrote:
On Thursday 17 April 2003 08:28, Bakhtiar A Hamid wrote:
the only difference is that on previous incarnation, we did not use ZCTextIndex. is that where the problem may be? or is it Catalog? or me?
After add/delete objects you should do (disk-based code example):
self.your_catalog.refreshCatalog(clear = 1)
hmm .. but that would essentially be clearing the whole catalog, and recataloging (i.e update catalog) now, wouldn't it? i thought reindex_object() ought to do that per object. less overhead i subclassed CatalogAwareness, since then there was no CatalogPathAwareness. would that help? if so, i need to subclass that. can't try now since it's a zclass. will do it tomorrow, when there's more time at hands (there's a how to change base subclass, even for zclass :P). thanks
On Thursday 17 April 2003 02:28 am, Bakhtiar A Hamid wrote:
hi to all i have sites that's running on zope2.6 (cvs 2.6 branch) on linux and py2.1.3
and we have these errors[1] many times in the day. sometimes, the error is caused by adding stuff, sometimes while searching the catalog.
updating the catalog clears the problem that arose from the searching of ZCatalog, but not while adding or modifying objects (that are CatalogAware. this is a ZClass that's been working from 2.1.6 up till 2.5.1)
the only difference is that on previous incarnation, we did not use ZCTextIndex. is that where the problem may be? or is it Catalog? or me?
thanks for any help
This (obviously) should not happen. This might be a bug in ZCTextIndex, or possibly an inconsistency bug in BTrees exposed by ZCTextIndex. Here is what I suggest: Try to write a script which creates and destroys these ZClasses in sufficient quantity to expose the error (possibly with an external XMLRPC script). If you can get it to fail over time with any consistency, then I would be interested in seeing this script and your ZODB. In leiu of that, you could run the BTree consistency check right after the error occurs (but before recataloging) to see if the BTrees are getting hosed. From the python command line (run from Zope/lib/python) that would look like:
import Zope; a = Zope.app() idx = a.Catalog._catalog.getIndex('Name of ZCTextIndex') from BTrees.check import check check(idx.index._wordinfo) check(idx.index._docweight)
If either of those check() calls fail then it is likely a BTree bug. Otherwise its a bug somewhere else. -Casey
On Thursday 17 April 2003 09:40 pm, Casey Duncan wrote:
This (obviously) should not happen. This might be a bug in ZCTextIndex, or possibly an inconsistency bug in BTrees exposed by ZCTextIndex.
Here is what I suggest:
Try to write a script which creates and destroys these ZClasses in sufficient quantity to expose the error (possibly with an external XMLRPC script). If you can get it to fail over time with any consistency, then I would be interested in seeing this script and your ZODB.
i'll try and do this this weekend, or next week. ill post the result, if any
In leiu of that, you could run the BTree consistency check right after the error occurs (but before recataloging) to see if the BTrees are getting hosed. From the python command line (run from Zope/lib/python) that would
look like:
import Zope; a = Zope.app() idx = a.Catalog._catalog.getIndex('Name of ZCTextIndex') from BTrees.check import check check(idx.index._wordinfo) check(idx.index._docweight)
i have two ZCTextIndex for this particular app - bodytext, and title the test passed for bodytext, but fot title i got:
check(idx.index._wordinfo) Traceback (most recent call last): File "<stdin>", line 1, in ? File "/usr/local/Zope-2.6-cvs/lib/python/ZODB/Connection.py", line 525, in setstate raise ReadConflictError(object=object) ZODB.POSException.ReadConflictError: database read conflict error (oid 0000000001205126, class Products.ZCTextIndex.ZCTextIndex.ZCTextIndex)
If either of those check() calls fail then it is likely a BTree bug. Otherwise its a bug somewhere else.
-Casey
thanks
On Thursday 17 April 2003 09:27 pm, Bakhtiar A Hamid wrote: [snip]
look like:
import Zope; a = Zope.app() idx = a.Catalog._catalog.getIndex('Name of ZCTextIndex') from BTrees.check import check check(idx.index._wordinfo) check(idx.index._docweight)
i have two ZCTextIndex for this particular app - bodytext, and title the test passed for bodytext, but fot title i got:
check(idx.index._wordinfo) Traceback (most recent call last): File "<stdin>", line 1, in ? File "/usr/local/Zope-2.6-cvs/lib/python/ZODB/Connection.py", line 525, in setstate raise ReadConflictError(object=object) ZODB.POSException.ReadConflictError: database read conflict error (oid 0000000001205126, class Products.ZCTextIndex.ZCTextIndex.ZCTextIndex)
Hmmm, does it do that consistently? -Casey
On Fri, 18 Apr 2003, Casey Duncan wrote:
On Thursday 17 April 2003 09:27 pm, Bakhtiar A Hamid wrote: [snip]
look like:
import Zope; a = Zope.app() idx = a.Catalog._catalog.getIndex('Name of ZCTextIndex') from BTrees.check import check check(idx.index._wordinfo) check(idx.index._docweight)
i have two ZCTextIndex for this particular app - bodytext, and title the test passed for bodytext, but fot title i got:
check(idx.index._wordinfo) Traceback (most recent call last): File "<stdin>", line 1, in ? File "/usr/local/Zope-2.6-cvs/lib/python/ZODB/Connection.py", line 525, in setstate raise ReadConflictError(object=object) ZODB.POSException.ReadConflictError: database read conflict error (oid 0000000001205126, class Products.ZCTextIndex.ZCTextIndex.ZCTextIndex)
Hmmm, does it do that consistently?
the three times i did it, i got that. tonight, when i tried that again no tracebacks. i have recataloged this morning (after i run the test)
-Casey
_______________________________________________ Zope maillist - Zope@zope.org http://mail.zope.org/mailman/listinfo/zope ** No cross posts or HTML encoding! ** (Related lists - http://mail.zope.org/mailman/listinfo/zope-announce http://mail.zope.org/mailman/listinfo/zope-dev )
Bakhtiar A Hamid wrote at 2003-4-18 09:27 +0800:
i have two ZCTextIndex for this particular app - bodytext, and title the test passed for bodytext, but fot title i got:
check(idx.index._wordinfo) Traceback (most recent call last): File "<stdin>", line 1, in ? File "/usr/local/Zope-2.6-cvs/lib/python/ZODB/Connection.py", line 525, in setstate raise ReadConflictError(object=object) ZODB.POSException.ReadConflictError: database read conflict error (oid 0000000001205126, class Products.ZCTextIndex.ZCTextIndex.ZCTextIndex)
This is not a ZCatalog exception. It means that a concurrent process has changed the index and the ZODB was unable to provide a consistent state for it (with respect to other objects loaded in this transaction). Repeat your test when there are no writes to the index. An alternative would be to use my "No-more-ReadConflictError" patch <http://www.dieter.handshake.de/pyprojects/zope> Dieter
On Fri, 18 Apr 2003, Dieter Maurer wrote:
Bakhtiar A Hamid wrote at 2003-4-18 09:27 +0800:
i have two ZCTextIndex for this particular app - bodytext, and title the test passed for bodytext, but fot title i got:
check(idx.index._wordinfo) Traceback (most recent call last): File "<stdin>", line 1, in ? File "/usr/local/Zope-2.6-cvs/lib/python/ZODB/Connection.py", line 525, in setstate raise ReadConflictError(object=object) ZODB.POSException.ReadConflictError: database read conflict error (oid 0000000001205126, class Products.ZCTextIndex.ZCTextIndex.ZCTextIndex)
This is not a ZCatalog exception.
It means that a concurrent process has changed the index and the ZODB was unable to provide a consistent state for it (with respect to other objects loaded in this transaction).
Repeat your test when there are no writes to the index.
thanks dieter for your suggestion. i did as told, and i still got ReadConflict even when there's no write to thezodb (i'm 95% sure since there was no activity in the Undo, and it was the editors break time after i run Update catalog, there's no such error in either of the index. i want to use ZCTextIndex for the relevance, and some other features.. and would not want to use TextIndex, unless really really have to.
An alternative would be to use my "No-more-ReadConflictError" patch
i tried again, but i got mlformed patch (for the last file, but as you said i might patch it manually) when i did patched -p0 < ReadCommitted.pad
Dieter
thanks again
participants (5)
-
Bakhtiar A Hamid -
Bakhtiar A Hamid -
Bo M. Maryniuck -
Casey Duncan -
Dieter Maurer