Hmm.. it looks like the actual problem is a ConflictError during commit. What version of ZPatterns? Can you do subcommits along the way? I'm wondering if TransactionAgents might help here? -steve On Wednesday, December 12, 2001, at 09:18 AM, Jean Jordaan wrote:

Hi Chris

...

It can be fixed by rebuilding the catalog via "update catalog".

Hmm, sounds wonderful, but I've just done the following:

- Create new blank ZCatalog

- Run addIndexes script .. a lot of lines like::

catalog.manage_addIndex('ComponentVersions', 'KeywordIndex') catalog.manage_addIndex('CreatedBy', 'TextIndex')

- Run catalogEverything script .. a lot of loops like::

## ZClass instances ids = container.Reminders.getReminderIDs() for id in ids: object = container.Reminders[id] catalog.catalog_object(object, string.join(object.getPhysicalPath(), '/'))

## ZPatterns DataSkins ids = container.Contacts.Resellers.defaultRack.getPersistentItemIDs() for id in ids: object = container.Contacts.Resellers.defaultRack.getItem(id) catalog.catalog_object(object, string.join(object.getPhysicalPath(), '/'))

And directly thereafter::

2001-12-12T13:51:56 ERROR(200) UnKeywordIndex unindex_object could not remove documentId -215648165 from index ComponentVersions. This should not happen. Traceback (innermost last): File SearchIndex/UnIndex.py, line 236, in removeForwardIndexEntry KeyError: -215648165

This is when I attempt::

license_propertysheet = license.propertysheets.basic license_propertysheet.manage_changeProperties( SystemReminders = system_reminders)

on the same 'license' instance (ZPatterns DataSkin ZClass) as before. Entry -215648165 in the Catalog is the right one, indexed properly.

I see that I noticed the UnKeywordIndex errors before: http://lists.zope.org/pipermail/zope/2001-November/104190.html but then without a Zope Error being raised.

Currently, I'm seeing::

Error Type: KeyError Error Value: AM¿ñÛŠ

Traceback (innermost last): File ZPublisher/Publish.py, line 223, in publish_module File ZPublisher/Publish.py, line 187, in publish File Zope/__init__.py, line 221, in zpublisher_exception_hook (Object: ProviderContainer) File ZPublisher/Publish.py, line 175, in publish File Zope/__init__.py, line 235, in commit File ZODB/Transaction.py, line 300, in commit File ZODB/Connection.py, line 377, in commit (Info: (('BTrees.Length', 'Length'), '\000\000\000\000\000\002D\367', '')) File ZODB/FileStorage.py, line 667, in store (Object: /home/httpd/zope-instances/dev-creme/var/Data.fs) File ZODB/ConflictResolution.py, line 174, in tryToResolveConflict (Object: /home/httpd/zope-instances/dev-creme/var/Data.fs) File ZODB/ConflictResolution.py, line 104, in state (Object: /home/httpd/zope-instances/dev-creme/var/Data.fs) File ZODB/FileStorage.py, line 612, in loadSerial (Object: /home/httpd/zope-instances/dev-creme/var/Data.fs) KeyError: (see above)

This one is actually the showstopper.

Jean

_______________________________________________ ZPatterns mailing list ZPatterns@eby-sarna.com http://www.eby-sarna.com/mailman/listinfo/zpatterns

Hi Steve

Hmm.. it looks like the actual problem is a ConflictError during commit. What version of ZPatterns?

[jean@blommie python]$ cat Products/ZPatterns/version.txt ZPatterns-0-4-3p2

Can you do subcommits along the way?

Uhm, I have no idea ..

I'm wondering if TransactionAgents might help here?

I haven't used TransactionAgents yet, so don't know .. This isn't a pure ZPatterns app .. the 'license' instance at issue is a DataSkin, but earlier in the method which does the 'manage_changeProperties' call on it, I also do this a couple of times, creating some reminders:: password_reminder_id = container.addReminder() container.editReminder(password_reminder_id) Reminder is a ZClass subclassing XronDTMLMethod. 'addReminder' calls 'manage_edit', and 'editReminder' calls both 'manage_edit' and 'manage_changeProperties'. Is it OK to do this multiple times in a transaction? Jean

The two errors may be related but the FileStorage key error is definitely the one you need to fix first. This indicates that the FileStorage index has become out-of-date or that the bits that make up the data in the FileStorage are not in the place that the index thinks they are. Dont worry about the other error until you have this sorted out. Try a Data.fs backup or use the utilities/fsrecover.py script (or Ty's tranalyzer) to attempt to detect where there is an index inconsistency and fix the Data.fs. This may be due to flaky hardware. Hmm, you're not using your FileStorage from an NFS mount are you? ----- Original Message ----- From: "Jean Jordaan" <jean@upfrontsystems.co.za> To: "Chris McDonough" <chrism@zope.com> Cc: <zpatterns@eby-sarna.com>; <zope@zope.org> Sent: Wednesday, December 12, 2001 9:18 AM Subject: RE: [Zope] ZCatalog: UnKeywordIndex error Hi Chris

It can be fixed by rebuilding the catalog via "update catalog".

Hmm, sounds wonderful, but I've just done the following: - Create new blank ZCatalog - Run addIndexes script .. a lot of lines like:: catalog.manage_addIndex('ComponentVersions', 'KeywordIndex') catalog.manage_addIndex('CreatedBy', 'TextIndex') - Run catalogEverything script .. a lot of loops like:: ## ZClass instances ids = container.Reminders.getReminderIDs() for id in ids: object = container.Reminders[id] catalog.catalog_object(object, string.join(object.getPhysicalPath(), '/')) ## ZPatterns DataSkins ids = container.Contacts.Resellers.defaultRack.getPersistentItemIDs() for id in ids: object = container.Contacts.Resellers.defaultRack.getItem(id) catalog.catalog_object(object, string.join(object.getPhysicalPath(), '/')) And directly thereafter:: 2001-12-12T13:51:56 ERROR(200) UnKeywordIndex unindex_object could not remove documentId -215648165 from index ComponentVersions. This should not happen. Traceback (innermost last): File SearchIndex/UnIndex.py, line 236, in removeForwardIndexEntry KeyError: -215648165 This is when I attempt:: license_propertysheet = license.propertysheets.basic license_propertysheet.manage_changeProperties( SystemReminders = system_reminders) on the same 'license' instance (ZPatterns DataSkin ZClass) as before. Entry -215648165 in the Catalog is the right one, indexed properly. I see that I noticed the UnKeywordIndex errors before: http://lists.zope.org/pipermail/zope/2001-November/104190.html but then without a Zope Error being raised. Currently, I'm seeing:: Error Type: KeyError Error Value: AM¿ñÛS Traceback (innermost last): File ZPublisher/Publish.py, line 223, in publish_module File ZPublisher/Publish.py, line 187, in publish File Zope/__init__.py, line 221, in zpublisher_exception_hook (Object: ProviderContainer) File ZPublisher/Publish.py, line 175, in publish File Zope/__init__.py, line 235, in commit File ZODB/Transaction.py, line 300, in commit File ZODB/Connection.py, line 377, in commit (Info: (('BTrees.Length', 'Length'), '\000\000\000\000\000\002D\367', '')) File ZODB/FileStorage.py, line 667, in store (Object: /home/httpd/zope-instances/dev-creme/var/Data.fs) File ZODB/ConflictResolution.py, line 174, in tryToResolveConflict (Object: /home/httpd/zope-instances/dev-creme/var/Data.fs) File ZODB/ConflictResolution.py, line 104, in state (Object: /home/httpd/zope-instances/dev-creme/var/Data.fs) File ZODB/FileStorage.py, line 612, in loadSerial (Object: /home/httpd/zope-instances/dev-creme/var/Data.fs) KeyError: (see above) This one is actually the showstopper. Jean _______________________________________________ Zope maillist - Zope@zope.org http://lists.zope.org/mailman/listinfo/zope ** No cross posts or HTML encoding! ** (Related lists - http://lists.zope.org/mailman/listinfo/zope-announce http://lists.zope.org/mailman/listinfo/zope-dev )

OK, none of that points to any particular problem. :-( ----- Original Message ----- From: "Jean Jordaan" <jean@upfrontsystems.co.za> To: "Chris McDonough" <chrism@zope.com> Cc: <zpatterns@eby-sarna.com>; <zope@zope.org> Sent: Wednesday, December 12, 2001 11:00 AM Subject: RE: [Zope] ZCatalog: UnKeywordIndex error

Hi Chris ..

...

Try a Data.fs backup or use the utilities/fsrecover.py script (or Ty's tranalyzer) to attempt to detect where there is an index inconsistency and fix the Data.fs.

OK, I'm off to try .. (about to go in way over my head here :)

[jean@blommie python]$ pwd /usr/local/zope/creme/2-3-0/lib/python [jean@blommie python]$ python ZODB/fsrecover.py ../../var/Data.fs Nothing to recover

Tranalyzer shows me 72000-odd lines of OIDs, but it doesn't complain about anything .. here's the first couple of lines of the last call to the method that raises the error:

TID: 3414D8832DE9622 @ 25617740 obs 82 len 93317 By jean "/Reminders/doRemindersForLicense" OID: 18f99 len 4178 19015, 18fa7, 18fb0, 18fb2 20c5b, 20c5a, 18f9a, 19003 <snip>

...

This may be due to flaky hardware. Hmm, you're not using your FileStorage from an NFS mount are you?

Nope, it's a local file on a Linux box, ext2fs filesystem.

Jean

Yikes. I dont understand what caused the conflict error if only one thread was accessing the app. That said, I think the only thing to do is to wait until you see problems again, and maybe we can attack it fresh then. - C ----- Original Message ----- From: "Jean Jordaan" <jean@upfrontsystems.co.za> To: "Chris McDonough" <chrism@zope.com> Cc: <zope@zope.org> Sent: Thursday, December 13, 2001 8:07 AM Subject: RE: [Zope] ZCatalog: UnKeywordIndex error

Hi Chris

Final note on this topic .. I've solved the problem for me by doing this:

WHAT I HAD

'Licenses.extendLicense' called - 'Reminders.doRemindersForLicense(license_id)'

'Reminders.doRemindersForLicense' called - 'manage_changeProperties' and 'catalog_object' on new Reminder instances, - and called 'manage_changeProperties' on a License instance, which triggered SkinScript calls to 'uncatalog_object' and 'catalog_object' on the License instance.

This caused the conflict error which started this thread.

WHAT I HAVE NOW

'Licenses.extendLicense' calls - 'reminder_ids = Reminders.doRemindersForLicense(license_id)'

'Reminders.doRemindersForLicense' calls - 'manage_changeProperties' and 'catalog_object' on new Reminder instances. - It does *not* call any methods that change the License instance, but instead returns values to the calling method.

Now 'Licenses.extendLicense' proceeds to call - 'manage_changeProperties' on the License instance, etc.

STATUS QUO

The UnKeywordIndex error is still showing up in the logs, but it's not halting the app. The KeyError, which did, is gone. I'm still not sure how to formulate the caveat in this case ..

Thanks for all the help, Jean

_______________________________________________ Zope maillist - Zope@zope.org http://lists.zope.org/mailman/listinfo/zope ** No cross posts or HTML encoding! ** (Related lists - http://lists.zope.org/mailman/listinfo/zope-announce http://lists.zope.org/mailman/listinfo/zope-dev )