Bulletproof ZCatalog proposal
I have written a proposal. http://dev.zope.org/Wikis/DevSite/Proposals/ArmoredCatalog Could it be that we're involved in yet another Battle of Fredericksburg and we're trying to save innocent ZCatalogs from a ZODB conflict? Nawww... ;-) Shane
On Wed, 30 May 2001, Shane Hathaway wrote:
I have written a proposal.
So, if I understand correctly, when this is implemented, will the ZCatalog always resolve conflicts and never raise ConflictErrors? If there are no ConflictErrors in the main object system? I think I'm missing the point here.
On Thu, 31 May 2001, Erik Enge wrote:
On Wed, 30 May 2001, Shane Hathaway wrote:
I have written a proposal.
So, if I understand correctly, when this is implemented, will the ZCatalog always resolve conflicts and never raise ConflictErrors? If there are no ConflictErrors in the main object system?
I think I'm missing the point here.
How did you miss the point? That's exactly the point! :-) Right now ZCatalog randomly generates ConflictErrors even if there are no conflicts in the data being indexed. It's quite rare, however, and there's machinery that retries the entire transaction up to three times when it happens. But I want it to be more reliable, especially so that ZCatalog can be used in ZODB applications outside Zope. Shane
On Thu, 31 May 2001 10:03:31 -0400 (EDT), Shane Hathaway <shane@digicool.com> wrote:
On Thu, 31 May 2001, Erik Enge wrote:
On Wed, 30 May 2001, Shane Hathaway wrote:
I have written a proposal.
So, if I understand correctly, when this is implemented, will the ZCatalog always resolve conflicts and never raise ConflictErrors? If there are no ConflictErrors in the main object system?
I think I'm missing the point here.
How did you miss the point? That's exactly the point! :-)
Right now ZCatalog randomly generates ConflictErrors even if there are no conflicts in the data being indexed. It's quite rare, however, and there's machinery that retries the entire transaction up to three times when it happens.
If the current mechanism is reliable, and efficient (given that conflict/retries are rare) then I dont understand the motivation....
But I want it to be more reliable, especially so that ZCatalog can be used in ZODB applications outside Zope.
Is it because other non-Zope applications might not be so proficient at performing retries? Toby Dickenson tdickenson@geminidataloggers.com
On Thu, 31 May 2001, Toby Dickenson wrote:
On Thu, 31 May 2001 10:03:31 -0400 (EDT), Shane Hathaway <shane@digicool.com> wrote:
Right now ZCatalog randomly generates ConflictErrors even if there are no conflicts in the data being indexed. It's quite rare, however, and there's machinery that retries the entire transaction up to three times when it happens.
If the current mechanism is reliable, and efficient (given that conflict/retries are rare) then I dont understand the motivation....
The system becomes less reliable as server load increases. Large catalog updates (where every object is reindexed) also generate a lot of conflicts.
But I want it to be more reliable, especially so that ZCatalog can be used in ZODB applications outside Zope.
Is it because other non-Zope applications might not be so proficient at performing retries?
Yes. Long-running transactions are especially problematic. Shane
On Thu, 31 May 2001 10:36:55 -0400 (EDT), Shane Hathaway <shane@digicool.com> wrote:
On Thu, 31 May 2001, Toby Dickenson wrote:
On Thu, 31 May 2001 10:03:31 -0400 (EDT), Shane Hathaway <shane@digicool.com> wrote:
Right now ZCatalog randomly generates ConflictErrors even if there are no conflicts in the data being indexed. It's quite rare, however, and there's machinery that retries the entire transaction up to three times when it happens.
If the current mechanism is reliable, and efficient (given that conflict/retries are rare) then I dont understand the motivation....
The system becomes less reliable as server load increases.
Large catalog updates (where every object is reindexed) also generate a lot of conflicts.
Is that last bit true? I thought 'Update Catalog' created *new* indexes. There might be a conflict on the root catalog opbject, but inside the index BTrees. ???? Toby Dickenson tdickenson@geminidataloggers.com
My last post didnt make much sense. This time ill include all the words in my sentence....
Large catalog updates (where every object is reindexed) also generate a lot of conflicts.
Is that last bit true? I thought 'Update Catalog' created *new* indexes. There might be a conflict on the root catalog object, but not on the index BTrees. Toby Dickenson tdickenson@geminidataloggers.com
On Thursday 07 June 2001 07:43, Toby Dickenson wrote:
Large catalog updates (where every object is reindexed) also generate a lot of conflicts.
Is that last bit true? I thought 'Update Catalog' created *new* indexes. There might be a conflict on the root catalog object, but not on the index BTrees.
I should say large catalog updates tend to generate a conflict. It really doesn't matter how many conflicts there are. Within a single transaction, 1 conflict is as bad as 100. One thing I didn't make clear in the proposal is that I'm interested in repurposing ZCatalog as a general ZODB indexing mechanism and essentially moving it down from the application layer to the database layer. Catalogs would be updated automatically (in a lazy fashion to avoid performance penalties). There would potentially be a large number of catalogs, though, so the number of conflicts would increase significantly. But I think I have a solution for all of the issues in conflict resolution. If you, or anyone else, is also interested in this, please show support (if only by saying "please do this"!) :-) I can't work on it unless I have a reason to. Shane
Shane Hathaway wrote:
But I think I have a solution for all of the issues in conflict resolution. If you, or anyone else, is also interested in this, please show support (if only by saying "please do this"!) :-) I can't work on it unless I have a reason to.
Please do this! :-) I'm particularly interested in ZCatalog becoming a standard ZODB feature, and I know there are otehrs on ZODB-dev who are dying for this right now... cheers, Chris
Me too! Me too! John On Thursday 07 June 2001 08:49, Chris Withers wrote:
Shane Hathaway wrote:
But I think I have a solution for all of the issues in conflict resolution. If you, or anyone else, is also interested in this, please show support (if only by saying "please do this"!) :-) I can't work on it unless I have a reason to.
Please do this! :-)
I'm particularly interested in ZCatalog becoming a standard ZODB feature, and I know there are otehrs on ZODB-dev who are dying for this right now...
cheers,
Chris
_______________________________________________ Zope-Dev maillist - Zope-Dev@zope.org http://lists.zope.org/mailman/listinfo/zope-dev ** No cross posts or HTML encoding! ** (Related lists - http://lists.zope.org/mailman/listinfo/zope-announce http://lists.zope.org/mailman/listinfo/zope )
-- . . . . . . . . . . . . . . . . . . . . . . . . John D. Heintz | Senior Engineer 1016 La Posada Dr. | Suite 240 | Austin TX 78752 T 512.633.1198 | jheintz@isogen.com w w w . d a t a c h a n n e l . c o m
On Thu, 7 Jun 2001, Shane Hathaway wrote:
It really doesn't matter how many conflicts there are. Within a single transaction, 1 conflict is as bad as 100.
Why?
But I think I have a solution for all of the issues in conflict resolution.
Whee!
If you, or anyone else, is also interested in this, please show support (if only by saying "please do this"!) :-) I can't work on it unless I have a reason to.
Please do this!
On Thursday 07 June 2001 11:51, Erik Enge wrote:
On Thu, 7 Jun 2001, Shane Hathaway wrote:
It really doesn't matter how many conflicts there are. Within a single transaction, 1 conflict is as bad as 100.
Why?
Because in Zope it means the whole request is processed again (which can lead to strange behavior if non-transactional data is changed by the request). In other apps it might mean messy exception handling.
If you, or anyone else, is also interested in this, please show support (if only by saying "please do this"!) :-) I can't work on it unless I have a reason to.
Please do this!
I should have stated that it also needs to be a priority. Within DC we've been using better project management and it's getting harder to do projects just because we might think they'll be useful for someone. Shane
At 09:34 AM 6/7/01 -0400, Shane Hathaway wrote:
One thing I didn't make clear in the proposal is that I'm interested in repurposing ZCatalog as a general ZODB indexing mechanism and essentially moving it down from the application layer to the database layer. Catalogs would be updated automatically (in a lazy fashion to avoid performance penalties). There would potentially be a large number of catalogs, though, so the number of conflicts would increase significantly.
But I think I have a solution for all of the issues in conflict resolution. If you, or anyone else, is also interested in this, please show support (if only by saying "please do this"!) :-) I can't work on it unless I have a reason to.
Sounds cool. At one time Ty and I thought we might do something like this by adding to a linked list of "pending updates" which the catalog would use to alter its search results, until the list got "too long", at which point it would post the actual updates. The only catch was that this would still produce conflicts at the head end of the linked list. :( Of course, that was in the days before ZODB conflict resolution. Nowadays, you could probably implement it as a simple sequence object with the conflict resolution method implemented. But then there'd be the question of how to resolve conflicts if more than one thread or ZEO client decided to apply the queued changes... a 100 conflicts vs 1 situation. Ugh. Anyway, I'd be really interested in seeing "a solution for all the issues in conflict resolution". Will it help with arguments, too? How about world peace? ;)
--On Thursday, June 07, 2001 09:34:54 AM -0400 Shane Hathaway <shane@digicool.com> wrote:
But I think I have a solution for all of the issues in conflict resolution. If you, or anyone else, is also interested in this, please show support (if only by saying "please do this"!) :-) I can't work on it unless I have a reason to.
Please do this! It is the sort of thing that can save us and others a lot of time by keeping use of Zope simple and stable.
On Thu, 31 May 2001, Shane Hathaway wrote:
How did you miss the point? That's exactly the point! :-)
Oh :)
Right now ZCatalog randomly generates ConflictErrors even if there are no conflicts in the data being indexed.
This is why I think I've missed a/the point: wouldn't it be better to figure out why ZCatalog randomly generates these ConflictErrors instead of making them disappear using other means?
On Thu, 31 May 2001, Erik Enge wrote:
On Thu, 31 May 2001, Shane Hathaway wrote:
How did you miss the point? That's exactly the point! :-)
Oh :)
Right now ZCatalog randomly generates ConflictErrors even if there are no conflicts in the data being indexed.
This is why I think I've missed a/the point: wouldn't it be better to figure out why ZCatalog randomly generates these ConflictErrors instead of making them disappear using other means?
We know why they happen. They are only *apparently* random--they actually make perfect sense. (Maybe I didn't make that clear.) If you concurrently modify objects that happen to be placed near enough each other in one of the indexes then a conflict will occur. We have a partial solution in place which tries to merge the buckets that make up the index, but this solution can't solve the whole problem. Think about what SQL databases do. You can add entries to a table simultaneously from different connections. As long as the contents of the entries don't conflict, there will not be any kind of conflict in the indexes either. ZCatalog is to ZODB what indexes are to SQL databases. I'd like ZCatalog to have the same level of reliability. Shane
participants (7)
-
Chris Withers -
Dan L. Pierson -
Erik Enge -
John D. Heintz -
Phillip J. Eby -
Shane Hathaway -
Toby Dickenson