[ZODB-Dev] design issue: job queue is concurrency bottleneck

[John Belmonte]

Christian Robottom Reis wrote:
> On Thu, Mar 04, 2004 at 12:04:36AM -0500, John Belmonte wrote:
> 
>>I finally got through the first round of concurrency testing on my app. 
>>   Despite being a high-write application, virtually no write conflicts 
>>occur-- with one major exception.  I have a job queue, which is a BTree
>>keyed by scheduled job time.  It's the only BTree in my app that doesn't 
>>have random keys.  Incoming requests to my server tend to generate a few 
>>jobs, and unfortunately they must be scheduled at various constant time 
>>periods in the future.  As a result, simultaneous requests generate jobs 
>>scheduled for approximately the same time, mapping to the same area of 
>>the BTree, and causing write conflicts.
>>
>>Short of adding some random offset to scheduled job times, which I feel 
>>would not be very effective (depends on BTree size, etc.), does anyone 
>>have ideas for designing around this bottleneck?
> 
> So your data structure doesn't take into account the fact that multiple
> jobs can be allocated to the same timeslot?

Yes it does.  I left out the detail that job keys are a (time, ID) 
tuple, where ID is a unique string (for example, crypto hash of the 
request message that generated the job).

> Have you considered trying to add the key, catching ConflictError, and
> trying again adding a small time offset?

My conflict errors are not due to writing to the same keys.  From my 
limited understanding of BTrees, adding keys may cause buckets to be 
shuffled around, resulting in a write error.

-John


-- 
http:// if  ile.org/