[ZODB-Dev] design issue: job queue is concurrency bottleneck

[Steve Alexander]

John Belmonte wrote:
> I finally got through the first round of concurrency testing on my app. 
>    Despite being a high-write application, virtually no write conflicts 
> occur-- with one major exception.  I have a job queue, which is a BTree
> keyed by scheduled job time.  It's the only BTree in my app that doesn't 
> have random keys.  Incoming requests to my server tend to generate a few 
> jobs, and unfortunately they must be scheduled at various constant time 
> periods in the future.  As a result, simultaneous requests generate jobs 
> scheduled for approximately the same time, mapping to the same area of 
> the BTree, and causing write conflicts.
> 
> Short of adding some random offset to scheduled job times, which I feel 
> would not be very effective (depends on BTree size, etc.), does anyone 
> have ideas for designing around this bottleneck?

Write your own Queue class that derives from Persistent and implements 
conflict resolution.  This will be more or less successful depending on 
how few or many operations you allow on your queue.

--
Steve Alexander