[Zope] Re: database conflict errors

[Jonathan]

----- Original Message ----- 
From: "Tres Seaver" <tseaver at palladion.com>
>
> Jonathan wrote:
>> During recent load testing of a new application 3.1% to 7.6% of all http 
>> requests resulted in conflict errors
>> (3.1% with 10 simultaneous users; 7.6% with 50 simultaneous users).
>>
>> The conflict error occurs when the application attempts to write a small 
>> image object into a temporary folder, and each conflict error generates 
>> the same traceback:
>>
>> ConflictError: database conflict error (oid 0x07, class 
>> Products.TemporaryFolder.TemporaryFolder.SimpleTemporaryContainer)
>>
>>
>>
>> I am running Zope 2.9.2 on CentOS 4.3 (linux).
>>
>> Does anyone have any ideas as to what I could do to reduce/eliminate 
>> these conflict errors?
>
> First, make sure that your application is not trying to overwrite the
> *same* image in each request.  If it needs to do that, then the
> conflicts are unavoidable.
>
> Next, making simultaneous writes into any naive container is going to
> cause conflicts.  You need to think carefully about how you want those
> writes to work, and plan to minimize conflicts:
>
>  - Use a BTreeFolder, rather than a normal OFS.Folder (or derivative).
>
>  - Arrange for the IDs of your images to be substantially different,
>    typically by mangling in a random number (or, for instance, the
>    microseconds value of the current timestamp).


The ids are assigned sequentially, so they never collide, but should id be 
'substantially different' to improve BTreeFolder2 performance?

As per your suggestion I added a BTreeFolder2 folder inside of the temporary 
folder and used that for image storage and ran some more load tests.  There 
was a significant improvement:  with 25 simultaneous users the error 
conflict error rate dropped from 4.4% to 0.6% and there was an unexpected 
side effect - a 20% improvement in server performance!!!

Thanks for the most excellent suggestion Tres!



Jonathan