[Zope] Re: database conflict errors

[Jean-Marc Orliaguet]

Tres Seaver wrote:
> -----BEGIN PGP SIGNED MESSAGE-----
> Hash: SHA1
>
> Jonathan wrote:
>   
>> ----- Original Message ----- From: "Tres Seaver" <tseaver at palladion.com>
>>     
>>> Jonathan wrote:
>>>       
>>>> During recent load testing of a new application 3.1% to 7.6% of all
>>>> http requests resulted in conflict errors
>>>> (3.1% with 10 simultaneous users; 7.6% with 50 simultaneous users).
>>>>
>>>> The conflict error occurs when the application attempts to write a
>>>> small image object into a temporary folder, and each conflict error
>>>> generates the same traceback:
>>>>
>>>> ConflictError: database conflict error (oid 0x07, class
>>>> Products.TemporaryFolder.TemporaryFolder.SimpleTemporaryContainer)
>>>>
>>>>
>>>>
>>>> I am running Zope 2.9.2 on CentOS 4.3 (linux).
>>>>
>>>> Does anyone have any ideas as to what I could do to reduce/eliminate
>>>> these conflict errors?
>>>>         
>>> First, make sure that your application is not trying to overwrite the
>>> *same* image in each request.  If it needs to do that, then the
>>> conflicts are unavoidable.
>>>
>>> Next, making simultaneous writes into any naive container is going to
>>> cause conflicts.  You need to think carefully about how you want those
>>> writes to work, and plan to minimize conflicts:
>>>
>>>  - Use a BTreeFolder, rather than a normal OFS.Folder (or derivative).
>>>
>>>  - Arrange for the IDs of your images to be substantially different,
>>>    typically by mangling in a random number (or, for instance, the
>>>    microseconds value of the current timestamp).
>>>       
>> The ids are assigned sequentially, so they never collide, but should id
>> be 'substantially different' to improve BTreeFolder2 performance?
>>     
>
> Sequential IDs can be problematic:  they lead to many more bucket splits
> (and therefore conflicts) in the BTree.  Adding some "entropy" (like a
> random number or the milliseconds of the time) *earlier* in the key than
> any sequence number, will help keep splits down to a minimum.
>
>   

that's interesting. I did a test once to see what effect it would have 
to add objects with a completely random id to a BTree folder (OOBTree in 
that case) instead of using the object's type nam and add a number at 
the end - and the result was the opposite in term of read performance. 
Looking up keys was much faster if the ids followed a pattern like:

- something-1
- something-2
...

I will try again.

/JM