[Zope] Re: database conflict errors

[Florent Guillaume]

Jean-Marc Orliaguet wrote:
> Tres Seaver wrote:
>> -----BEGIN PGP SIGNED MESSAGE-----
>> Hash: SHA1
>>
>> Jonathan wrote:
>>  
>>> ----- Original Message ----- From: "Tres Seaver" 
>>> <tseaver-npLdOuuzvjyaMJb+Lgu22Q at public.gmane.org>
>>>    
>>>> Jonathan wrote:
>>>>      
>>>>> During recent load testing of a new application 3.1% to 7.6% of all
>>>>> http requests resulted in conflict errors
>>>>> (3.1% with 10 simultaneous users; 7.6% with 50 simultaneous users).
>>>>>
>>>>> The conflict error occurs when the application attempts to write a
>>>>> small image object into a temporary folder, and each conflict error
>>>>> generates the same traceback:
>>>>>
>>>>> ConflictError: database conflict error (oid 0x07, class
>>>>> Products.TemporaryFolder.TemporaryFolder.SimpleTemporaryContainer)
>>>>>
>>>>>
>>>>>
>>>>> I am running Zope 2.9.2 on CentOS 4.3 (linux).
>>>>>
>>>>> Does anyone have any ideas as to what I could do to reduce/eliminate
>>>>> these conflict errors?
>>>>>         
>>>> First, make sure that your application is not trying to overwrite the
>>>> *same* image in each request.  If it needs to do that, then the
>>>> conflicts are unavoidable.
>>>>
>>>> Next, making simultaneous writes into any naive container is going to
>>>> cause conflicts.  You need to think carefully about how you want those
>>>> writes to work, and plan to minimize conflicts:
>>>>
>>>>  - Use a BTreeFolder, rather than a normal OFS.Folder (or derivative).
>>>>
>>>>  - Arrange for the IDs of your images to be substantially different,
>>>>    typically by mangling in a random number (or, for instance, the
>>>>    microseconds value of the current timestamp).
>>>>       
>>> The ids are assigned sequentially, so they never collide, but should id
>>> be 'substantially different' to improve BTreeFolder2 performance?
>>>     
>>
>> Sequential IDs can be problematic:  they lead to many more bucket splits
>> (and therefore conflicts) in the BTree.  Adding some "entropy" (like a
>> random number or the milliseconds of the time) *earlier* in the key than
>> any sequence number, will help keep splits down to a minimum.
>>
>>   
> 
> that's interesting. I did a test once to see what effect it would have 
> to add objects with a completely random id to a BTree folder (OOBTree in 
> that case) instead of using the object's type nam and add a number at 
> the end - and the result was the opposite in term of read performance. 
> Looking up keys was much faster if the ids followed a pattern like:
> 
> - something-1
> - something-2
> ...

Sure, in single-threaded mode this will decrease performance because the 
keys are spread randomly among all the buckets so many more buckets get 
written.

But in multi-threaded mode, this very spreading leads to better conflict 
resolution behavior.

Florent

-- 
Florent Guillaume, Nuxeo (Paris, France)   Director of R&D
+33 1 40 33 71 59   http://nuxeo.com   fg at nuxeo.com