[Zope] Re: database conflict errors

[Tres Seaver]

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

Jonathan wrote:
> 
> ----- Original Message ----- From: "Tres Seaver" <tseaver at palladion.com>
>>
>> Jonathan wrote:
>>> During recent load testing of a new application 3.1% to 7.6% of all
>>> http requests resulted in conflict errors
>>> (3.1% with 10 simultaneous users; 7.6% with 50 simultaneous users).
>>>
>>> The conflict error occurs when the application attempts to write a
>>> small image object into a temporary folder, and each conflict error
>>> generates the same traceback:
>>>
>>> ConflictError: database conflict error (oid 0x07, class
>>> Products.TemporaryFolder.TemporaryFolder.SimpleTemporaryContainer)
>>>
>>>
>>>
>>> I am running Zope 2.9.2 on CentOS 4.3 (linux).
>>>
>>> Does anyone have any ideas as to what I could do to reduce/eliminate
>>> these conflict errors?
>>
>> First, make sure that your application is not trying to overwrite the
>> *same* image in each request.  If it needs to do that, then the
>> conflicts are unavoidable.
>>
>> Next, making simultaneous writes into any naive container is going to
>> cause conflicts.  You need to think carefully about how you want those
>> writes to work, and plan to minimize conflicts:
>>
>>  - Use a BTreeFolder, rather than a normal OFS.Folder (or derivative).
>>
>>  - Arrange for the IDs of your images to be substantially different,
>>    typically by mangling in a random number (or, for instance, the
>>    microseconds value of the current timestamp).
> 
> 
> The ids are assigned sequentially, so they never collide, but should id
> be 'substantially different' to improve BTreeFolder2 performance?

Sequential IDs can be problematic:  they lead to many more bucket splits
(and therefore conflicts) in the BTree.  Adding some "entropy" (like a
random number or the milliseconds of the time) *earlier* in the key than
any sequence number, will help keep splits down to a minimum.

> As per your suggestion I added a BTreeFolder2 folder inside of the
> temporary folder and used that for image storage and ran some more load
> tests.  There was a significant improvement:  with 25 simultaneous users
> the error conflict error rate dropped from 4.4% to 0.6% and there was an
> unexpected side effect - a 20% improvement in server performance!!!
> 
> Thanks for the most excellent suggestion Tres!

You're welcome.


Tres.
- --
===================================================================
Tres Seaver          +1 202-558-7113          tseaver at palladion.com
Palladion Software   "Excellence by Design"    http://palladion.com
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.2.2 (GNU/Linux)
Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org

iD8DBQFEo/9H+gerLs4ltQ4RAqgRAJ9zWJMtCLWT1rkyoOM3GIPpovlccwCeOlnl
5yioYSdimjt7FyXWRODDA6A=
=tNlC
-----END PGP SIGNATURE-----