[ZODB-Dev] Understanding the ZODB cache-size option

[Adam GROSZER]

Hello,

On top of that, be aware that those numbers 'cache-size' (or
cache-size-bytes) are for each connection. By default a Z3 ZODB can
have up to 7 connections in the pool. That means you might easily end
with 7 times the cache size set.

Most people tend to limit the number of connections too.

Monday, March 22, 2010, 8:01:30 PM, you wrote:

HS> On Mon, Mar 22, 2010 at 7:05 PM, Jeff Shell <jeff at bottlerocket.net> wrote:
>> Are there any metrics about how to set the ZODB 'cache-size' (or cache-size-bytes) option?
>> We've been using '5000' (arbitrarily chosen) for our Zope 3.4-ish app servers. We have access
>> to zc.z3monitor which can output the "number of objects in the object caches (combined)" and
>> "number of non-ghost objects in the object caches (combined)".
>>
>> But I don't know understand how to interpret those numbers and use them to make better
>> settings.

HS> ZODB cache size optimization is a typical case of black art. There's
HS> no real good way to find the perfect number. I would advise against
HS> using the "cache-size-bytes" option. There's a known critical problem
HS> with it, that is currently only fixed on the 3.9 SVN branch. So stick
HS> to the old object count cache-size for now.

HS> But generally database caching is a trade-off between performance and
HS> available RAM. As the upper limit, you could have your entire data set
HS> in each ZODB cache. So you could look at the number of persistent
HS> objects in the database and match your cache-size to that number.
HS> That's usually not want you want.

HS> As a real strategy, you should set up detailed monitoring of the
HS> server. Monitor and graph overall RAM usage, RAM usage per Zope
HS> process, number of DB loads and writes over time. Preferably include
HS> some way of measuring application request performance and track CPU
HS> and I/O usage on the server hosting the database.

HS> If you have those numbers, you can play around with the cache setting
HS> and increase it. See what impact it has on your application and data
HS> set. At some point you run out of memory and need to decrease the
HS> number or the increased cache size doesn't actually buy you any
HS> application performance anymore.

HS> For general Zope 3 application there are no rules of thumb that I
HS> know. The dataset and load patterns of the applications are too
HS> different to have any of those. It's affected a lot by the dataset and
HS> if you use ZODB blobs or another mechanism to store large binary
HS> content outside the DB. 5000 persistent objects including 1000 images
HS> of 5mb each are obviously very different, than 1000 BTree buckets
HS> containing only integers. One main advantage of blobs is that they
HS> aren't loaded into the ZODB connection caches, so they lower the
HS> memory requirements for applications with binary content a lot.

HS> In the Plone context I generally use something like "number of content
HS> objects in the catalog" + 5000 objects for the general application as
HS> a starting point. But that has a lot of assumptions on the type of
HS> content and the application in it.

HS> Hanno
HS> _______________________________________________
HS> For more information about ZODB, see the ZODB Wiki:
HS> http://www.zope.org/Wikis/ZODB/

HS> ZODB-Dev mailing list  -  ZODB-Dev at zope.org
HS> https://mail.zope.org/mailman/listinfo/zodb-dev


-- 
Best regards,
 Adam GROSZER                            mailto:agroszer at gmail.com
--
Quote of the day:
You can't cross a chasm in two small jumps.