[ZODB-Dev] what's the latest on zodb/zeo+memcached?

[Jim Fulton]

On Tue, Jan 15, 2013 at 2:45 PM, Claudiu Saftoiu <csaftoiu at gmail.com> wrote:
> On Tue, Jan 15, 2013 at 2:40 PM, Jim Fulton <jim at zope.com> wrote:
>>
>> So, first, a concise partial answer to a previous question:
>>
>> ZODB provides an in-memory object cache.  This is non-persistent.
>> If you restart, it is lost.  There is a cache per connection and the
>> cache size is limited by both object count and total object size (as
>> estimated by database record size).
>>
>> ZEO also provides a disk-based cache of database records read
>> from the server.  This is normally much larger than the in-memory cache.
>> It can be configured to be persistent.  If you're using blobs, then there
>> is a separate blob cache.
>>
>> On Tue, Jan 15, 2013 at 2:15 PM, Claudiu Saftoiu <csaftoiu at gmail.com>
>> wrote:
>> >> You can't cause a specific object (or collection of objects) to stay
>> >> ion the cache, but if you're working set is small enough to fit in
>> >> the memory or client cache, you can get the same effect.
>> >
>> >
>> > That makes sense. So, is there any way to give ZODB a Persistent and
>> > tell it
>> > "load everything about the object now for this transaction" so  that the
>> > cache mechanism then gets triggered, or do I have to do a custom search
>> > through every aspect of the object, touching all Persistents it touches,
>> > etc, in order to get everything loaded? Essentially, when  the server
>> > restarts, I'd like to pre-load all these objects (my cache is indeed big
>> > enough), so that if a few hours later someone makes a request that uses
>> > it,
>> > the objects will already be cached instead of starting to be cached
>> > right
>> > then.
>>
>> ZODB doesn't provide any pre-warming facility.  This would be
>> application dependent.
>>
>> You're probably better off using a persistent ZEO cache
>> and letting the cache fill with objects you actually use.
>
>
> Okay, that makes sense. Would that be a server-side cache, or a client-side
> cache?

There are no server-side caches (other than the OS disk cache).

> I believe I've already succeeded in getting a client-side persistent
> disk-based cache to work (my zodb_indexdb_uri is
> "zeo://%(here)s/zeo_indexdb.sock?cache_size=2000MB&connection_cache_size=500000&connection_pool_size=5&var=zeocache&client=index"),

This configuration syntax isn't part of ZODB.  I'm not familiar with
the options there.

> but this doesn't seem to be what you're referring to as that is exactly the
> same size as the in-memory cache.

I doubt it, but who knows?

> Could you provide some pointers as to how
> to get a persistent disk-based cache on the ZEO server, if that is what you
> meant?

ZODB is configured via ZConfig.  The parameters are defined here:

  https://github.com/zopefoundation/ZODB/blob/master/src/ZODB/component.xml

Not too readable, but at least precise. :/

Look at the parameters for zodb and zeoclient.

Here's an example:

<zodb main>
  cache-size 100000
  pool-size 7

  <zeoclient>
    blob-cache-size 1GB
    blob-dir /home/zope/foo-classifieds/blob-cache
    cache-size 2GB
    server das-head1.foo.zope.net:11100
    server das-head2.foo.zope.net:11100
  </zeoclient>
</zodb>

If you want to use this syntax with paste, see:

  http://pypi.python.org/pypi/zc.zodbwsgi

Jim

-- 
Jim Fulton
http://www.linkedin.com/in/jimfulton
Jerky is better than bacon! http://zo.pe/Kqm