[ZODB-Dev] Cache warm up time

[Leonardo Rochael Almeida]

The ZEO server *has* a cache already: it's called the Operating System
Disk Cache.

Remember: other than coordinating transactions, the only thing ZEO
really does is to sling the bytes of the pickles (and eventually the
bytes of blobs) across the ZEO connection to the client.

Any memory allocation that ZEO did to keep these bytes around in
memory would subtract from the memory available to the OS to do disk
caching of these exact same bytes.

(it could perhaps be made a bit more efficient if the underlying
FileStorage used mmap() instead of stdio calls to read these bytes.
This would eliminate a temporary copy of the data that ZEO does
between reading the bytes and sending them across the network).

Cheers,

Leo

On Fri, Mar 8, 2013 at 1:16 PM, Claudiu Saftoiu <csaftoiu at gmail.com> wrote:
> I'd be curious to know what your results are, whichever path you decide to
> take! Might help inform me as to what might help on my server...
>
> One thing I haven't yet understood is - how come the ZEO server itself
> doesn't have a cache? It seems that would be a logical place to put one as
> the ZEO server generally rarely gets restarted, at least for the use case of
> running both the ZEO server and the clients on the same machine.
>
>
> On Fri, Mar 8, 2013 at 1:46 AM, Roché Compaan <roche at upfrontsystems.co.za>
> wrote:
>>
>> Thanks, there are definitely some settings relating to the persistent
>> cache that I haven't tried before, simply because I've been avoiding
>> them.
>>
>> I'd still be interested to know if one can leverage the Relstorage
>> memcache code for a ZEO cache, so if Shane doesn't get around to it
>> I'll have a stab at it myself. Loading objects from a persistent cache
>> will still cause IO so to me it seems that it would be a big win to
>> keep the cache in memory even while restarting.
>>
>> --
>> Roché Compaan
>> Upfront Systems                   http://www.upfrontsystems.co.za
>>
>>
>>
>>
>> On Thu, Mar 7, 2013 at 9:35 PM, Leonardo Rochael Almeida
>> <leorochael at gmail.com> wrote:
>> > This mail from Jim at this list a couple of years ago was stocked full
>> > of nice tips:
>> >
>> > https://mail.zope.org/pipermail/zodb-dev/2011-May/014180.html
>> >
>> > In particular:
>> >
>> > - Yes, use persistent cache. Recent versions are reliable. Make it as
>> > large as resonable (e.g at most the size of your packed database, at
>> > least the size of objects that you want to be around after a restart).
>> >
>> > - Consider using zc.zlibstorage to compress the data that's stored in
>> > ZODB
>> >
>> > - set drop-cache-rather-verify to true on the client (avoid long
>> > restart time where your client is revalidating the ZEO cache)
>> >
>> > - set invalidation-age on the server to at least an hour or two so
>> >   that you deal with being disconnected from the storage server for a
>> >   reasonable period of time without having to verify.
>> >
>> > Cheers,
>> >
>> > Leo
>> >
>> > On Thu, Mar 7, 2013 at 3:54 PM, Roché Compaan
>> > <roche at upfrontsystems.co.za> wrote:
>> >> We have a setup that is running just fine when the caches are warm but
>> >> it takes several minutes after a restart before the cache warms up.
>> >> As per usual, big catalog indexes seem to be the problem.
>> >>
>> >> I was wondering about two things. Firstly, in 2011 in this thread
>> >> https://mail.zope.org/pipermail/zodb-dev/2011-October/014398.html
>> >> about zeo.memcache, Shane said that he could adapt the caching code in
>> >> RelStorage for ZEO. Shane do you still plan to do this? Do you think
>> >> an instance can restart without having to reload most objects into the
>> >> cache?
>> >>
>> >> Secondly, I was wondering to what extent using persistent caches can
>> >> improve cache warm up time and if persistent caches are usable or not,
>> >> given that at various times in the past, it was recommended that one
>> >> try and avoid them.
>> >>
>> >> --
>> >> Roché Compaan
>> >> Upfront Systems                   http://www.upfrontsystems.co.za
>> >> _______________________________________________
>> >> For more information about ZODB, see http://zodb.org/
>> >>
>> >> ZODB-Dev mailing list  -  ZODB-Dev at zope.org
>> >> https://mail.zope.org/mailman/listinfo/zodb-dev
>> _______________________________________________
>> For more information about ZODB, see http://zodb.org/
>>
>> ZODB-Dev mailing list  -  ZODB-Dev at zope.org
>> https://mail.zope.org/mailman/listinfo/zodb-dev
>
>