[ZODB-Dev] Cache warm up time

[Roché Compaan]

A very simple alternative to prefetching would be to load the whole DB
into memory indiscriminately, if it is configured to do so. This way,
you can store your catalog in a separate db and request all of it from
the ZEO server and cache it straight away.

I'm still partial to a memcached cache that can survive a restart. The
first prize would be if it's possible to share the cache between zeo
clients.

--
Roché Compaan
Upfront Systems                   http://www.upfrontsystems.co.za


On Fri, Mar 8, 2013 at 7:50 PM, Laurence Rowe <l at lrowe.co.uk> wrote:
> On 8 March 2013 09:38, Claudiu Saftoiu <csaftoiu at gmail.com> wrote:
>> On Fri, Mar 8, 2013 at 12:31 PM, Leonardo Santagada <santagada at gmail.com>
>> wrote:
>>>
>>>
>>> On Fri, Mar 8, 2013 at 2:17 PM, Claudiu Saftoiu <csaftoiu at gmail.com>
>>> wrote:
>>>>
>>>> Once I know the difference I'll probably be able to answer this myself,
>>>> but I wonder why the ZEO server doesn't do the sort of caching that allow
>>>> the client to operate so quickly on the indices once they are loaded.
>>>
>>>
>>> IIRC zeo not only takes bytes from the storage and put them on a socket,
>>> it has a kind of heavy protocol for sending objects that has overhead on
>>> each object, so lots of small objects (that are 400mb in size) take a lot
>>> more time than sending a 400mb blob.
>>
>>
>> Ah that would make perfect sense. So ZEO and catalog indices really don't
>> mix well at all.
>
> The slowdown is largely because ZODB only loads objects one at a time.
> Loading a large catalogue requires paying that latency (network +
> software) each time, a 400mb of catalogue data may well equate to
> something like 10000 objects, and therefore 10000 loads in series.
> Once the data is loaded into the object cache you only need to fetch
> invalidated objects.
>
> It would be great if there was a way to advise ZODB in advance that
> certain objects would be required so it could fetch multiple object
> states in a single request to the storage server.
>
> Laurence
> _______________________________________________
> For more information about ZODB, see http://zodb.org/
>
> ZODB-Dev mailing list  -  ZODB-Dev at zope.org
> https://mail.zope.org/mailman/listinfo/zodb-dev