[ZODB-Dev] Cache warm up time

[Laurence Rowe]

On 8 March 2013 09:38, Claudiu Saftoiu <csaftoiu at gmail.com> wrote:
> On Fri, Mar 8, 2013 at 12:31 PM, Leonardo Santagada <santagada at gmail.com>
> wrote:
>>
>>
>> On Fri, Mar 8, 2013 at 2:17 PM, Claudiu Saftoiu <csaftoiu at gmail.com>
>> wrote:
>>>
>>> Once I know the difference I'll probably be able to answer this myself,
>>> but I wonder why the ZEO server doesn't do the sort of caching that allow
>>> the client to operate so quickly on the indices once they are loaded.
>>
>>
>> IIRC zeo not only takes bytes from the storage and put them on a socket,
>> it has a kind of heavy protocol for sending objects that has overhead on
>> each object, so lots of small objects (that are 400mb in size) take a lot
>> more time than sending a 400mb blob.
>
>
> Ah that would make perfect sense. So ZEO and catalog indices really don't
> mix well at all.

The slowdown is largely because ZODB only loads objects one at a time.
Loading a large catalogue requires paying that latency (network +
software) each time, a 400mb of catalogue data may well equate to
something like 10000 objects, and therefore 10000 loads in series.
Once the data is loaded into the object cache you only need to fetch
invalidated objects.

It would be great if there was a way to advise ZODB in advance that
certain objects would be required so it could fetch multiple object
states in a single request to the storage server.

Laurence