[ZODB-Dev] zeo client persistent cache files?

[Tim Peters]

[Chris McDonough, on Zope 2.7.3, using a persistent ZEO client cache but
 finding that it doesn't appear to be doing him any good across client
 restarts]

...

[Tim Peters]
>> + How big is your ZEO client cache (measured in bytes).

[Chris]
> 1 GB.

>> + How big is your ZODB pickle cache (measured in # of objects).

> 5000 objects

>> + How big are your ~700 objects in aggregate (measured in bytes).

> 64K apiece * 700 = ~ 42MB.

So in your first scenario (loading the same 700 objects across transactions
without restarting the client) it's likely all requests after the first were
satisfied out of ZODB's in-memory pickle cache.  I can personally guarantee
that the ZEO cache will never be as fast as that <wink>.

...

> I am indeed using a fast local network.  But as I said before I think
> that the ZEO persistent cache isn't being consulted due to log output on
> the ZEO server when I send a request which I believe should be serviced
> by the disk cache (the pattern of log messages is indisinguishable from
> the log messages resulting from real load requests).

Right, *now* that surprises me too.  Before, I was guessing that your ZEO
client cache was so small (as the default size is) that there was no reason
to believe it *could* hold a useful amount of data between restarts.  But it
turns out it's plenty big enough to hold all your useful data.  Then I agree
it sounds like something's not working right.

> I don't know the ins and outs of how the cache does its invalidation
> though, so perhaps this is an incorrect assumption.

I'm afraid I don't either -- I have a dim understanding of how ZEO works,
and even dimmer wrt ZODB 3.2 than 3.3 (the ZEO cache implementation was
replaced, and I did some work on the latter but none on the former).

It's a general truth (across the ZODB and ZEO caches, and across 3.2 and
3.3) that invalidations get sent out only when an object is modified.
There's also a cache verification dance at ZEO client startup time, which is
a natural area to suspect.

Try invoking .cacheMinimize() on the Connection at the start of some
transaction.  That will ghostify all the objects in the pickle cache.  The
point then is to see whether the next requests get satisfied out of the ZEO
client cache, or are refetched from the ZEO server.  If the former, then we
have to suspect that ZEO client cache startup verification isn't working
correctly, but that the ZEO client cache is working OK after startup; if the
latter, the ZEO client cache is plain useless <wink>.