[ZODB-Dev] ZEO client persistent disk cache freshness

[Florent Guillaume]

Tim, first let me thank you for your invariably helpful and detailed 
responses.

To followup on your answer, it seems to mean that having a big 
persistent ZEO cache (say, 1GB) is not necessarily the best way to have 
a fast startup of the ZEO client, as it will have to verify its objects 
upon restart (if there's been a sufficiently long delay between stop 
and restart).

Hmm, I'll have to bench...

Thanks again,

Florent

On 16 Nov 2004, at 2:12, Tim Peters wrote:
> Let me answer the last first:
>
> [Florent Guillaume]
>> Sometimes ClientStorage logs a "Waiting for cache verification to 
>> finish"
>> which can take some time, is this related to this?
>
> Yes, it has entirely to do with this.
>
> Now back to the prelude <wink>:
>
>> I'm wondering about the ZEO client persistent disk cache.
>>
>> When a client stops and is later restarted and reconnects to the 
>> server,
>> how are issues of freshness dealt with, if at all? Is the data that is
>> stored in the cache automatically valid?
>
> Not necessarily at the instant you restart the client, but it will be 
> by the
> time cache verification (see above) completes.  While your client was 
> down,
> any number of objects may have been modified on the server, by other 
> ZEO
> clients.  The client you're starting asks the server which of the 
> objects it
> has cached are out of date, and purges those from its ZEO cache.  If 
> you
> reference such objects later, they'll be refetched from the ZEO server.
> OTOH, if the client has a cached object and the server tells the 
> client it's
> still current, the client cache holds on to that object.
>
> The details of this startup verification dance have changed with 
> virtually
> every ZODB/ZEO release, usually aiming at speeding it (in general, or 
> in
> common special cases; an example of the latter is a client that comes 
> up
> again soon after disconnecting, in which case a 
> cheaper-than-all-purpose
> cache verification algorithm can be used; an example of the former was
> adding an in-memory oid->serial mapping to FileStorage in 3.2, 
> primarily so
> that a ZEO server could know the current revision for an oid without 
> needing
> to do disk seeks).
>
>> Does this mean that "things" (pickles?) stored on the ZEO server never
>> change for a given unique identifier (OID? transaction id?) once they
>> have been created ?
>
> A persistent object has an oid (object id) that never changes.  It 
> retains
> this oid no matter how many times, or how radically, the object gets
> mutated.  Say you create an empty BTree B, and commit it.  It gets an 
> oid
> then (invented by the transaction that committed B's creation).  If 
> you add
> a million items to B, and commit that, the oid remains the same.  If 
> you
> commit changes to B a million more times, the oid still remains the 
> same.
>
> When you commit a change to a persistent object, there are two 
> "revisions"
> of that object:  the state the object had before the change, and the 
> state
> the object has after the change.  The oid does not change, and the 
> mapping
>
>     (oid, revision) -> pickle
>
> never changes after a commit (where "pickle" means the same thing as 
> "state"
> -- the pickle *is* (a serial-ized representation of) the object's 
> state).
> The process of packing a FileStorage throws away (physically removes) 
> the
> pickles for older revisions of an oid.
>
> Before ZODB 3.3, revisions were often called "serials", but in 3.3 
> they're
> the same as transaction ids (tids).  Well, OK, even before 3.3 serials 
> and
> tids were almost always exactly the same thing, but I guess it would 
> have
> made the code too easy to follow to give them the same name <wink>.
>
> Anyway,
>
> - The ZEO client cache knows, for each oid whose state it has cached,
>   the revision of that cached state.
>
> - The ZEO server knows, for each oid, the current revision of its
>   state == the tid of the transaction that most recently changed the
>   oid's state.
>
> - In general, cache verification compares these revisions, until the
>   client cache is sure it has purged itself of all non-current
>   revisions.
>
>
-- 
Florent Guillaume, Nuxeo (Paris, France)   CTO, Director of R&D
+33 1 40 33 71 59   http://nuxeo.com   fg at nuxeo.com