[Zope] large installations and conflict errors

[Sascha Ottolski]

Am Dienstag, 9. August 2005 02:03 schrieb Andrew Langmead:
> (on these  
> machines, the ZEO client cache is set to 2GB and a cache flip occurs  
> maybe twice a week.)

I think this part to be quite interesting. We too are experimenting with the 
optimal cache sizes, where our (daily) packed Data.fs is "only" about 7 GB. 
Its only since friday that we updated to the latest 2.8-svn codebase, to get 
the latest fixes of the ZODB/ZEO cache tracing parts.

Our setup is one ZEO server, and two clients on seperate machines. A third 
client is on the same machine as the server, which is only used for 
admin/maintenance tasks.

What we've seen so far: we used to set the cache-size to 250 MB; and every 
know and than we have to observe that after a restart of a client a cache 
verification is necessary (while not always obvious, why this happened), 
which often takes "forever" (that is, 10 to 20 minutes are possible). And 
that during such a verification, the whole system seems to be down (in the 
sense that delivery of pages is slow as hell :-(). I think that in such a 
case simply deleting the cache file and starting with a fresh one is way 
faster, but have no hard numbers to prove that feeling.

Now we look what happens at different cache sizes. While it might be too early 
to really make conclusions, at least it might give some hints: We have set 
the cache on one machine to be 200 MB, and on the other to 100 MB. Up to now, 
even the 100 MB cache gives a hitrate of better than 98 % (!) most of the 
time.

What I hope is, that the smaller the cache size, the faster the verification, 
if necessary. Again, no real hard numbers to check this (seems to be hard to 
get some, sometimes the log says things like

2005-08-08T00:10:55 INFO ZEO.ClientStorage (27891) No verification necessary 
(last_inval_tid up-to-date)
------
2005-08-08T00:10:55 INFO ZEO.ClientStorage (27891) Waiting for cache 
verification to finish
------
2005-08-08T00:10:55 INFO ZEO.ClientStorage (27891) Verifying cache
------
2005-08-08T00:10:55 INFO ZEO.ClientStorage (27891) endVerify finishing
------
2005-08-08T00:10:55 INFO ZEO.ClientStorage (27891) endVerify finished

huh? no verifictation necesarry, but waiting for verification to finish? 
sometimes, the "endVerify" messages never appear; and you never know which of 
our three storages the messages are about...). We increased the servers 
invalidation-queue-size from 100 to 1000 along the way, don't know yet if 
this has any effect.

Having said all this, our scenario might not be common, as a major part of the 
traffic concentrates on a small part of the site, which might be the reason 
that even a small cache seems to be sufficient here.

I'm wondering if anybody else has seen long lasting cache verifications, which 
seems to block the whole system. Or if someone knows how to prevent this from 
happening at all...


Cheers,

Sascha