[ZODB-Dev] tweaking invalidation-queue-size

[Tim Peters]

[Paul Winkler]
> invalidation-queue-size is described in ZEO/component.xml like so:
>
>        The storage server keeps a queue of the objects modified by the
>         last N transactions, where N == invalidation_queue_size.  This
>         queue is used to speed client cache verification when a client
>         disconnects for a short period of time.
>
> Well, OK, but that doesn't tell me how to use it. When would I want to
> make it larger?

It's an optimization.  The ZEO server doesn't have a memory cache, except
for a list whose maximum length is invalidation_queue_size.  If, from
looking at ZEO client logs, you think you're spending more time doing ZEO
cache verification (which occurs at ZEO client startup time) than you'd
like, and can afford the RAM on the ZEO server, you can increase it.  If the
list is "too small", the ZEO server has to go rummaging thru disk files to
figure out which objects have changed since the last transaction the client
knew about, and presumably that's slower (nope, I've never measured it).

If you care about this, you want to make invalidation_queue_size at least as
large as the total number of transactions _other_ ZEO clients typically
commit when you lose (or bring down) a ZEO client, over the duration of the
latter's downtime.  That depends entirely on how many other clients you
have, how frequently they commit transactions, and how long a client
typically stays disconnected/down before it reconnects or is brought up
again.

> When would I want to make it smaller?

When you're less bothered by ZEO client cache verification time than you are
by burning ZEO server RAM.  Unfortunately, the unit for
invalidation_queue_size is number of transactions, and that has no obvious
relationship to RAM consumption (e.g., some transactions modify one object,
others may modify thousands).

> How do I verify that I've made things better or worse?

By looking at the time spent in clients doing startup cache verification.

> I just re-read Chris M.'s plone symposium slides,
> http://www.plope.com/static/presentations/zodb.pdf ...which says:
>
> """you may want to try tweaking the server's invalidation-queue-size if
> the connection between your ZEO server and your clients is less than
> reliable or if you frequently take your clients up and down. """
>
> Again, this is kinda vague about *how* to do this tweaking. I don't even
> know how to tell if I need any tweaking. All I know is that at least one
> of my zope servers has lost its ZEO connection several times recently, so
> it seems to be "less than reliable".

It's not a reliability knob, it's an optimization knob.  Changing it
shouldn't make your clients any more or less reliable, but if, for whatever
reason, your clients lose connections frequently (or, as Chris said, you
frequently restart them), and you're bothered by cache verification time
when they reconnect, turning the knob may decrease the time needed for that
step.
 
> Google didn't turn up anything of interest either.

Alas, I'm afraid that will remain true even after this reply <0.9 wink>.