[ZODB-Dev] Packing BIG fat Data.fs

[Jeremy Hylton]

>>>>> "SS" == Steve Spicklemire <steve@spvi.com> writes:

  SS> For folks not currently using ZEO (say folks running 5 or 6 Zope
  SS> instances on a single machine) it seems to me that the memory
  SS> tax of running ZEO would be significant, since each instance
  SS> will need it's own ZEO server, at least one client, and another
  SS> client while your script is running. When my single client is
  SS> running in object/memory equilibrium, how much memory is
  SS> consumed by the client and the server? How does this compare
  SS> with a "simple" stand alone Zope instance under the same
  SS> circumstances? I'd like to be wrong about this, since ZEO is
  SS> really slick!

Steve,

I'm not sure I understand the question.  A Zope instance running ZEO
results in more processes being run, but I'm not sure how that relates
to a "memory tax."  

If a non-ZEO setup loads a FileStorage in its custom_zodb, then there
is one extra process in the ZEO version: The custom_zodb loads a
ZEO.ClientStorage, which connects to a ZEO.StorageServer that is
started separately; the StorageServer loads the FileStorage.  But I
don't think your concern is about the memory consumption caused by an
extra Python process.

Are you worried about the ZEO client or server using a lot of extra
memory to serve objects?  I don't believe we've done a careful
analysis of that.  At a high-level, the client and server don't do
anything special to keep objects in memory, although each client can
use an on-disk cache for objects.  

If you look under the covers, there is some per-transaction overhead.
For each transaction, the ZEO client stores a list of OIDs and serial
numbers for the objects modified in the current transaction and an
on-disk log of all the objects modified by the current transaction.
The object pickles will be created in the client process and copied to
the server process via socket communication.  All the objects get
loaded at the same time in the client; objects are loaded one at a
time in the server.  When the update occurs, the ZEO server sends it
to the storage.  (Say it's a FileStorage.)  The storage writes it to
an on-disk log, then reads it back from the log when the transaction
commits.

To summarize all those details, I don't think there is a lot of extra
memory consumed as a result of using ZEO, aside from the extra memory
used by more Python processes.  There is a lot of extra disk I/O, and
there's obviously a performance penalty to be paid.

Jeremy