[ZODB-Dev] ZODB memory problems

[Tim Peters]

[Jeremy Hylton]
>> It's really too bad that ZEO only allows a single outstanding request.
>>  Restructuring the protocol to allow multiple simulatenous requests
>> was on the task list years ago, but the protocol implementation is so
>> complex I doubt it will get done :-(.  I can't help but think building
>> on top of an existing message/RPC layer would be profitable.  (What's
>> twisted's RPC layer?)  Or at least something less difficult to use than
>> asyncore.

[Shane Hathaway]
> Do you think the RPC layer is the source of the problem?

Probably depends on what "the problem" refers to?  If the protocol allows
for at most one outstanding request, then that's clearly _a_ bottleneck,
right?

I get the impression that Jim thinks the ZEO protocol is simple.  I don't
know -- because I haven't had to "fix bugs" in it recently, I know little
about it.  It sure isn't obvious from staring at 8000+ lines of ZEO code,
and Jeremy, Guido and I spent weeks a few years ago "fixing bugs" then.  I
felt pretty lost the whole time, never sure how many threads there were,
which code those threads may be executing, how exactly asyncore cooperated
(or fought) with the threads, or even clear on which RPC calls were
synchronous and which async.  There's so much machinery of various kinds
that it's hard to divine the _intent_ of it all.  I remember that sometimes
the letter "x" gets sent to a socket, and that's important <wink>.

It was my ignorant impression at the time that asyncore didn't make much
sense here, because mixing threads with asyncore is always a nightmare in
reality, and a ZEO server doesn't intend to service hundreds of clients
simultaneously regardless.

Anyway, to the extent that the RPC machinery is mysterious, that's "a
problem" of its own.

> I feel like the way ZODB uses references in pickles is the main thing
> that slows it down.  Even if you have a protocol that can request many
> objects at once, the unpickling machinery only asks for one at a time.

I'm unclear on what "the unpickling machinery" means.  The most obvious
meaning is cPickle, but that doesn't ask for anything.  In general, no
object's state gets unpickled before something in the application _needs_
its state, so unpickling is currently driven by the application.  Maybe
you're suggesting some form of "eager" unpickling/state-materialization?