[Zope] Versioned connectors from ZODB

[Tim Peters]

[Etienne Labuschagne]
>>> . . . Versions solves this for me.

[Tim Peters]
>> Maybe like death would solve my problem with overdue taxes <wink>.

[Etienne]
> I did get the versioned connections to work (so far), BUT, I will
> definately take your word on it and seek another solution :)

If that works for you, don't let nay-sayers scare you away.  I don't
think there are any reports of version bugs open in the Zope collector
at present -- but that could just mean that everyone stays away from
them now.

>> <snip>  Like, e.g., in the ZODB 3.2 line,
>>
>>     otherdb = ZODB.DB(storage, cache_size=100, pool_size=2)
>>
>> Then connections obtained via otherdb.open() will hang if two threads
>> already have connections from `otherdb` (that's the effect of
>> `pool_size`), and will have ZODB memory caches that strive to keep no
>> more than 100 objects in memory across transaction boundaries (the
>> effect of `cache_size`).
>> <snip>
 
> to double check:
> 
> otherdb = ZODB.DB(existingdb._storage, cache_size=100, pool_size=2)
> 
> is ok?  It seems that you can create more than one DB instance that
> shares one storage object.

The code won't stop you from doing that, but as I said last time, I'd
use ZEO and use a fresh ClientStorage for each DB.  ZEO was designed
to support this kind of use; nothing else was.

> I hit upon the idea of creating another DB instance and sharing the
> storage object myself yesterday, but wasn't sure what the
> repurcussions will be.

Neither am I, if you don't use ZEO.  Normally I'd spend time digging
into the code trying to find answers, but I don't have time for that
today.  It's possible that if you asked on the zodb-dev list, Jim
Fulton or Jeremy Hylton would know more answers off the tops of their
heads.  Sorry, but I don't.

> Your post answers most of my questions.

At least the ZEO part did <wink>.

>  I have one left, though:  if I do decide to share the storage object
> (and not go ZEO for whatever reason), will the caches between the two
> DB objects not get out of sync?  In other words, will one DB object
> know to invalidate objects in it's caches should that object be
> changed through another DB instance?  I know ZEO does this for you,
> but I'd like to know what the case would be for two DBs in one
> process.

See above:  ZEO should work fine.  If you try to do it without ZEO,
I'm not sure what will happen.  I pointed out one "obvious" bad
consequence of trying to share a storage last time (that closing any
DB will close the storage across all DB's sharing that storage).

In general, invalidations get sent out by a DB, to all (& only) the
connections obtained from that DB.  So yes, if you're not using ZEO
(which goes on to broadcast invalidations to all connected clients),
caches can get out of synch across DBs.  But I don't know whether that
matters to you either.  For example, perhaps you're willing to create
a new DB whenever you need a temporary connection, and what you do
with it then is read-only and finishes quickly, or ... I just don't
know.

> My other option is to create the connections "by hand" (that way I can
> control the cache size easily) and keep my own little pool of
> connections with a modified close method that does not put my
> connections back into the "normal" pool.  But I'm afraid I may end up
> with a new can of worms that way.

I'd definitely advise against that.  The Connection constructor isn't
meant to be called outside of ZODB internals.  Note that you can't
even call it without passing a db, and there's an intricate dance
between Connection and DB methods that's mostly undocumented and hard
to get right.

....

> I use the ZODB directly, but from within Zope.  The connections are
> used in long-running processes that are not nescesarily
> browser-triggered.  Some of them are scheduled events that are started
> up in their own thread.  From there the need to get new connections to
> the ZODB.  I have quite a bit of experience working safely with
> multiple threads and the ZODB, so I'm sure I have that part right.  My
> problem had more to do with "cache contamination" and reserving
> "special connections" for specific processes.

Since there's no machinery aiming specifically at that, I'm afraid
it's bound to be painful one way or another -- except that, using ZEO,
it sounds quite straightforward.