[ZODB-Dev] Missing loader for multidatabase refs?
Sidnei da Silva
sidnei at enfoldsystems.com
Fri Mar 30 12:01:15 EDT 2007
On 3/30/07, Jim Fulton <jim at zope.com> wrote:
> > What can be done to avoid that?
>
> By "that" I assume you mean dangling referenced.
I really meant avoiding the PosKeyError. ;)
> You could create extra references in the source database. For
> example, if you make a cross-database reference, you could also add a
> corresponding reference in the source database.
Right. And removing this corresponding difference when the object in
the target database goes away.
> > Not packing the databases at all?
>
> That will work too. :)
Not an option!
> > Would it be possible to make the loading of a cross-database reference
> > that is gone return some sort of 'BrokenObject' instead of a
> > PosKeyError, to work around the problem temporarily?
>
> Sure. I suppose the same could and perhaps should be done whenever
> you would get a POSKeyError.
That sounds like a candidate for a ZODB proposal.
> > Specially if the databases happen to be on different ZEO servers
> > *wink*.
>
> That is a complication, yes. You would need some sort of distributed
> GC protocol.
Ouch. :)
> >> - A non-GC pack that got rid of old records but didn't bother with
> >> GC.
> >> This would be advantagious for lots of folks independent of cross-
> >> database reference issues.
> >
> > Sounds like this would be the easiest way to solve the above issue?
>
> It is a fairly straightforward way, depending on the application.
The application is named Zope 2. You might have heard about it. :)
> Unfortunately, it's probably non-trivial, as the FileStorage packing
> code is fairly intense. :)
>
> I've been wanting to redo this code for some time to reduce the
> amount of disk I/O. That would certainly be an opportunity to make
> GC optional. But maybe it wouldn't be too hard to disable GC in the
> current implementation -- I don't know.
Let's see. GC right now is done when the object has 0 references
pointing to it. A quick workaround would be making sure the code that
computes the references (referencesf?) always returns at least one
reference pointing to it?
> >> Cross database references are pretty transparent and automatic.
> >>
> >> Maybe there should be an option to make them less so.
> >
> > Yes, such an option would be great.
>
> Maybe. It depends on what semantics you want. In fact, I don't know
> how Zope 2 is setting up multi-databases. If you don't want cross-
> database references at all, you could change the startup code to open
> separate databases without making them part of a multi-database.
>
> Hm, brainstorming:
>
> I suppose there could be could be an option that says you can't have
> a cross reference unless the source database has a root key equal to
> the object id of the source object with the value being the source
> object. Then, an intentional cross reference would be made like this::
>
> # Make a reference to foo, which is in another DB:
> foo._p_jar.root()[foo._p_oid] = foo
> self.x = foo
>
> The code that creates cross-database references would, if this option
> is available, do a check something like:
>
> if self.explicit_cross_database_references:
> if other._p_jar.root().get(other._p_oid) is not other:
> raise InvalidCrosDatabaseReference(...)
That looks good. If not for solving the problem completely, at least
for catching where in Zope implicit references might be created, and
possibly fixing those places.
> I suspect you know what the patch is at this point -- at least to get
> the pack done. Perhaps you can try it and let us know how it goes.
Packing worked fine, and packing the other database didn't generate a
PosKeyError... yet. That makes me really nervous though. I will try to
come up with a patch that creates a BrokenObject where a PosKeyError
would occur in the case of multi databases.
--
Sidnei da Silva
Enfold Systems http://enfoldsystems.com
Fax +1 832 201 8856 Office +1 713 942 2377 Ext 214
More information about the ZODB-Dev
mailing list