[Zope] [Catalog/ZCatalog] external RID references: bad idea?

[Gary Poster]

From: "Casey Duncan" <cduncan@kaivo.com>
> Gary Poster wrote:
> >
> > Hi.
> >
> > Faced with a need to make large numbers of references to objects within
> > other objects, wanting to get certain information about the linked
objects
> > without waking them up in the ZODB, etc., and wanting to store the
> > references efficiently,
[snip]
>
> Another option to explore might be to store the oid (a unique 8 byte
> string generated for each object by the ZODB) as the reference along
> with some cached metadata that you'll lookup often.

[ snip]

Thanks!  I will explore it.  I haven't delved into the ZODB code much yet.

Precipitately, the advantages I see to the catalog RID possibility are two:
the metadata caching mechanism is already built, and objects are already
supposed to inform the catalog upon the need to reindex (keeping the
metadata fresh).  The advantages to the OID are that (as you informed me)
the interface elements I need are already in place (and therefore presumably
somewhat stable), and that an object can probably be referenced successfully
before it is cataloged--maybe at an earlier stage in the object's life.
I'll look into it more.

> Perhaps describing your application in more detail would yield better
> ideas...

OK.  Here goes.  I put a dashed line where I think you can probably stop
reading and get the gist. ;-)

While I keep an eye to contributing back to the community by making my
solutions as flexible as possible, I'm putting a super-bibliography for
musicians, especially vocalists, into Zope.  It stores objects describing
compositions, books, texts, recordings, publications, people, topics, and
other items.

On a simple level, I need the kind of referencing I describe for connecting
people objects as creators to other objects; for connecting any object to
another (particularly topics) in a "describes" relationship; for connecting
same-class objects in a parent-child relationship; and other similar tasks.
(Obviously, I'm coming from a bit of a RDBM background on this but I'm
enjoying the better modeling possible with the ZODB, among other things.)

When a composition object displays, for instance, it needs to both know the
name and address of all of it's creators, ideally without waking up the
creators yet.  Similarly, a person needs to know back links--what objects
claim me as a creator?  Rather than caching a page or an object, I have
decided it will be best to cache the relationships and metadata somehow.

----------------- Here there be dragons -------------------

The modelling for compositions is particularly complex, at least to me,
since I include instruments needed, if any, and voices needed, if any; the
voices themselves have high and low range extremes I am keeping track of,
and even multiple options for those.  If they are published, each song might
be transposed by a given number of half steps (producing a new set of the
high and low extremes for the composition).  If the composition's parent is
published and transposed, that means that will produce yet another set of
high and low extremes.

Displaying and searching by range extremes thus becomes quite complex, and a
high, high candidate for caching.  Even so, expecting my code to keep the
cached information fresh when the relationships are so far-flung makes me
nervous: I think I'll only be able to cache so far down the chain, and rely
on live checks (or at least secondary cached metadata checks) for the rest.

I'm figuring I'm going to need a new pluggable index, based on the work in
PathIndex, for the complicated range searches and some other needs; an
interlinking class that manages inter-object back and forward links behind
the scenes for caching and getting the cached metadata I described; and some
simple subclasses that will represent each of the data types.  I have plans
from there as well, but those are first steps.

OK, I'm stopping there; hoping that is enough, or more than enough; and
hoping it is useful.

> --
> | Casey Duncan
> | Kaivo, Inc.
> | cduncan@kaivo.com
> `------------------>

Thanks.

Gary