[ZODB-Dev] Re: so your coming from the rdbms world tutorial?

[Gary Poster]

On Feb 27, 2008, at 10:03 AM, Sean Allen wrote:

>
> On Feb 27, 2008, at 5:33 AM, Laurence Rowe wrote:
>
>> Sean Allen wrote:
>>> been looking for anything along those lines.
>>> in particular, strategies and gotchas for how to store objects.
>>> everything i've found is basically just a single type of object  
>>> being stored.
>>> i'm really interested in tutorials and information on the best  
>>> ways to setup
>>> large complicated hierarchies and all the various gotchas etc.
>>> anything like that exist?
>>
>> A good jumping off point is the wiki: http://wiki.zope.org/ZODB/Documentation
>>
>> The beauty of using the ZODB is that it really makes no difference  
>> whether you are storing homogeneous or heterogeneous hierarchies of  
>> objects.
>>
>> If you are likely to have a large number of objects in any one  
>> folder then make sure it is BTree based. This is a scalable data  
>> structure, so the whole list does not have to be loaded into memory  
>> to access a single child object.
>
> Been over that. Still have serious questions. And yes, we have a  
> large number of objects, something in the area of 16 million at  
> present if you were to map each relational table row to an object.
>
> The biggest concern I have is how do to the layout/storage so that  
> this slightly contrived example works:
>
> Product has a brand.
> There are many brands.
>
> How do I store so that I can find all products simply and all brands  
> simply and also so that changes in a brand instance are reflected when
> the product instance is deserialized. By 'simply' I mean that it  
> doesnt really work on our end to have to walk all Products looking
> for unique brands. Should just be able to go directly in and get  
> said brands ( using keys() or similar call ).
>
> If I create 'brand' and 'product' as btrees, then if i do something  
> like
>
> some_product.brand.name = 'something entirely different'
>
> and that brand already exists in 'brand', would it be updated? are  
> references maintained in that fashion?
> do we have to handle manually on update and creation?
>
> Note that we would just be using ZODB not Zope in this scenario.

Back references are not maintained automatically.

I'd identify two classic solutions to this sort of thing.

One is to make a custom mapping (using a BTree as the inner data  
structure) that maintains back-references when objects are placed in  
them or removed.  zope.app(.container? .folder? I'd have to look) has  
code that does this, along with firing events.  For simple stories  
like the one you describe here, that's what I'd probably recommend.   
It works to the strengths of the ZODB, which particularly shines in  
terms of readability when you just need to walk a tree of attributes  
to get what you want.

The other is to keep an external index, a la zc.extrinsicreference or  
zc.relation.

zc.extrinsicreference does not have too many dependencies beyond ZODB,  
and as long as zope.app.keyreference doesn't drag much along with it,  
might be usable as a library.  That said, it's also very simple, and  
could be used as a model for you, even if you don't use it directly.   
It would also be a reasonable choice for a simple situation like the  
one you describe.  It relies on events to update its data structures.

zc.relation an almost-released-revision of zc.relationship that  
drastically reduces dependencies--actually, it has no additional  
dependencies to ZODB, as you can see at http://svn.zope.org/zc.relation/trunk/setup.py?view=markup 
.  It's also a bit overwhelming and low-level: see the README: http://svn.zope.org/zc.relation/trunk/src/zc/relation/README.txt?view=auto 
  .  It doesn't hook anything up for you: you set the relationship  
catalog up and you arrange for it to be updated, via events or direct  
messages.  That said, if you need its power, it is well-tested and  
would be a good choice for some jobs from at least some perspectives  
(caveat read-or: I'm the author).

HTH

Gary