[ZODB-Dev] Advice needed

[Dieter Maurer]

Roché Compaan wrote at 2006-6-25 11:13 +0200:
> ...
>> Careful design with respect to the granularity and locality
>> of persistent objects:
>> 
>>   Move groups of large and rarely used attributes out
>>   into persistent subobjects.
>
>Will this lead to smaller transaction sizes for objects that store large
>attributes as persistent subobjects?

It can mean fewer large transactions sizes as you get a large
transaction only when you modify the large attribute and
not when you do an innocent modification (on a small attribute).

It can also considerably reduce load time as you fetch
the subobject with the large attributes only when you need this
large content (and not when you happen to access the object
for other unrelated reasons). You memory consumption can go down
as well.

> ...
>This is interesting! ZCatalog metadata is stored in a IOBTree. Are you
>saying the values in the IOBTree should be wrapped in a class that
>subclasses Persistent, and not stored as raw values.

Yes -- at least as soon as the value gets larger.

>The ZCatalog and indexing in general seems to be the biggest problem
>with ZODB applications.

That's because they have to handle huge amounts of data and objects.

>BTrees do not seem to fix the problem.

If you speak about conflicts: they reduce the conflict probability
by about 30 to 80 times (depending on type). That's not too bad...

> ...
>Can somebody describe a ZODB indexing implementation

There is no ZODB indexing implementation. There is a "ZCatalog" one.

>where an
>application can index new objects concurrently? Given the above, this is
>probably not a stupid question.

We won't be able to ban conflict errors from the ZODB -- not even
when we consider only indexing applications.


I thought that it might be possible to extend the BTrees conflict
resolution which now it restricted to the leaf level to
one level up. This would reduce the conflict probability again
by a factor of 50 to 100. However, conflict resolution is a *very*
tricky task and it is quite easy to err and to introduce nasty highly
non-deterministic bugs. I haven't dared to attack this problem...



-- 
Dieter