[ZODB-Dev] Spatial indices / zodb standard library

[Jim Fulton]

I drafted this response a few days ago and decided not to send it,
however, since you keep pushing this issue...

On Wed, Jun 16, 2010 at 2:00 PM, Nitro <nitro at dr-code.org> wrote:
> Am 16.06.2010, 17:55 Uhr, schrieb Jim Fulton <jim at zope.com>:
>
>> On Wed, Jun 16, 2010 at 11:45 AM, Nitro <nitro at dr-code.org> wrote:
>> ...
>>> Would it make sense to add a default spatial index to ZODB?
>>
>> No.
>>
>> Which isn't to say that a spatial wouldn't be a good idea. It just
>> doesn't need to be part of ZODB.  While it isn't an urgent priority,
>> in the future, I want to make ZODB smaller.
>>
>> We have a packaging system. We don't need no stink'n batteries
>> included. :)
>
> I agree with you about separating ZODB and an indexing package on the
> technical level.
>
> I was more referring to the semantical level. What I miss a bit in the
> zodb/zope/plone world is a focus on certain packages. Usually there are
> 1-10 packages for a given task maintained by different people. In most of
> the cases this is a plain waste of resources. Time could often be better
> spent by picking a good default package and then collaborating together to
> improve it.
>
> Imo this also applies to ZODB. During my current zodb related work, I've
> come across a number of useful or interesting packages:
>
> - indexing: zcatalog, zcatalog standalone, IndexedCatalog
> - spatial indexing: zgeo.spatialindex, Quadtree
> - searching/relations: gocept.objectquery, dm.incrementalsearch ,
> zc.relation, zc.relationship, gocept.reference
> - storage history: zc.beforestorage, dm.historical, [zc.vault]
> - schema evolution: repoze.evolution, zope.app.generations
> - types: zc.blist, PersistentList, OOTreeSet, zc.queue
> - there are probably more duplicates...
> - many useful packages like zc.zodbgc which have no duplicate
>
> Each time I have to go and read through all the documentation (if there is
> any decent documentation) and then decide if I will use it or not.

Uh huh. Reading the documentation before you use something is a good
idea.

> Often
> the differences are minor.

And often the packages do entirely different things. Many of the
groupings you list above are of packages that solve different
problems.

> Sometimes I just stumble across the packages
> just by accident. Sometimes stuff is out-of-date. Rather often the
> packaging system breaks, because the .eggs are broken/outdated.

Hm. This happens to me sometimes, rarely, certainly not "rather often".

> Why do I
> have to accidentally discover that there's a zc.zodbgc module to pack
> cross reference databases if zodb core allows me to create cross
> references?

Because zc.zodbdgc is still experimental.  The "core" currently
contains all sorts of things that someone crapped there that either
I'm stuck with trying to support, I remove and make someone mad at me,
or just sits there like a bear trap waiting for someone to fall into
it. zc.zodbdgc is likely to make it into the core because it addresses
a core issue.  This isn't generally true of indexing.

> What indexing package do I choose? Is that package now for
> zope, plone or zodb?

Indexing is an application level function. The right choice largely
depends on your application.  ZODB provides core functionality, in the
form of BTrees.  (It could be argued that it shouldn't even provide
this much.)  Going beyone BTrees often involved application specific
trade-offs.

> In my opinion it would make sense to have a ZODB core package and then a
> second package which installs all the commonly used modules. Just like
> python is not very useful without the standard library. I know the modules
> in pythons standard library are good, tested and maintained. Many people
> use them, I can find docs and help.

IMO, the standard library is a bloated mess that lacks important core
features while containing many modules that it shouldn't.  The long
release cycle of the standard library means that packages there are
largely stagnant or forked.

> The zodb standard library should have a default package for
> indexing/searching.

It does, for better or worse. It's called BTrees.  This makes
innovation on BTrees difficult, for at least 2 reasons:

- If I change the data structure, I have to provide conversion
  scripts,

- The long release cycle discourages working on something I need now.

> One for basic data types like lists, sets and trees.

ZODB has all of these. The support for lists is a sad sad bug magnet.

> One for managing history, ... .

I'm sorry to point out that the list you gave as an example above
shows a lack of understanding of the problems the packages address or
a lack of depth in your analysis.

> The zodb standard library should then be endorsed by this list/maintainers
> so everybody knows where the focus is.

That is a recipe for disaster. We don't need more bureaucracy.  We
need a healthy ecosystem of packages that people can endorse with
their feet.

> And so I knows who's working on
> which things, so I can possibly collaborate with people.

Having a standard cesspool^h^h^h^h^h^h^h^hlibrary doesn't achive
that. Quite the opposite.

> The excessive use
> of packages (on the semantic level) splits a small developer community
> into even smaller pieces and makes collaboration hard to impossible.

I can't argue with a baseless claim.

> Imo
> this is also one of the reasons for zodb's weak documentation.

The reason for ZODB's weak documentation is that documentation hasn't
been a priority.  Good documentation is hard.  If anything, the
excessive size of ZODB makes documenting it harder.

> For a start the standard library could be a package which pulls all
> commonly useful, up-to-date 3rd party packages into one place.

I will never support this. Never. Don't try to convince me.

> Please don't take this as a bashing of any sorts,

I don't. I just take it as a bad idea. Please don't take *that* as
bashing. :)

> after all I am still
> using ZODB.

Cool.

> I just wanted to show you my thoughts after a few weeks in the
> zodb/plone universe and hope they can be used in a constructive way.

Thanks.

Jim

--
Jim Fulton