[Zope] Building a fast, scalable yet small Zope application

[Morten W. Petersen]

Lennart Regebro skrev:
> On Sat, Apr 25, 2009 at 13:24, Morten W. Petersen <morten at nidelven-it.no> wrote:
>   
>> So far, I've been contemplating disabling undo (if that's possible),
>>     
>
> I doubt that it would make a difference. The Undo functionality comes
> out of the database being logging, and changing that would mean pretty
> much a complete rewrite.
>   

OK.  Well, I'm concerned about how much a database would grow.  I'm 
thinking if
I use one BTree for all the entries, would the database grow just a 
little or a lot when
you start getting into the millions of entries when inserting one small 
item?

>> and using BTree structures, maybe segmenting objects into different groups
>> (folders) to further speed up lookups.
>>     
>
> Yes, in my experience putting small objects in to BTree structures is
> quite fast. You may be talking about BTreeFolders, and in that case I
> don't know, I haven't done any sort of performance testing on those, I
> have used BTrees directly though, and that was fast. I haven't
> compared to SQL, but others have, and ZODB itself seems according to
> those tests quite fast. We know Plone slows everything down immensly
> in any case.
>
> I don't know if BTrees get slow when they get very big, so you would
> need to test that.
>   

Mm.  Yes, Plone is a bit sluggish, that's why I want to write a purely 
Zope-based app.

Yeah, I'll have to try different storage strategies in the ZODB, to see 
if a BTreeFolder
containing BTrees in the [0-9|A-Z|a-z] ranges would do, or if I need to 
partition it
up further with BTreeFolders containing BTreeFolders.

On the one hand I'm concerned about lookup speed, on the other about 
speed of
inserts and how much the entire database will grow inserting a < 1 KB 
object.

>>  Should I consider using the ZCatalog for faster lookups?
>>     
>
> Maybe. You probably need to not only store the objects in BTrees, but
> also somehow have indexes. These you do by storing the values you want
> to search on in BTrees as well. The ZCatalog does this in a
> configurable way for you, so if you need configurability, yes. If not,
> it's probably faster to make your own indexes with your own BTrees.
>   

Mm.  I guess I could be OK with one "index", it being the id/path of the 
object.  However,
it would be nice to build for the future and include the ability to 
search all objects.  Maybe
a combination of the two could work.

-Morten

-- 
Morten W. Petersen
Manager
Nidelven IT Ltd

Phone: +47 45 44 00 69
Email: morten at nidelven-it.no