[Zope] Fwd: Building a fast, scalable yet small Zope application

[Patrick Gerken]

Argl, why does this mailing list not use the reply-to: field?


---------- Forwarded message ----------
From: Patrick Gerken <do3ccqrv at googlemail.com>
Date: Sat, Apr 25, 2009 at 15:43
Subject: Re: [Zope] Building a fast, scalable yet small Zope application
To: "Morten W. Petersen" <morten at nidelven-it.no>


On Sat, Apr 25, 2009 at 13:24, Morten W. Petersen <morten at nidelven-it.no> wrote:
> Hi,
>
> I'm considering building a large scale, but small in features site.  It
> will contain
> lots of small objects (millions, tens of millions, hundreds of millions)
> of objects,
> where each object has a couple of strings and maybe some other light
> attributes.
>
> So far, I've been contemplating disabling undo (if that's possible), and
> using
> BTree structures, maybe segmenting objects into different groups
> (folders) to
> further speed up lookups.  Scalability is also an issue, should I
> consider using
> RelStorage?  Should I consider using the ZCatalog for faster lookups?
>
> Has anyone else developed something similar?  Are there Zope product
> examples out there that fit the bill?

Hi, I once was involved in developing a system that supposedly should
be able to handle a very big number of objects. They chose to put all
the base objects into mysql too, and use mysql for object lookup. I do
not know what type of benchmarks they did to conclude that this was
the best choice. But they regulary did queries for which relational
databases were made.
If your data contains lots of small items, that are all similar of
content, and you want to do queries that involves many of these
attributes, you might want to take a look at orm tools.
relstorage will not help you with your queries, because it stores
everything as a pickle, afaik.

I suggest you experiment a bit. Create 100 million objects, and do
some of the actions you are planning to do on them.


Best regards,

           Patrick