[Zope-dev] 100k+ objects, or...Improving Performance of BTreeFolder...

[Tony McDonald]

On 10/12/01 2:54 pm, "Phillip J. Eby" <pje@telecommunity.com> wrote:

> I'm not sure if this is taken into consideration in your work so far/future
> plans...  but just in case you were unaware, it is not necessary for you to
> persistently store objects in the ZODB that you intend to index in a
> ZCatalog.  All that is required is that the object to be cataloged is
> accessible via a URL path.  ZSQL methods can be set up to be
> URL-traversable, and to wrap a class around the returned row.  To load the
> items into the catalog, you can use a PythonScript or similar to loop over
> a multi-row query, passing the objects directly to the catalog along with a
> path that matches the one they'll be retrievable from.  This approach would
> eliminate the need for BTreeFolder altogether, although of course it
> requires access to the RDBMS for retrievals.  This should reduce the number
> of writes and allow for bigger subtransactions in a given quantity of memory.

Gad! - are you saying you don't need to store a 1Mb .doc file into the ZODB,
but can still index the thing, store the index information in the Zcatalog
(presumably a lot smaller than 1Mb) and have the actual file accessible from
a file system URL? If so, that's really neat!

We're using Kaivo's document library and think it's great, but don't want to
store loads of files into the ZODB, preferring to keep them in a file system
(we want a lot of people to use the system and are worried about the ZODB
becoming gargantuan).

Tone
-- 
Dr Tony McDonald,  Assistant Director, FMCC, http://www.fmcc.org.uk/
The Medical School, Newcastle University Tel: +44 191 243 6140
A Zope list for UK HE/FE  http://www.fmcc.org.uk/mailman/listinfo/zope