[Zope] searching and serving large textfiles ~120 Mb

[Small Business Services]

> Small Business Services wrote at 2003-12-5 08:31 -0500:
> > ...
> >We also do a similar thing in that we locate the users search term within
> >the record and display the relevant sections of the search results
records.
> >We found that we had to include the full-text of the record within the
> >metadata table (even though the recommended practice is to have a maximum
of
> >200 bytes in the metadata table) because the time required to access the
> >original document was much too long.
>
> Something seems to be strange with your setup.
>
>   In general, it is no problem to load a few 10k objects from
>   the ZODB (this is different when you load hundreds or thousands).
>
>   Putting the text in your MetaData causes huge MetaData blocks.
>   With your avarage 10k content, you get 300k to 600k
>   MetaData blocks (each block contains metadata records
>   for up to 60 objects).

When we used getitem to access the fulltext field of the target document our
average response time (using Call Profiler) was in the range of 5-7 seconds
(we actually had to call getitem on 10 different documents for each search
result set).  When we moved the fulltext data to  metadata the average
response time dropped to 1-1.5 seconds.

There were no other differences between the tests, so we concluded that
doing a getitem call to the actual document was very expensive.


Jonathan