[Zope] searching and serving large textfiles ~120 Mb

Small Business Services toolkit at magma.ca
Fri Dec 5 08:31:56 EST 2003


Sebastian wrote:
> I need to access large textfiles (~120Mb) from zope. I know the python
lager
> file support and that it is better to keep large files out of the ZODB.
> I have to do a full text search on these files residing in a folder
hierachy
> on the server, show their content around the location of the found string
> and allow scrolling in that files source from zope.
>
> Has anybody done something similar to this with that lager files and would
> share his experiences?
> Are there any do's and don'ts or best ways to do it?

We have a similar application, but with many smaller text records instead of
one large one.  We currently use ZCatalog with ZCTextIndex to maintain a
database of about 700,000 text records; average record size is about 10k
bytes.  The total ZODB size is about 5Gb.

We also do a similar thing in that we locate the users search term within
the record and display the relevant sections of the search results records.
We found that we had to include the full-text of the record within the
metadata table (even though the recommended practice is to have a maximum of
200 bytes in the metadata table) because the time required to access the
original document was much too long.

There are some downsides to our approach though.  Retrieval speed is
excellent, however it currently takes us about 30 days of processing on a
dedicated server to rebuild the database.

We are currently running Zope 2.6.1 on linux servers and will be upgrading
to 2.6.2 as soon as our current update cycle completes.  We are also looking
for alternative ways to store the full-text.

HTH

Jonathan




More information about the Zope mailing list