Sebastian wrote:
I need to access large textfiles (~120Mb) from zope. I know the python lager file support and that it is better to keep large files out of the ZODB. I have to do a full text search on these files residing in a folder hierachy on the server, show their content around the location of the found string and allow scrolling in that files source from zope.
Has anybody done something similar to this with that lager files and would share his experiences? Are there any do's and don'ts or best ways to do it?
We have a similar application, but with many smaller text records instead of one large one. We currently use ZCatalog with ZCTextIndex to maintain a database of about 700,000 text records; average record size is about 10k bytes. The total ZODB size is about 5Gb. We also do a similar thing in that we locate the users search term within the record and display the relevant sections of the search results records. We found that we had to include the full-text of the record within the metadata table (even though the recommended practice is to have a maximum of 200 bytes in the metadata table) because the time required to access the original document was much too long. There are some downsides to our approach though. Retrieval speed is excellent, however it currently takes us about 30 days of processing on a dedicated server to rebuild the database. We are currently running Zope 2.6.1 on linux servers and will be upgrading to 2.6.2 as soon as our current update cycle completes. We are also looking for alternative ways to store the full-text. HTH Jonathan