Small Business Services wrote at 2003-12-5 08:31 -0500:
... We also do a similar thing in that we locate the users search term within the record and display the relevant sections of the search results records. We found that we had to include the full-text of the record within the metadata table (even though the recommended practice is to have a maximum of 200 bytes in the metadata table) because the time required to access the original document was much too long.
Something seems to be strange with your setup.
In general, it is no problem to load a few 10k objects from the ZODB (this is different when you load hundreds or thousands).
Putting the text in your MetaData causes huge MetaData blocks. With your avarage 10k content, you get 300k to 600k MetaData blocks (each block contains metadata records for up to 60 objects).
When we used getitem to access the fulltext field of the target document our average response time (using Call Profiler) was in the range of 5-7 seconds (we actually had to call getitem on 10 different documents for each search result set). When we moved the fulltext data to metadata the average response time dropped to 1-1.5 seconds. There were no other differences between the tests, so we concluded that doing a getitem call to the actual document was very expensive. Jonathan