On Wednesday 14 August 2002 06:03 pm, Guido van Rossum wrote:
Fix for issue #505 ZCTextIndex is now associated by path to its lexicon. After replacing a lexicon used by an index, clear the index to make it use the new lexicon.
So the semantics are that when you replace the lexicon, the index is reset to empty, right? Why not create a new index instead? Then the lexicon could be internal to the index. Sharing lexicons doesn't sound like a probable use case, the more I think about it.
--Guido van Rossum (home page: http://www.python.org/~guido/)
I don't disagree. This was a conceptual holdover from the previous generation TextIndex. I'm switching this over to zope-dev for wider discussion: The current implementation of ZCTextIndex is like the old TextIndex in that you can create one Lexicon (the sucessor to Vocabularies) shared by multiple ZCTextIndexes. I imagine the thought was that there are only a finite number of words and that sharing the lexicon would save space and possibly index time, since a given word would only need to be inserted once into the lexicon regardless of the number of indexes it occurred in. More significant might be the (cache) memory savings of only having to keep one copy of the words in memory across several indexes. Plus fewer loads and stores to the database overall by sharing the word list. On the other hand I think query speeds may be compromised since one large lexicon would take longer to search for a given word (or words) then several smaller ones. This would be especially true for small indexes sharing a lexicon with a much larger one. The other downside (as illustrated by issue #505) is the complication of linking index to lexicon and making the link manageable so that you can tweak the indexing system easily. My fix is not entirely complete because a hard ref to the lexicon is still stored in the low-level index (to which the ZCTextIndex class delegates). In order to fix this effectively without introducing Zope dependancies at the low level (which we have looked to avoid) I would need to create some sort of Lexicon proxy that can access the correct lexicon on demand by a path efficiently. This proxy would be referenced by the low level index in place of the actual lexicon. Of course the other solution, which is much simpler is to dispense with this notion of sharing lexicons entirely and as Guido suggests, just make the lexicon part of the index. Without hard use cases to the contrary, I lean toward that simpler design. However I would like to perform some additional testing on large corpuses with many indexes to assess the memory/performance differences between these two approaches. We have already ascertained that with the new ZODB cache code in 2.6, the cache setting can have a profound affect on query performance (like a factor of 10), so I think testing would be helpful. Anyone care to weigh in with use cases for shared lexicons? -Casey