September 1999 - Zope-Dev

International Catalog
by Michel Pelletier 14 Sep '99

14 Sep '99

I cc'ed some other people on this, because there is some important locale information in this message. > -----Original Message----- > From: Martijn Faassen [mailto:m.faassen@vet.uu.nl] > Sent: Tuesday, September 14, 1999 7:33 AM > Cc: zope-dev(a)zope.org > Subject: Re: [Zope-dev] Re: [Zope] Need a list of words not indexed by > Catalog > > Rik Hoekstra wrote: > > > > > Terrel Shumway wrote: > > > > > > > > near the end of > > > > lib/python/SearchIndex/TextIndex.py > > > > is a list called 'stop_words' > > > > > > > > [Zope Dev] It would be good to move this out of the .py > file into an > > > > editable, internationalizable resource file. > > > > > > Agreed! And then there's the *multi* lingual issue too. > What if I have > > > Dutch and English on my site? > [snip] > > It seems like you run into a _lot_ of complexities with > multilingual issues, > > and still these are real issues for many of us. > > Yes, very real issues. Suddenly ZCatalog isn't the > almost-ready tool to > add searchability to the website I'm building anymore.. Now I > need to do > quite a bit of extra work, I imagine.. I am thinking heavily about this very problem as we speak. You all correctly pointed out some of the toughest of the problems. Here are my ideas so far: Have 'vocabulary objects' store the stopwords, synonyms, stemming rules, and lexicon (collection of uniquely indexed words) in a drop-in object for ZCatalog. This way, a 'French', 'Dutch' etc. vocabulary object could be developed by a third party. TextIndexes can then reference (or acquire) a vocabular object through which it can stop, syn, stem and store words in it's lexicon. There are many other issues like sharing lexicons between similar language indexes, and having multiple back-end 'index/vocabularies' that all look like one index, so you can search a 'document source' for either 'community' or 'communauté' or 'Gemeinschaft' and get only documents relevant to that language (my applogies if these words are wrong, I'm using babelfish). I think this problem could be intractable though, if you searched for 'walking' in english, the word would stem down into 'walk', if you search for 'marche' en francais, should it stem down to 'promenade'? Anyways, there is some good news. For those of you tracking CVS we have added the ability to set your locale in Zope. This means that, forexample, the splitter/stemmer in the catalog will recognize all of those umlauts and accented letters and whatnots that english doesn't have. We would like a few people all over the place to try this out. If your locale has a different language or monetary system than the US (just about everywhere except some of canada) this might make the catalog and other parts of Zope more useful for you. local can be activated from the z2.py command like with the '-L' option. "-L ''" (an empty string) will cause local to try and autodetect your locale from your environment variables (you must set the env variables yourself, see 'man 7 locale'). Alternativly, you can say "-L de" and set your local to German. Please folks, test this out for us. We don't really have the means to do it here. -Michel

1 0

Re: [Zope-dev] Caching
by Josh Zeidner 14 Sep '99

14 Sep '99

Hi Damien, I solved this problem for a web-site( S&S ) based in Cold-Fusion by simulating browser requests and copying the contents retrieved into a cache directory structure. The caching took place on a time-bases rather than some function of hits or timestamp. I do not know, however, if this caching mechanism could be developed( to work efficiently ) as part of Zope as any object in Zope is dynamically created. I good idea may be to write a simple web-spider that tracks the contents of the site and automatically caches all the pages on a regular basis... -josh z >From: morton(a)dennisinter.com (Damian Morton) >To: "Christopher Petrilli" <petrilli(a)digicool.com>, "Paul Mokbel" ><pm(a)webhosting.com>, <Zope-Dev(a)zope.org> >Subject: Re: [Zope-dev] Caching >Date: Sun, 12 Sep 1999 22:13:36 -0400 > >This is an interesting problem; a significant proportion of a dynamic >website is not driven by the user requesting a page, but is rather driven >by certain triggering conditions. > >In an ideal world, you be able to write your website as a dynamic website, >and some kind of magic would determine what the conditions triggring the >rendering of a page were and then set up some kind of notification >mechanism to cause the re-rendering of the page as the conditions are >triggered. > >For example, a news page only needs to be re-rendered when the news on the >page changes. > >Being able to identify which parts of a page need to be re-rendered is even >more fine-grained. > >To do this, youd have to timestamp virtually every single object, from >variables on up, with a last-modified date. This could be tricky. > >----- Original Message ----- >From: Christopher Petrilli <petrilli(a)digicool.com> >To: Paul Mokbel <pm(a)webhosting.com>; <Zope-Dev(a)zope.org> >Sent: Sunday, September 12, 1999 9:54 PM >Subject: Re: [Zope-dev] Caching > > > > > In ROXEN (RXML); There is a tag line where by you can cache certain >content > > > within the <CACHE> brackets. Is there something similar within Zope? > > > > What has been thought of is doing more "rendered caching" (i.e. cache > > rendered results, thereby short-circuiting the DTML process). This >could be > > very advantageous, but gets into a lot of issues of management of the >cache > > which is non-trivial (Vignette, one of our competitors that does > > "consultant-ware" has several patents on it), so... patches accepted. >Note > > that performance will be one of our "watch words" in the coming months. > > > > Chris > > -- > > | Christopher Petrilli Python Powered Digital Creations, >Inc. > > | petrilli(a)digicool.com >http://www.digicool.com > > > > _______________________________________________ > > Zope-Dev maillist - Zope-Dev(a)zope.org > > http://www.zope.org/mailman/listinfo/zope-dev > > > > (To receive general Zope announcements, see: > > http://www.zope.org/mailman/listinfo/zope-announce > > > > For non-developer, user-level issues, > > zope(a)zope.org, http://www.zope.org/mailman/listinfo/zope ) > > >_______________________________________________ >Zope-Dev maillist - Zope-Dev(a)zope.org >http://www.zope.org/mailman/listinfo/zope-dev > >(To receive general Zope announcements, see: >http://www.zope.org/mailman/listinfo/zope-announce > >For non-developer, user-level issues, >zope(a)zope.org, http://www.zope.org/mailman/listinfo/zope ) ______________________________________________________ Get Your Private, Free Email at http://www.hotmail.com

2 1

Principia-ness in Zope 2.0.0 OFS/Folder.py
by Andreas Kostyrka 14 Sep '99

14 Sep '99

class Folder(ObjectManager, PropertyManager, RoleManager, Collection, SimpleItem.Item, FindSupport): """ The basic container object in Principia. Folders can hold almost all other Principia objects. """ Andreas -- Andreas Kostyrka | andreas(a)mtg.co.at phone: +43/1/7070750 | phone: +43/676/4091256 MTG Handelsges.m.b.H. | fax: +43/1/7065299 Raiffeisenstr. 16/9 | 2320 Zwoelfaxing AUSTRIA

1 0

Re: [Zope-dev] Re: [Zope] Need a list of words not indexed by Catalog
by Martijn Faassen 14 Sep '99

14 Sep '99

Rik Hoekstra wrote: > > > -----Original Message----- > > From: Martijn.Faassen(a)vet.uu.nl [mailto:Martijn.Faassen@vet.uu.nl]On > > Behalf Of Martijn Faassen > > Sent: Tuesday, September 14, 1999 11:35 AM > > Cc: 'zope-dev(a)zope.org' > > Subject: [Zope-dev] Re: [Zope] Need a list of words not indexed by > > Catalog > > > > > > Terrel Shumway wrote: > > > > > > near the end of > > > lib/python/SearchIndex/TextIndex.py > > > is a list called 'stop_words' > > > > > > [Zope Dev] It would be good to move this out of the .py file into an > > > editable, internationalizable resource file. > > > > Agreed! And then there's the *multi* lingual issue too. What if I have > > Dutch and English on my site? [snip] > It seems like you run into a _lot_ of complexities with multilingual issues, > and still these are real issues for many of us. Yes, very real issues. Suddenly ZCatalog isn't the almost-ready tool to add searchability to the website I'm building anymore.. Now I need to do quite a bit of extra work, I imagine.. Regards, Martijn

1 0

Examples of non-unique XML node ids?
by Martijn Faassen 14 Sep '99

14 Sep '99

Hi there, About XMLDocuments: Does anyone has any example scenarios of when a non unique XML node id occurs? It'd be nice if we could rely on the uniqueness of an XML node id, but apparently sometimes they're not. Stupidly enough I can't come up with a scenario that will *cause* a node id to be duplicated, and I wonder if anyone else has a clue. If I can see the scenarios I'll try to hack the XMLDocument source so that each node has an unique id (in a single XML Document) at all time. Regards, Martijn

1 0

RE: [Zope] Need a list of words not indexed by Catalog
by Terrel Shumway 14 Sep '99

14 Sep '99

near the end of lib/python/SearchIndex/TextIndex.py is a list called 'stop_words' [Zope Dev] It would be good to move this out of the .py file into an editable, internationalizable resource file. -----Original Message----- From: Jason Spisak [mailto:webmaster@hiretechs.com] Sent: Monday, September 13, 1999 9:31 AM To: zope(a)zope.org Subject: [Zope] Need a list of words not indexed by Catalog Can we get a public list of the words not indexed by the Catalog? I just spent a few hours being tortured trying to find out why a guy in the Datbase of 10,000 whose name is 'Max Many', doesn't come up on a last name search, and guess what, the Catalog doesn't index his last name. The word 'many'. Suprize! This is the third gotcha word we've expericened (don't try to find C/C++ in a document) and it's makes people doubt the software. (I explain about the size of indexes and such, and I understand it, but every surprise word you can't find is another step backward.) You might publish a list of words you'll never find in the Catalog. Just in case one of them happens to be a very big part of someone's business. All my best, -- Jason Spisak webmaster(a)hiretechs.com _______________________________________________ Zope maillist - Zope(a)zope.org http://www.zope.org/mailman/listinfo/zope (To receive general Zope announcements, see: http://www.zope.org/mailman/listinfo/zope-announce For developer-specific issues, zope-dev(a)zope.org - http://www.zope.org/mailman/listinfo/zope-dev )

3 2

RE: [Zope-dev] Multiple inheritance woes
by Rik Hoekstra 13 Sep '99

13 Sep '99

> -----Original Message----- > From: zope-dev-admin(a)zope.org [mailto:zope-dev-admin@zope.org]On Behalf > Of Martijn Pieters > Sent: Friday, September 10, 1999 10:24 AM > To: Phillip J. Eby; Zope-dev(a)zope.org > Subject: Re: [Zope-dev] Multiple inheritance woes > > > At 20:05 09/09/99 , Phillip J. Eby wrote: > >At 02:54 PM 9/9/99 +0200, Martijn Pieters wrote: > > >At 14:33 09/09/99 , Rik Hoekstra wrote: > > >> > > > >> > I define a baseclass derived from CatalogAware and DTMLMethod: > > >> > > > >> > class Base(CatalogAware, DTMLMethod): > > >> > def index_object(self): > > >> > # check a few things I want to know about > > >> > # before indexing, maybe even not allowing > > >> > # the indexing > > >> > CatalogAware.index_object(self) > > >> > > > >> > > > >> > > >>A shot in the dark: Shouldn't this be > > >>CatalogAware().index_object(self)? At least this explicitly > > >>instanciates CatalogAware and stops raising TypeErrors. But I'm not > > >>sure this completely solves your problem > > >> > > >>Rik > > > > > >Sorry, you missed =). In normal python this would work, but as Philip > > >pointed out, when using Acquisition and ExtensionClasses, things are > > >different, and throw the Python interpreter off. > > > > > > >Actually, no, it wouldn't, because then you'd get a different TypeError, > >this one for having passed too many arguments to the > index_object() method. > > And besides, you don't want to call another object's index_object() > >method, you want to call an overridden version of *your* index_object() > >method. > > Heh, why does noone ever understand what I am saying =). I ment that _my_ > code would normally work in Python. I wasn't commenting on Rik's. > There is > a reasonable change I might meet him one day. After all, he's > only about an > hours drive away. =) > You bet :-\ (But I've learnt my lesson - I'll never post so quick again) Rik

5 4

ZClass superclassing DTML Document
by Jason Spisak 13 Sep '99

13 Sep '99

Zopifites, I have posted this question before to no avail and submitted it to the collector. I probably didn't phrase it succinctly enough so I'll try again.:^) I have a ZClass that uss DTML Document as a Base Class. When I create an instance of it, the 'id' property shows up in the management interface, but doesn't show up with a <dtml-var id> inside itself, or in the tree tag. This same code worked in beta3 and 4 then it shanked in beta 5 and is still shanked. Any ideas? -- Jason Spisak webmaster(a)hiretechs.com

1 0

RE: [Zope-dev] ZClass superclassing DTML Document
by Michel Pelletier 13 Sep '99

13 Sep '99

> -----Original Message----- > From: Jason Spisak [mailto:webmaster@hiretechs.com] > Sent: Monday, September 13, 1999 10:02 AM > To: zope-dev(a)zope.org > Subject: [Zope-dev] ZClass superclassing DTML Document > > > Zopifites, > > I have posted this question before to no avail and submitted it to the > collector. I probably didn't phrase it succinctly enough so I'll try > again.:^) > > I have a ZClass that uss DTML Document as a Base Class. > When I create an instance of it, the 'id' property shows up in the > management interface, but doesn't show up with a <dtml-var id> inside > itself, or in the tree tag. This same code worked in beta3 and 4 then > it shanked in beta 5 and is still shanked. Any ideas? Try <dtml-var document_id> -Michel

2 1

RE: [Zope] Need a list of words not indexed by Catalog
by Michel Pelletier 13 Sep '99

13 Sep '99

> -----Original Message----- > From: Jason Spisak [mailto:webmaster@hiretechs.com] > Sent: Monday, September 13, 1999 12:31 PM > To: zope(a)zope.org > Subject: [Zope] Need a list of words not indexed by Catalog > > > Can we get a public list of the words not indexed by the Catalog? You asked for it: stop_words=( 'am', 'ii', 'iii', 'per', 'po', 're', 'a', 'about', 'above', 'across', 'after', 'afterwards', 'again', 'against', 'all', 'almost', 'alone', 'along', 'already', 'also', 'although', 'always', 'am', 'among', 'amongst', 'amoungst', 'amount', 'an', 'and', 'another', 'any', 'anyhow', 'anyone', 'anything', 'anyway', 'anywhere', 'are', 'around', 'as', 'at', 'back', 'be', 'became', 'because', 'become', 'becomes', 'becoming', 'been', 'before', 'beforehand', 'behind', 'being', 'below', 'beside', 'besides', 'between', 'beyond', 'bill', 'both', 'bottom', 'but', 'by', 'can', 'cannot', 'cant', 'con', 'could', 'couldnt', 'cry', 'describe', 'detail', 'do', 'done', 'down', 'due', 'during', 'each', 'eg', 'eight', 'either', 'eleven', 'else', 'elsewhere', 'empty', 'enough', 'even', 'ever', 'every', 'everyone', 'everything', 'everywhere', 'except', 'few', 'fifteen', 'fifty', 'fill', 'find', 'fire', 'first', 'five', 'for', 'former', 'formerly', 'forty', 'found', 'four', 'from', 'front', 'full', 'further', 'get', 'give', 'go', 'had', 'has', 'hasnt', 'have', 'he', 'hence', 'her', 'here', 'hereafter', 'hereby', 'herein', 'hereupon', 'hers', 'herself', 'him', 'himself', 'his', 'how', 'however', 'hundred', 'i', 'ie', 'if', 'in', 'inc', 'indeed', 'interest', 'into', 'is', 'it', 'its', 'itself', 'keep', 'last', 'latter', 'latterly', 'least', 'less', 'made', 'many', 'may', 'me', 'meanwhile', 'might', 'mill', 'mine', 'more', 'moreover', 'most', 'mostly', 'move', 'much', 'must', 'my', 'myself', 'name', 'namely', 'neither', 'never', 'nevertheless', 'next', 'nine', 'no', 'nobody', 'none', 'noone', 'nor', 'not', 'nothing', 'now', 'nowhere', 'of', 'off', 'often', 'on', 'once', 'one', 'only', 'onto', 'or', 'other', 'others', 'otherwise', 'our', 'ours', 'ourselves', 'out', 'over', 'own', 'per', 'perhaps', 'please', 'pre', 'put', 'rather', 're', 'same', 'see', 'seem', 'seemed', 'seeming', 'seems', 'serious', 'several', 'she', 'should', 'show', 'side', 'since', 'sincere', 'six', 'sixty', 'so', 'some', 'somehow', 'someone', 'something', 'sometime', 'sometimes', 'somewhere', 'still', 'such', 'take', 'ten', 'than', 'that', 'the', 'their', 'them', 'themselves', 'then', 'thence', 'there', 'thereafter', 'thereby', 'therefore', 'therein', 'thereupon', 'these', 'they', 'thick', 'thin', 'third', 'this', 'those', 'though', 'three', 'through', 'throughout', 'thru', 'thus', 'to', 'together', 'too', 'toward', 'towards', 'twelve', 'twenty', 'two', 'un', 'under', 'until', 'up', 'upon', 'us', 'very', 'via', 'was', 'we', 'well', 'were', 'what', 'whatever', 'when', 'whence', 'whenever', 'where', 'whereafter', 'whereas', 'whereby', 'wherein', 'whereupon', 'wherever', 'whether', 'which', 'while', 'whither', 'who', 'whoever', 'whole', 'whom', 'whose', 'why', 'will', 'with', 'within', 'without', 'would', 'yet', 'you', 'your', 'yours', 'yourself', 'yourselves', ) This is defined in lib/python/SearchIndex/UnTextIndex.py and TextIndex.py. ZCatalog uses UnTextIndexes. I believe confera uses TextIndexes. This list is obviously not going to work for lots of applications. What if you want to search for words like 'six' and 'five'? What if your indexing python code and want to find all ocourances of 'while' and 'for'? What if your indexing French or German? The reason why we have these as stop words is because, for most (English) 'documents' (meaning large bodies of text), these are the most common words encounterd, and have a higher chance of being in a document than not being in a document. The technical reasons get into the very precise and voodooish concepts of relevance, precision, and result set size. This is very deep stuff. Suffice to say, without stop words indexes grow huge, so clip this list at your own risk, and be aware that taking out 'Many' may improve your ability to index your employees, it may also drasticly reduce the relevance and precision while increasing the size of your result set for a different 'kind' of text corpus. > I just spent a few hours being tortured trying to find out > why a guy in > the Datbase of 10,000 whose name is 'Max Many', doesn't come up on a > last name search, and guess what, the Catalog doesn't index his last > name. The word 'many'. Suprize! This is the third gotcha word we've > expericened (don't try to find C/C++ in a document) and it's makes > people doubt the software. (I explain about the size of indexes and > such, and I understand it, but every surprise word you can't find is > another step backward.) The current TextIndexing machinery was intended to be used from a document-centric standpoint. The need to text index small values was not considered way back when the first TextIndex was written. Given the amount of feedback we have recieved on this, it is obviously on our radar. -Michel

2 1