August 2000 - Zope-Dev

"stemmed and stopped": problems with stopwords and the 'and' operator
by R. David Murray 17 Aug '00

17 Aug '00

OK, so the TextIndex of a ZCatalog says that it "stems and stops" the words before indexing them and, one would hope, before searching for them. I always thought that "stem" meant "derive the stem of the word" (so as to make the index smaller). I just peeked at the Splitter.c source code for the first time, and that sure ain't it. The American phrase would be "truncate and stop", I think. In any case, "stem" in the source code comments means truncate at MAX_WORD, which is 64 characters. That's an aside. Now, about stopping. There is a list of "stop words" that don't get indexed. Fine. I'm having quite a bit of trouble figuring out exactly where this is happening, but let's ignore that for now on the indexing side. It happens, that's enough for now. Now, what happens to stop words in an input search string? >From my single stepping the code, stopwords are still in the query string while it is being parsed, and get looked up in the index. So, here is the heart of my problem: consider the search string someword and someotherword Suppose 'someword' is a stopword. It doesn't get indexed because it is considered too common. Now, I would think that if this search string is submitted, the result would be to return the hits for 'someotherword'. This might, however, not be other people's opinions. So, is the fact that TextIndex appears to return the null set in this case a bug or a feature? I say 'appears' because I actually get 2 (out of about 2000 with the keyword 'car') hits in my database when I search on 'car and the'. I tried to single step through the logic using the debugger, but when the call is made made to the splitter with the stopword passed in, python core dumps. I can do 'from SearchIndex.Splitter import Splitter', and call Splitter, and see that stopwords are not removed, but I can't do 'from SearchIndex.UnTextIndex import Splitter' because it complains about not being able to import Persistent from Persistence. (*That* problem was reported by someone else in another context not too long ago.) However, it's pretty clear that this null set return is what is happening, since when the evaluate subroutine is entered, the stop word is in the partially parsed string, and is in fact passed to the Splitter in the __getitem__ of the text index. If the splitter stopped it, the returned result set would be None, If the splitter doesn't stop it, the text index is still going return a null set as the result for that word, since it doesn't appear in the index by definition. An 'and' of any result set with None is going to be the null set. So it looks like the thing was designed this way: the stop words get "deleted" from the search string by not being in the index and by therefore returning null sets when looked up. This works fine for 'or' logic, but not for 'and' logic, IMO. Contrary opinions? Helpful hints? If I'm right and this needs fixed, it's going to be a bit of a bear to do, I think. (Where those two hits are coming from is a *real* mystery, but one I'm going to ignore for a little while yet since I can't yet get the debugger to work for me without crashing. I have a sneaking suspicion it is related to my confusion about where stopwords get removed in the indexing process, but it will probably take a while for me to prove or disprove that notion.) --RDM

2 2

Data.fs
by Andre Schubert 17 Aug '00

17 Aug '00

Hi, I'am using Zope 2.1.6 and my Data.fs is going corrupted and i don't have found a way for backup this file. My Questions are: is there a way to delete the last transaction from Data.fs, and can i backup the Data.fs when Zope is running. Is the backup of Data.fs inconsitent when i make the backup at the same time a user writes to it? I want to a backup Data.fs twice a day. so i want to have 14 data.fs files every week. On this way i can find out when the error occurs and i can restore an older version of Data.fs. as

2 1

patch for &dtml-foo:bar:baz;
by Steve Alexander 17 Aug '00

17 Aug '00

This patch changes lib/python/DocumentTemplate/DT_HTML.py so that you can use paths for traversal in DTML entity syntax. The delimiter is ':', as it is valid as part of an XML entity, but not valid as part of a Zope object id. You can use &dtml-:foo:bar; to ensure that traversal occurs from the root object. This patch is against Zope 2.2.1b1. It relies on the restrictedTraverse() api. A 2.1.x version would be possible, but would be rather messy. -- Steve Alexander Software Engineer Cat-Box limited http://www.cat-box.net *** DT_HTML.py.old --- DT_HTML.py *************** *** 98,104 **** end_match=regex.compile('[\0- ]*\(/\|end\)', regex.casefold).match, start_search=regex.compile('[<&]').search, ! ent_name=regex.compile('[-a-zA-Z0-9_.]+').match, find=find, strip=strip, replace=replace, --- 98,104 ---- end_match=regex.compile('[\0- ]*\(/\|end\)', regex.casefold).match, start_search=regex.compile('[<&]').search, ! ent_name=regex.compile('[-a-zA-Z0-9_.:]+').match, find=find, strip=strip, replace=replace, *************** *** 156,162 **** d[1]=d['end']='' d[2]=d['name']='var' d[0]=text[s:e+1] ! d[3]=d['args']=args+' html_quote' return s else: nn=find(args,'-') --- 156,165 ---- d[1]=d['end']='' d[2]=d['name']='var' d[0]=text[s:e+1] ! d[3]=d['args']=':' in args \ ! and '"_.render(restrictedTraverse(\''+ \ ! replace(args,':','/')+'\'))" html_quote' \ ! or args+' html_quote' return s else: nn=find(args,'-') *************** *** 164,171 **** d[1]=d['end']='' d[2]=d['name']='var' d[0]=text[s:e+1] ! args=(args[nn+1:]+' '+ ! replace(args[:nn],'.',' ')) d[3]=d['args']=args return s --- 167,178 ---- d[1]=d['end']='' d[2]=d['name']='var' d[0]=text[s:e+1] ! path=args[nn+1:] ! args=(':' in path ! and '"_.render(restrictedTraverse(\''+ ! replace(path,':','/')+'\'))" ' ! or path+' ') \ ! +replace(args[:nn],'.',' ') d[3]=d['args']=args return s

1 0

Extending dtml entity syntax
by Steve Alexander 17 Aug '00

17 Aug '00

Chris Withers has suggested extending the DTML Entity syntax to include traversal information. For example: &dtml-foo/bar/baz; I just checked the XML standard. This isn't allowed in XML entities. http://www.w3.org/TR/1998/REC-xml-19980210#NT-Nmtoken EntityRef ::= '&' Name ';' NameChar ::= Letter | Digit | '.' | '-' | '_' | ':' | CombiningChar | Extender Name ::= (Letter | '_' | ':') (NameChar)* However, we could consider &dtml-foo.bar.baz; instead. I can't think of a nice alternative to mean &dtml-/foo/bar/baz; "from the root, traverse as follows". -- Steve Alexander Software Engineer Cat-Box limited http://www.cat-box.net

2 1

ZPatterns, Transactions, _register/_unregister
by Bob Pepin 17 Aug '00

17 Aug '00

Hi, I've encountered some weird behaviour in the ZPatterns Transactional class when I was trying to write a User Source for the Login Manager Product. I'm using Zope 2.2.0 with LoginManager 0.8.7a1 and ZPatterns 0.4.1snap1 The problem is that _unregister seems to be trying to delete the _v_registered attribute of an object that doesn't have it set whenever I return None from authenticateUser(). The weird thing is that the object seems to be _register()ed and _v_registered appears to be set to 1 in _register(), but when _unregister() is called it has the value 'None' again. I've attached my code that's trying to implement the UserSource. here is what my modified _register and _unregister functions look like: def _register(self): f=open('/tmp/zopelog', 'a', 0) f.write('Register %s(%s)\n' % (self.id, str(self))) f.write('Before: %s._v_registered=%s\n' % (self.id, getattr(self, '_v_registered', 'N/A'))) for i in traceback.format_stack(): f.write(i) if self._v_registered: return get_transaction().register(Reporter(self)) self._v_registered = 1 f.write('After: %s._v_registered=%s\n' % (self.id, getattr(self, '_v_registered', 'N/A'))) f.close() def _unregister(self): f=open('/tmp/zopelog', 'a', 0) f.write('Unregister %s(%s)\nBefore: %s._v_registered=%s\n' \ % (self.id, str(self), self.id, getattr(self, '_v_registered', 'N/A'))) f.close() del self._v_registered and this is a sample log: Register UserSource(<NisUserSource instance at 85ab2e8>) Before: UserSource._v_registered=None File "/home/picard/bpe/Zope-2.2.0-src/ZServer/PubCore/ZServerPublisher.py", line 95, in __init__ response=response) File "/home/picard/bpe/Zope-2.2.0-src/lib/python/ZPublisher/Publish.py", line 222, in publish_module response = publish(request, module_name, after_list, debug=debug) File "/home/picard/bpe/Zope-2.2.0-src/lib/python/ZPublisher/Publish.py", line 162, in publish object=request.traverse(path, validated_hook=validated_hook) File "/home/picard/bpe/Zope-2.2.0-src/lib/python/ZPublisher/BaseRequest.py", line 427, in traverse else: user=v(request, auth, roles) File "/home/picard/bpe/Zope-2.2.0-src/lib/python/Products/LoginManager/LoginManager.py", line 110, in validate user = _DefaultAuth.findLogin(self, request, auth, user, roles) File "/home/picard/bpe/Zope-2.2.0-src/lib/python/Products/LoginManager/LoginMethods.py", line 147, in findLogin user = manager.getItem(name) File "/home/picard/bpe/Zope-2.2.0-src/lib/python/Products/LoginManager/LoginManager.py", line 65, in getItem user = source.__of__(self).getItem(name) File "/home/picard/bpe/Zope-2.2.0-src/lib/python/Products/ZPatterns/Rack.py", line 61, in getItem self._registerCanonical(k,item) # XXX Should we cache non-existence? File "/home/picard/bpe/Zope-2.2.0-src/lib/python/Products/ZPatterns/DataManagers.py", line 52, in _registerCanonical self._register() File "/home/picard/bpe/Zope-2.2.0-src/lib/python/Products/ZPatterns/Transactions.py", line 47, in _register for i in traceback.format_stack(): After: UserSource._v_registered=1 Unregister UserSource(<NisUserSource instance at 85ab2e8>) Before: UserSource._v_registered=None

2 1

Re: [Zope-dev] ZClass properties and DTML
by Dieter Maurer 17 Aug '00

17 Aug '00

Carl Robitaille writes: > .... Is there a way to > include DTML tags in properties? If not, what are my other options? Not that I know of. Make your ZClass folderich (derive from Folder) and use DTML methods instead of properties. Dieter

2 1

Funny bug in Zope?
by Erik Enge 16 Aug '00

16 Aug '00

I'm not quite sure if this is already reported or not, so I won't give much information on it (I've looked for something similar, but I haven't found it). I'm runnig Zope 2.2.0 under Linux. I have this product I've made myself - called TravelAgent. I add an instance of it to the Zope root-folder, which I call TravelAgent. Everything works fine. If I rename this instance to something that has less than five characters in it, something very fish starts to happen. I should also mention that in the index_html method's body-tag I have an background-element which displays somepic.gif. Now, if the instance is called f ex Trav (ie. less that five characters) and I try to render the index_html, it displays the background image correctly, but it also pukes out the gif in ASCII on top - and nothing else. This happens very consistently, unless I reload fast four, five times - then it display the whole index_html just fine. I rename back to Trave (ie. five characters or more) and everything is fine. I can reproduce this all the time. Strange, huh?

1 0

Python class and ZClass
by Vincent 16 Aug '00

16 Aug '00

Hi, I would like to subclass a Python class with a ZClass. I mean that my DTML document will declare ZClass and I would like to be able to access the method declared in the python class (parent of the ZClass). Does anybody knows how to do that ? PS : I read a paper call "SubClassing from Custom Python classes", but I did not understood everything, so if somebody can give me an example or the main steps to solve my problem... Thank you very very much Vincent

1 0

zpatterns-0.4a4
by Jephte CLAIN 16 Aug '00

16 Aug '00

There is a bug (a feature?) in Specialist.getItem in zpatterns-0.4a4: def getItem(self, key): if hasattr(self.aq_base,'retrieveItem'): return self.retrieveItem(key=key) # XXX need DTML check? for rack in self.rackList: item = rack.__of__(self).getItem(key) return item This code should (IMHO) read: def getItem(self, key): if hasattr(self.aq_base,'retrieveItem'): return self.retrieveItem(key=key) # XXX need DTML check? for rack in self.rackList: item = rack.__of__(self).getItem(key) if item is not None: return item return None regards, jephte clain jclain(a)univ-reunion.fr

5 8

remote procedure calls to manage functions
by andrew 15 Aug '00

15 Aug '00

Have been delving into ZPublisher.Client to make remote procedure calls under Zope 2.2.1. Many of the manage_... functions work just fine through this interface (once the obvious bug on line 212 of Client.py is fixed, anyway ;-). Check out the example utilities/load_site.py if you want to get started in this very productive use of Zope. However, manage_addZSQLMethod refuses to work, always returning a 404 not found error. Don't have time at the moment to find out why, but I wondered if there was a simple reason for it (nothing seems simple at 3am does it)? In the meantime, calling it via an external method to act as glue is working fine.

7 10