[ZCM] [ZC] 206/12 Comment "ZCatalogs fail when > 32 elements are passed as part of the searchResults query on PathIndex indexes"
Collector: Zope Bugs and Patches ...
zope-coders@zope.org
Thu, 14 Feb 2002 14:41:57 -0500
Issue #206 Update (Comment) "ZCatalogs fail when > 32 elements are passed as part of the searchResults query on PathIndex indexes"
Status Accepted, Catalog/bug medium
To followup, visit:
http://collector.zope.org/Zope/206
==============================================================
= Comment - Entry #12 by chrisdeckard on Feb 14, 2002 2:41 pm
Ok, so we upgraded to Zope 2.5.0 and everything works fine now with the catalogs. I still can't pinpoint the problem in 2.4.3 and I regretfully don't know enough about the debugger to use it effectively. Sorry.
Not sure if you want to resolve the problem from the collector, but I want to make it clear that I don't think it's a size limitation issue, rather if you make a query which has a path that ends up not having results, then it breaks. Everything I have so far seems to work fine in Zope 2.5.0.
-Chris
________________________________________
= Comment - Entry #11 by ajung on Feb 12, 2002 9:01 am
You stepped with pdb into the UnIndex but no in the related
PathIndex. Please add something like that to use pdb only
when the PathIndex is queried:
if index.id=='path':
import pdb; pdb.set_trace()
________________________________________
= Comment - Entry #10 by chrisdeckard on Feb 11, 2002 4:45 pm
Some other "useful" data:
When I create some queries by hand, I get some strange results that point to this not being a "size" issue.
The catalog contains indexed "events" which are folders. The event folders are stored in a person's folder. It seems that if one of the people does not have any events, then it fails. I tested this with only three people. person1 had events, person2 had events, and person3 did not. If I just test with person1 and person2, it works fine. If I add person3, it fails with the same error message. If I just query for person3, it works fine, no error message, and no results (as expected).
Maybe that will help some?
-Chris
________________________________________
= Comment - Entry #9 by chrisdeckard on Feb 11, 2002 4:41 pm
Here's my step through... I'm going to play with it later on a system that isn't live. I had to cut it off because I didn't want other users to be affected. I also don't know how to interpret this at the moment since I don't know the values of any of the variables:
-Chris
--Return--
> /data/www/Zope.0/lib/python2.1/pdb.py(895)set_trace()->None
-> Pdb().set_trace()
(Pdb) (Pdb) (Pdb) (Pdb) s
> /data/www/Zope.0/Zope/lib/python/Products/ZCatalog/Catalog.py(542)_indexedSearch()
-> r=index._apply_index(request)
(Pdb) s
> /data/www/Zope.0/src/Zope-2.4.3-src/lib/python/Products/PluginIndexes/common/UnIndex.py(339)_apply_index()
-> def _apply_index(self, request, cid='', type=type, None=None):
(Pdb) s
> /data/www/Zope.0/src/Zope-2.4.3-src/lib/python/Products/PluginIndexes/common/UnIndex.py(378)_apply_index()
-> """
(Pdb) s
> /data/www/Zope.0/src/Zope-2.4.3-src/lib/python/Products/PluginIndexes/common/UnIndex.py(381)_apply_index()
-> record = parseIndexRequest(request, self.id, self.query_options)
(Pdb) s
> /data/www/Zope.0/src/Zope-2.4.3-src/lib/python/Products/PluginIndexes/common/util.py(130)__init__()
-> def __init__(self, request, iid, options=[]):
(Pdb) s
> /data/www/Zope.0/src/Zope-2.4.3-src/lib/python/Products/PluginIndexes/common/util.py(137)__init__()
-> """
(Pdb) s
> /data/www/Zope.0/src/Zope-2.4.3-src/lib/python/Products/PluginIndexes/common/util.py(139)__init__()
-> self.id = iid
(Pdb) s
> /data/www/Zope.0/src/Zope-2.4.3-src/lib/python/Products/PluginIndexes/common/util.py(141)__init__()
-> if not request.has_key(iid):
(Pdb) s
> /data/www/Zope.0/src/Zope-2.4.3-src/lib/python/Products/PluginIndexes/common/util.py(146)__init__()
-> usage_param = iid + '_usage'
(Pdb) s
> /data/www/Zope.0/src/Zope-2.4.3-src/lib/python/Products/PluginIndexes/common/util.py(147)__init__()
-> if request.has_key(usage_param):
(Pdb) s
> /data/www/Zope.0/src/Zope-2.4.3-src/lib/python/Products/PluginIndexes/common/util.py(150)__init__()
-> param = request[iid]
(Pdb) s
> /data/www/Zope.0/src/Zope-2.4.3-src/lib/python/Products/PluginIndexes/common/util.py(151)__init__()
-> keys = None
(Pdb) s
> /data/www/Zope.0/src/Zope-2.4.3-src/lib/python/Products/PluginIndexes/common/util.py(152)__init__()
-> t = type(param)
(Pdb) s
> /data/www/Zope.0/src/Zope-2.4.3-src/lib/python/Products/PluginIndexes/common/util.py(154)__init__()
-> if t is InstanceType and not isinstance(param, DateTime):
(Pdb) s
> /data/www/Zope.0/src/Zope-2.4.3-src/lib/python/Products/PluginIndexes/common/util.py(174)__init__()
-> elif t is DictType:
(Pdb) s
> /data/www/Zope.0/src/Zope-2.4.3-src/lib/python/Products/PluginIndexes/common/util.py(190)__init__()
-> """ query is tuple, list, string, number, or something else """
(Pdb) s
> /data/www/Zope.0/src/Zope-2.4.3-src/lib/python/Products/PluginIndexes/common/util.py(192)__init__()
-> if t in SequenceTypes:
(Pdb) s
> /data/www/Zope.0/src/Zope-2.4.3-src/lib/python/Products/PluginIndexes/common/util.py(193)__init__()
-> keys = param
(Pdb) s
> /data/www/Zope.0/src/Zope-2.4.3-src/lib/python/Products/PluginIndexes/common/util.py(197)__init__()
-> for op in options:
(Pdb) s
> /data/www/Zope.0/src/Zope-2.4.3-src/lib/python/Products/PluginIndexes/common/util.py(197)__init__()
-> for op in options:
(Pdb) s
> /data/www/Zope.0/src/Zope-2.4.3-src/lib/python/Products/PluginIndexes/common/util.py(198)__init__()
-> field = iid + "_" + op
(Pdb) s
> /data/www/Zope.0/src/Zope-2.4.3-src/lib/python/Products/PluginIndexes/common/util.py(199)__init__()
-> if request.has_key(field):
(Pdb) s
> /data/www/Zope.0/src/Zope-2.4.3-src/lib/python/Products/PluginIndexes/common/util.py(197)__init__()
-> for op in options:
(Pdb) s
> /data/www/Zope.0/src/Zope-2.4.3-src/lib/python/Products/PluginIndexes/common/util.py(198)__init__()
-> field = iid + "_" + op
(Pdb) s
> /data/www/Zope.0/src/Zope-2.4.3-src/lib/python/Products/PluginIndexes/common/util.py(199)__init__()
-> if request.has_key(field):
(Pdb) s
> /data/www/Zope.0/src/Zope-2.4.3-src/lib/python/Products/PluginIndexes/common/util.py(197)__init__()
-> for op in options:
(Pdb) s
> /data/www/Zope.0/src/Zope-2.4.3-src/lib/python/Products/PluginIndexes/common/util.py(210)__init__()
-> if not keys:
(Pdb) s
> /data/www/Zope.0/src/Zope-2.4.3-src/lib/python/Products/PluginIndexes/common/util.py(213)__init__()
-> self.keys = keys
(Pdb) s
--Return--
> /data/www/Zope.0/src/Zope-2.4.3-src/lib/python/Products/PluginIndexes/common/util.py(213)__init__()->None
-> self.keys = keys
(Pdb) s
> /data/www/Zope.0/src/Zope-2.4.3-src/lib/python/Products/PluginIndexes/common/UnIndex.py(382)_apply_index()
-> if record.keys==None: return None
(Pdb) s
> /data/www/Zope.0/src/Zope-2.4.3-src/lib/python/Products/PluginIndexes/common/UnIndex.py(384)_apply_index()
-> index = self._index
(Pdb) s
> /data/www/Zope.0/src/Zope-2.4.3-src/lib/python/Products/PluginIndexes/common/UnIndex.py(385)_apply_index()
-> r = None
(Pdb) s
> /data/www/Zope.0/src/Zope-2.4.3-src/lib/python/Products/PluginIndexes/common/UnIndex.py(386)_apply_index()
-> opr = None
(Pdb) s
> /data/www/Zope.0/src/Zope-2.4.3-src/lib/python/Products/PluginIndexes/common/UnIndex.py(390)_apply_index()
-> operator = record.get('operator',self.useOperator)
(Pdb) s
> /data/www/Zope.0/src/Zope-2.4.3-src/lib/python/Products/PluginIndexes/common/util.py(216)get()
-> def get(self,k,default_v=None):
(Pdb) s
> /data/www/Zope.0/src/Zope-2.4.3-src/lib/python/Products/PluginIndexes/common/util.py(218)get()
-> if hasattr(self,k):
(Pdb) s
> /data/www/Zope.0/src/Zope-2.4.3-src/lib/python/Products/PluginIndexes/common/util.py(223)get()
-> return default_v
(Pdb) s
--Return--
> /data/www/Zope.0/src/Zope-2.4.3-src/lib/python/Products/PluginIndexes/common/util.py(223)get()->'or'
-> return default_v
(Pdb) s
> /data/www/Zope.0/src/Zope-2.4.3-src/lib/python/Products/PluginIndexes/common/UnIndex.py(391)_apply_index()
-> if not operator in self.operators :
(Pdb) s
> /data/www/Zope.0/src/Zope-2.4.3-src/lib/python/Products/PluginIndexes/common/UnIndex.py(395)_apply_index()
-> if operator=="or": set_func = union
(Pdb) s
> /data/www/Zope.0/src/Zope-2.4.3-src/lib/python/Products/PluginIndexes/common/UnIndex.py(395)_apply_index()
-> if operator=="or": set_func = union
(Pdb) s
> /data/www/Zope.0/src/Zope-2.4.3-src/lib/python/Products/PluginIndexes/common/UnIndex.py(399)_apply_index()
-> range_arg = record.get('range',None)
(Pdb) s
> /data/www/Zope.0/src/Zope-2.4.3-src/lib/python/Products/PluginIndexes/common/util.py(216)get()
-> def get(self,k,default_v=None):
(Pdb) s
> /data/www/Zope.0/src/Zope-2.4.3-src/lib/python/Products/PluginIndexes/common/util.py(218)get()
-> if hasattr(self,k):
(Pdb) s
> /data/www/Zope.0/src/Zope-2.4.3-src/lib/python/Products/PluginIndexes/common/util.py(223)get()
-> return default_v
(Pdb) s
--Return--
> /data/www/Zope.0/src/Zope-2.4.3-src/lib/python/Products/PluginIndexes/common/util.py(223)get()->None
-> return default_v
(Pdb) s
> /data/www/Zope.0/src/Zope-2.4.3-src/lib/python/Products/PluginIndexes/common/UnIndex.py(400)_apply_index()
-> if range_arg is not None:
(Pdb) s
> /data/www/Zope.0/src/Zope-2.4.3-src/lib/python/Products/PluginIndexes/common/UnIndex.py(407)_apply_index()
-> if record.get('usage',None):
(Pdb) s
> /data/www/Zope.0/src/Zope-2.4.3-src/lib/python/Products/PluginIndexes/common/util.py(216)get()
-> def get(self,k,default_v=None):
(Pdb) s
> /data/www/Zope.0/src/Zope-2.4.3-src/lib/python/Products/PluginIndexes/common/util.py(218)get()
-> if hasattr(self,k):
(Pdb) s
> /data/www/Zope.0/src/Zope-2.4.3-src/lib/python/Products/PluginIndexes/common/util.py(223)get()
-> return default_v
(Pdb) s
--Return--
> /data/www/Zope.0/src/Zope-2.4.3-src/lib/python/Products/PluginIndexes/common/util.py(223)get()->None
-> return default_v
(Pdb) s
> /data/www/Zope.0/src/Zope-2.4.3-src/lib/python/Products/PluginIndexes/common/UnIndex.py(413)_apply_index()
-> if opr=="range": # range search
(Pdb) s
> /data/www/Zope.0/src/Zope-2.4.3-src/lib/python/Products/PluginIndexes/common/UnIndex.py(428)_apply_index()
-> for key in record.keys:
(Pdb) s
> /data/www/Zope.0/src/Zope-2.4.3-src/lib/python/Products/PluginIndexes/common/UnIndex.py(428)_apply_index()
-> for key in record.keys:
(Pdb) s
> /data/www/Zope.0/src/Zope-2.4.3-src/lib/python/Products/PluginIndexes/common/UnIndex.py(429)_apply_index()
-> set=index.get(key, None)
(Pdb) s
> /data/www/Zope.0/src/Zope-2.4.3-src/lib/python/ZODB/Connection.py(515)setstate()
-> def setstate(self, object):
(Pdb) s
> /data/www/Zope.0/src/Zope-2.4.3-src/lib/python/ZODB/Connection.py(516)setstate()
-> try:
(Pdb) s
> /data/www/Zope.0/src/Zope-2.4.3-src/lib/python/ZODB/Connection.py(517)setstate()
-> oid=object._p_oid
(Pdb) s
> /data/www/Zope.0/src/Zope-2.4.3-src/lib/python/ZODB/Connection.py(519)setstate()
-> p, serial = self._storage.load(oid, self._version)
(Pdb) s
> /data/www/Zope.0/src/Zope-2.4.3-src/lib/python/ZODB/FileStorage.py(586)load()
-> def load(self, oid, version, _stuff=None):
(Pdb) s
> /data/www/Zope.0/src/Zope-2.4.3-src/lib/python/ZODB/FileStorage.py(587)load()
-> self._lock_acquire()
(Pdb) s
> /data/www/Zope.0/src/Zope-2.4.3-src/lib/python/ZODB/FileStorage.py(588)load()
-> try: return self._load(oid, version, self._index, self._file)
(Pdb) s
> /data/www/Zope.0/src/Zope-2.4.3-src/lib/python/ZODB/FileStorage.py(588)load()
-> try: return self._load(oid, version, self._index, self._file)
(Pdb) s
> /data/www/Zope.0/src/Zope-2.4.3-src/lib/python/ZODB/FileStorage.py(563)_load()
-> def _load(self, oid, version, _index, file):
(Pdb) s
> /data/www/Zope.0/src/Zope-2.4.3-src/lib/python/ZODB/FileStorage.py(564)_load()
-> pos=_index[oid]
(Pdb) s
> /data/www/Zope.0/src/Zope-2.4.3-src/lib/python/ZODB/FileStorage.py(565)_load()
-> file.seek(pos)
(Pdb) s
> /data/www/Zope.0/src/Zope-2.4.3-src/lib/python/ZODB/FileStorage.py(566)_load()
-> read=file.read
(Pdb) s
> /data/www/Zope.0/src/Zope-2.4.3-src/lib/python/ZODB/FileStorage.py(567)_load()
-> h=read(42)
(Pdb) s
> /data/www/Zope.0/src/Zope-2.4.3-src/lib/python/ZODB/FileStorage.py(568)_load()
-> doid,serial,prev,tloc,vlen,plen = unpack(">8s8s8s8sH8s", h)
(Pdb) s
> /data/www/Zope.0/src/Zope-2.4.3-src/lib/python/ZODB/FileStorage.py(569)_load()
-> if doid != oid: raise CorruptedDataError, h
(Pdb) s
> /data/www/Zope.0/src/Zope-2.4.3-src/lib/python/ZODB/FileStorage.py(570)_load()
-> if vlen:
(Pdb) s
> /data/www/Zope.0/src/Zope-2.4.3-src/lib/python/ZODB/FileStorage.py(571)_load()
-> pnv=read(8) # Read location of non-version data
(Pdb) s
> /data/www/Zope.0/src/Zope-2.4.3-src/lib/python/ZODB/FileStorage.py(572)_load()
-> if (not version or len(version) != vlen or
(Pdb) s
> /data/www/Zope.0/src/Zope-2.4.3-src/lib/python/ZODB/FileStorage.py(580)_load()
-> if plen != z64: return read(U64(plen)), serial
(Pdb) s
> /data/www/Zope.0/src/Zope-2.4.3-src/lib/python/ZODB/FileStorage.py(580)_load()
-> if plen != z64: return read(U64(plen)), serial
(Pdb) s
> /data/www/Zope.0/src/Zope-2.4.3-src/lib/python/ZODB/utils.py(107)U64()
-> def U64(v, unpack=struct.unpack):
(Pdb) s
> /data/www/Zope.0/src/Zope-2.4.3-src/lib/python/ZODB/utils.py(108)U64()
-> """Same as u64 but always returns a long."""
(Pdb) s
> /data/www/Zope.0/src/Zope-2.4.3-src/lib/python/ZODB/utils.py(109)U64()
-> h, v = unpack(">II", v)
(Pdb) s
> /data/www/Zope.0/src/Zope-2.4.3-src/lib/python/ZODB/utils.py(110)U64()
-> if h:
(Pdb) s
> /data/www/Zope.0/src/Zope-2.4.3-src/lib/python/ZODB/utils.py(112)U64()
-> return v
(Pdb) s
--Return--
> /data/www/Zope.0/src/Zope-2.4.3-src/lib/python/ZODB/utils.py(112)U64()->423L
-> return v
(Pdb) s
--Return--
> /data/www/Zope.0/src/Zope-2.4.3-src/lib/python/ZODB/FileStorage.py(580)_load()->('((U\x0eBTree...ttQttttq\x17.', '\x03B~\x17\xb5[\x99;')
-> if plen != z64: return read(U64(plen)), serial
(Pdb) s
> /data/www/Zope.0/src/Zope-2.4.3-src/lib/python/ZODB/FileStorage.py(589)load()
-> finally: self._lock_release()
(Pdb) s
> /data/www/Zope.0/src/Zope-2.4.3-src/lib/python/ZODB/FileStorage.py(589)load()
-> finally: self._lock_release()
(Pdb) s
--Return--
> /data/www/Zope.0/src/Zope-2.4.3-src/lib/python/ZODB/FileStorage.py(589)load()->('((U\x0eBTree...ttQttttq\x17.', '\x03B~\x17\xb5[\x99;')
-> finally: self._lock_release()
________________________________________
= Comment - Entry #8 by ajung on Feb 11, 2002 2:15 pm
the request dictionary looks fine and works on my sandbox.
can you try to "import pdb; pdb.set_trace()" before the
_apply_index() call and then step into the function and see
where is breaks?
- aj
________________________________________
= Comment - Entry #7 by chrisdeckard on Feb 11, 2002 2:04 pm
Uploaded: "foobar"
- http://collector.zope.org/Zope/206/foobar/view
I'm attaching a file with a printout of the "request" object, and also the keys of self.indexes.
Because I can't get ahold of the person in charge of the data, I've renamed all of my "people" names to xxx.xxx. They are originally unique and if there is a way I can get them to you without them becoming public, I can probably arrange it.
-Chris
________________________________________
= Comment - Entry #6 by ajung on Feb 11, 2002 11:18 am
I can not reproduce the problem yet.
Please add some debug code before the call
"r=index._apply_index(request)" in Catalog.py in _indexedSearch().
I would like to see how the 'request' variable looks like.
- aj
________________________________________
= Comment - Entry #5 by chrisdeckard on Feb 11, 2002 11:00 am
Was requested that I try this:
dict = { 'path': { 'query': [your long lists ..] } }
zcatalog.searchResults(dict)
but it gives the same error.
I also took out the other indexes that I search, and got the same error.
-Chris
________________________________________
= Comment - Entry #4 by ajung on Feb 11, 2002 9:01 am
check if the following works:
dict = { 'path': { 'query': [your long lists ..] } }
zcatalog.searchResults(dict)
________________________________________
= Assign - Entry #3 by ajung on Feb 11, 2002 9:00 am
Status: Pending => Accepted
Supporters added: ajung
________________________________________
= Comment - Entry #2 by chrisdeckard on Feb 8, 2002 4:53 pm
I posted the wrong version info... Sorry.
It's Zope 2.4.3 with python 2.1.1.
Thanks,
-Chris
________________________________________
= Request - Entry #1 by chrisdeckard on Feb 8, 2002 4:52 pm
Cannot query a zcatalog when I have more than 32 elements in the 'path' portion of a query.
Example code followed by traceback:
Index on a zcatalog object has a PathIndex index called "path".
My query looks like this:
query = {}
query['path'] = [list with > 32 elements]
zcatalog.searchResults(query)
I checked, and there are no duplicate elements (though I don't think
that should matter). My searchResults() works on anything else that
has less than 32 elements in the path list. I haven't tried it for
the other indexes.
And I get the error message:
Zope Error!
Zope has encountered an error while publishing this resource.
If this error persists, please Send Feedback.
Error Type: TypeError
Error Value: invalid argument
Troubleshooting Suggestions
* The URL may be incorrect.
* The parameters passed to this resource may be incorrect.
* A resource that this resource relies on may be encountering an
error.
For more detailed information about the error, please refer to the HTML
source for this page.
If the error persists please contact the site maintainer. Thank you for
your patience.
Traceback (innermost last):
File
/data/www/Zope.0/src/Zope-2.4.3-src/lib/python/ZPublisher/Publish.py, line
171, in publish
File
/data/www/Zope.0/src/Zope-2.4.3-src/lib/python/ZPublisher/mapply.py, line
160, in mapply
(Object: sumEvents)
File
/data/www/Zope.0/src/Zope-2.4.3-src/lib/python/ZPublisher/Publish.py, line
112, in call_object
(Object: sumEvents)
File
/data/www/Zope.0/src/Zope-2.4.3-src/lib/python/Shared/DC/Scripts/Bindings.py,
line 324, in __call__
(Object: sumEvents)
File
/data/www/Zope.0/src/Zope-2.4.3-src/lib/python/Shared/DC/Scripts/Bindings.py,
line 354, in _bindAndExec
(Object: sumEvents)
File
/data/www/Zope.0/src/Zope-2.4.3-src/lib/python/Products/PythonScripts/PythonScript.py,
line 363, in _exec
(Object: sumEvents)
(Info: ({'script': , 'context': , 'container': , 'traverse_subpath':
[]}, ('papers_and_proposals_reviewed', '1.0', '2001', 'papers_reviewed',
'person_type=[person_federal_unit_or_cru_staff]', None, None, None), {},
(None, None, None, None, None, None, None, None)))
File Script (Python), line 185, in sumEvents
File
/data/www/Zope.0/src/Zope-2.4.3-src/lib/python/Products/ZCatalog/ZCatalog.py,
line 653, in searchResults
(Object: CatalogEvents)
File
/data/www/Zope.0/src/Zope-2.4.3-src/lib/python/Products/ZCatalog/Catalog.py,
line 650, in searchResults
File
/data/www/Zope.0/src/Zope-2.4.3-src/lib/python/Products/ZCatalog/Catalog.py,
line 538, in _indexedSearch
File
/data/www/Zope.0/src/Zope-2.4.3-src/lib/python/Products/PluginIndexes/PathIndex/PathIndex.py,
line 362, in _apply_index
(Object: path)
TypeError: invalid argument
==============================================================