[Zope] Zope hangs with Ape and large number of objects

[Francis Kelly]

I'm having an issue with Ape that I'm hoping someone might have some 
insight into. I'm running Zope 2.7.3 on a Fedora Core 3 dual processor 
Xeon box with a gig of memory. I've got a FileStorage (generic Data.fs) 
as well as a file-based Ape storage. The Ape storage is based on the 
default file-storage that is described in the component.xml file that 
comes with Ape (I didn't create any mappers or anything). This is a 
development environment, so I'm running Zope in debug mode & I'm using 
VerboseSecurity.

I've loaded about 700,000 objects, each of which is fairly small, 
probably <1k in size. I was interested in using Ape for a couple of 
reasons. Since the system will have a lot of objects in it (probably > 
5M), I wanted to explore ways of storing those objects in more than one 
file to avoid a massive Data.fs file. Ape seemed appealing over mounted 
storage (though I still need to play around a bit more with that) 
because I don't care about undo support (once the objects are loaded 
they won't be modified hardly at all).

My Ape storage is mounted at /fs. When I try to go into /fs via the ZMI, 
the system essentially hangs. Zope starts to consume more and more 
system resources (swapping increases as does top's average load value) 
and the system gets slower and slower. If I cancel the operation, that 
has no effect on the consumption of system resources; I have to restart 
Zope. I've attached a dump produced by the DeadlockDebugger product that 
I think captures the place where Zope seemed to be spending most of its 
time in the event that's helpful. Likewise, I used a reference counting 
routine to dump out the refs in the system. Here are the top 5, you'll 
see that Ape tops the list (which may be normal, I don't know):

(2509, <class apelib.zodb3.connection.UnmanagedJar at 0xb7867e0c>)
(301, <class Shared.DC.Scripts.Bindings.NameAssignments at 0xb79939ec>)
(294, <class Shared.DC.Scripts.Signature.FuncCode at 0xb7993f8c>)
(162, <class DocumentTemplate.DT_Util.Eval at 0xb79d1d1c>)
(161, <class Products.ZGadflyDA.gadfly.kjParser.ParseRule at 0xb7191dac>)

Any insight anyone has on this problem would be very useful. In general, 
recommendations on handling large numbers (>1M) of objects would be very 
useful as well.

Thanks,
Francis

----- stacktrace -------

Thread 126962608 (GET /fs/manage_main):
  File "/usr/local/zope2.7.3/lib/python/ZServer/PubCore/ZServerPublisher.py", line 23, in __init__
    response=response)
  File "/var/zopescale/Products/ZopeProfiler/MonkeyPatcher.py", line 35, in __call__
    return self._function(*args,**kw)
  File "/var/zopescale/Products/ZopeProfiler/ZopeProfiler.py", line 357, in _profilePublishModule
    request=request, response=response)
  File "/usr/local/zope2.7.3/lib/python/ZPublisher/Publish.py", line 384, in publish_module
    environ, debug, request, response)
  File "/usr/local/zope2.7.3/lib/python/ZPublisher/Publish.py", line 175, in publish_module_standard
    response = publish(request, module_name, after_list, debug=debug)
  File "/usr/local/zope2.7.3/lib/python/ZPublisher/Publish.py", line 101, in publish
    request, bind=1)
  File "/usr/local/zope2.7.3/lib/python/ZPublisher/mapply.py", line 88, in mapply
    if debug is not None: return debug(object,args,context)
  File "/usr/local/zope2.7.3/lib/python/ZPublisher/Publish.py", line 39, in call_object
    result=apply(object,args) # Type s<cr> to step into published object.
  File "/usr/local/zope2.7.3/lib/python/Shared/DC/Scripts/Bindings.py", line 306, in __call__
    return self._bindAndExec(args, kw, None)
  File "/usr/local/zope2.7.3/lib/python/Shared/DC/Scripts/Bindings.py", line 343, in _bindAndExec
    return self._exec(bound_data, args, kw)
  File "/usr/local/zope2.7.3/lib/python/App/special_dtml.py", line 175, in _exec
    try: result = render_blocks(self._v_blocks, ns)
  File "/usr/local/zope2.7.3/lib/python/DocumentTemplate/DT_In.py", line 643, in renderwob
    sequence=self.sort_sequence(sequence, md)
  File "/usr/local/zope2.7.3/lib/python/DocumentTemplate/DT_In.py", line 771, in sort_sequence
    else: k = getattr(v, sort)
  File "/var/zopescale/Products/Ape/lib/apelib/zodb3/connection.py", line 359, in setstate
    event = osio.deserialize(oid, obj, classification, state)
  File "/var/zopescale/Products/Ape/lib/apelib/core/io.py", line 160, in deserialize
    mapper.serializer.deserialize(event, state)
  File "/var/zopescale/Products/Ape/lib/apelib/core/serializers.py", line 121, in deserialize
    s.deserialize(event, state)
  File "/var/zopescale/Products/Ape/lib/apelib/zope2/products.py", line 69, in deserialize
    obj._setOb(id, subob)
  File "/var/zopescale/Products/BTreeFolder2/BTreeFolder2.py", line 230, in _setOb
    meta_type = getattr(object, 'meta_type', None)
  File "/var/zopescale/Products/Ape/lib/apelib/zodb3/connection.py", line 359, in setstate
    event = osio.deserialize(oid, obj, classification, state)
  File "/var/zopescale/Products/Ape/lib/apelib/core/io.py", line 160, in deserialize
    mapper.serializer.deserialize(event, state)
  File "/var/zopescale/Products/Ape/lib/apelib/core/serializers.py", line 121, in deserialize
    s.deserialize(event, state)
  File "/var/zopescale/Products/Ape/lib/apelib/zope2/products.py", line 69, in deserialize
    obj._setOb(id, subob)
  File "/var/zopescale/Products/BTreeFolder2/BTreeFolder2.py", line 230, in _setOb
    meta_type = getattr(object, 'meta_type', None)
  File "/var/zopescale/Products/Ape/lib/apelib/zodb3/connection.py", line 322, in setstate
    p, serial = self._storage.load(oid, self._version)
  File "/var/zopescale/Products/Ape/lib/apelib/zodb3/storage.py", line 102, in load
    event, classification, state, hash_value = self._gwio.load(oid)
  File "/var/zopescale/Products/Ape/lib/apelib/core/io.py", line 99, in load
    state, hash_value = mapper.gateway.load(event)
  File "/var/zopescale/Products/Ape/lib/apelib/core/gateways.py", line 67, in load
    state, serial = gw.load(event)
  File "/var/zopescale/Products/Ape/lib/apelib/fs/structure.py", line 111, in load
    classification = event.classify(child_oid)
  File "/var/zopescale/Products/Ape/lib/apelib/core/events.py", line 67, in classify
    return self.conf.classifier.classify_state(sub_event)
  File "/var/zopescale/Products/Ape/lib/apelib/zope2/classifier.py", line 136, in classify_state
    classification, serial = self.gateway.load(event)
  File "/var/zopescale/Products/Ape/lib/apelib/fs/classification.py", line 36, in load
    text = fs_conn.read_annotation(oid, 'classification', '')
  File "/var/zopescale/Products/Ape/lib/apelib/fs/connection.py", line 143, in read_annotation
    annotations = self.afs.get_annotations(path)
  File "/var/zopescale/Products/Ape/lib/apelib/fs/annotated.py", line 98, in get_annotations
    data = self.ops.readfile(props_fn, 1)
  File "/var/zopescale/Products/Ape/lib/apelib/fs/fileops.py", line 47, in readfile
    return f.read()