[Zope-dev] ZCatalog, REQUEST, misc.

[Erik Enge]

Hi.

I was adding a couple of objects to my system.  Turns out, it's over a
million of them.  I have a 1GHz Pentium with 1GB RAM and 1GB swap.  After
I added all the objects with a little script (that took about 12 hours), I
was going to index them to the Catalog I have. (I had to uncomment the
index_object method's innards in CatalogAwareness.py because of a problem
I mention further down.)

After using up all my RAM and swap, taking five hours and making nothing
happen, I gave up trying to index the objects.  (This was _with_
subtransactions enabled; set to "10000".)

This is Zope 2.3.1b1, by the way.  I'm changing to Zope 2.3.2b2 as we
speak, but I don't think it will improve performance that much.

So, to the origin of one of the problems.

This is part of my script to add all these objects:

"""
from AccessControl.SecurityManagement import newSecurityManager
from ZPublisher.Request import Request
from ZPublisher.Response import Response

import sys
import Zope
import os

os.environ['SERVER_NAME'] = 'localhost'
os.environ['SERVER_PORT'] = '80'

response=Response()
REQUEST = Request(sys.stdin, os.environ, response)

applic=Zope.app()
#applic.REQUEST = REQUEST

uf=applic.acl_users
user=uf.getUser('sysadm').__of__(uf)
newSecurityManager(None, user)
print applic
"""

Nice, that works.  What happends from now on in the script is quite
boring; it just adds object.  Kinda like:

  applic['object'].manage_add_something()

If I do not uncomment those things in CatalogAwareness.py I get an error
on SERVER_URL.  I think it is resolve_url() that tries to make a run for
it.

Clever as I thought I was, I added the "applic.REQUEST = REQUEST" line
(which, as you can see, is now commented out) and thought that it would
fix my problems.  Initially, it looked good.  Adding the objects (with the
CatalogAwareness.py stuff not commented out) was working.

However, when I tried to access any of the newly added objects, Zope died,
restarted and pretended like nothing had happend.  This is reproducable.

What I would really like, is to avoid this after-add-reindex procedure
(since I can't make it happen).  Is there any way I can produce a "good
enough" REQUEST for Zope?  So that it doesn't do that die-repeat cycle on
me?

Any hints greatly appretiated! :-)

(I haven't got much confidence that the Catalog will be able to respond to
any queries with over a million objects indexed, but time will show I
guess.  I tried a 2.3.1b1 Catalog with 250.000 odd objects, and it wasn't
very talkative.)