I have a 1GHz Pentium with 1GB RAM and 1GB swap. After I added all the objects with a little script (that took about 12 hours), I was going to index them to the Catalog I have. (I had to uncomment the index_object method's innards in CatalogAwareness.py because of a problem I mention further down.)
After using up all my RAM and swap, taking five hours and making nothing happen, I gave up trying to index the objects. (This was _with_ subtransactions enabled; set to "10000".)
This is Zope 2.3.1b1, by the way. I'm changing to Zope 2.3.2b2 as we speak, but I don't think it will improve performance that much.
Have you read http://www.zope.org/Members/mcdonc/HowTos/UpgradeToNewCatalog/index_html ? I suspect there will be improvement.
So, to the origin of one of the problems.
This is part of my script to add all these objects:
""" from AccessControl.SecurityManagement import newSecurityManager from ZPublisher.Request import Request from ZPublisher.Response import Response
import sys import Zope import os
os.environ['SERVER_NAME'] = 'localhost' os.environ['SERVER_PORT'] = '80'
response=Response() REQUEST = Request(sys.stdin, os.environ, response)
applic=Zope.app() #applic.REQUEST = REQUEST
uf=applic.acl_users user=uf.getUser('sysadm').__of__(uf) newSecurityManager(None, user) print applic """
Nice, that works. What happends from now on in the script is quite boring; it just adds object. Kinda like:
applic['object'].manage_add_something()
If I do not uncomment those things in CatalogAwareness.py I get an error on SERVER_URL. I think it is resolve_url() that tries to make a run for it.
What are you referring to when you say "those things" in the sentence above? Have you seen the "makerequest.py" script inside of the lib/python/Testing directory? It does what you're trying to do when you set applic.REQUEST=REQUEST (the major difference being that it works ;-). If you read the Zope Developer's Guide testing and debugging chapter from its CVS repository, I believe it's documented in there. If you don't have the lib/python/Testing/makerequest.py file, look on cvs.zope.org for it.
Clever as I thought I was, I added the "applic.REQUEST = REQUEST" line (which, as you can see, is now commented out) and thought that it would fix my problems. Initially, it looked good. Adding the objects (with the CatalogAwareness.py stuff not commented out) was working.
However, when I tried to access any of the newly added objects, Zope died, restarted and pretended like nothing had happend. This is reproducable.
I don't know why this is, but I'd suggest you use makerequest.
What I would really like, is to avoid this after-add-reindex procedure (since I can't make it happen). Is there any way I can produce a "good enough" REQUEST for Zope? So that it doesn't do that die-repeat cycle on me?
Any hints greatly appretiated! :-)
(I haven't got much confidence that the Catalog will be able to respond to any queries with over a million objects indexed, but time will show I guess. I tried a 2.3.1b1 Catalog with 250.000 odd objects, and it wasn't very talkative.)
I'll be curious to see the results. Hopefully you'll have better luck under 2.3.1b2.