[Zope-dev] limit to number of operations in a transaction?

[ChristopherN.Deckard]

Hello,
So I've run into a problem.  I had the need to migrate our a
database of people from an old storage format to a new one.  The
old format uses Folders and Properties to store information about
a person.  Name, phone, email, address, etc.  The new format uses a
Product that I wrote which stores said data in a ParsedXML document.

I have roughly 1,500 people in the database.  I wrote a script to
migrate the people.  Basically it does getProperty for each property
on each person folder, then creates the new person object which uses
XML.  Zope apparently cannot handle this number of operations in one
transaction.  Zope gets slower and slower and eventually becomes on
responsive.  It looks like the script has completed, but nothing is
ever committed to the ZODB, and since Zope is unresponsive it must
be restarted.  This, as expected, kills that entire transaction
which was never committed.

Is it known that large numbers of operations, such as above, in a
single transaction can cause problems?  A transaction, of course,
being a request, and an operation being something like
manage_addProduct.  

I've solved the "problem" by using xmlrpc and for person in people
calling my migrate_person script for only one person at a time. 
This is SO MUCH FASTER.  I previously ran the script that migrates
all of the people, and after 8 hours it still had not completed.

Thoughts?

-Chris