[Zope] Why can't I add lots of objects at once?
Kyler B. Laird
laird@ecn.purdue.edu
Wed, 11 Apr 2001 12:42:48 -0500
A month or two ago I asked here for advice on putting
roster data (~233K entries/semester) in our database.
One of the suggestions was to use XML-RPC in order to
avoid the timeouts I was getting.
I've been working on an XML-RPC solution and it is
quite adequate for small data sets, but if I try to
add a few hundred student/class entires, it hangs.
After a few iterations, I've come up with this script
on the Zope side. Essentially, it creates a
structure like
AAE
203
01
01
students
kyler.b.laird.1
sally.r.smith.5
===================================================
if (not csubject in courses.objectIds()):
courses.manage_addFolder(csubject, csubject)
if (not cnumber in courses[csubject].objectIds()):
courses[csubject].manage_addFolder(cnumber, cnumber)
if (not division in courses[csubject][cnumber].objectIds()):
courses[csubject][cnumber].manage_addFolder(division, division)
if (not section in courses[csubject][cnumber][division].objectIds()):
courses[csubject][cnumber][division].manage_addFolder(section, section)
if (not 'students' in courses[csubject][cnumber][division][section].objectIds()):
courses[csubject][cnumber][division][section].manage_addFolder('students', 'students')
current_students = courses[csubject][cnumber][division][section]['students'].objectIds()
for student in students:
if (not student in current_students):
courses[csubject][cnumber][division][section]['students'].manage_addFolder(student, student)
===================================================
Terribly naive programming aside, what's wrong with
calling this a couple hundred times? It seems to
just keep taking more and more of the processor
time. Eventually I give up, kill the server and
restart it. Then I can run the script again and it
will get a little bit further (because it doesn't
have to create the existing objects, I assume).
Of course, I'd prefer not to have to break this up
into one thousand chunks with server restarts in
between each one.
Any suggestions?
Thank you.
--kyler