[Zope] To thread or not to thread ...

[Terry Hancock]

Hi all,

As yet another iteration in a very image intensive product,
I'm trying to build a tree of images. They are generated
procedurally, and may differ from installation to installation
(they don't currently, but they probably will).

So it's much more convenient to generate them procedurally
in the manage_add<my-product>() function than to store them
in, say, a .zexp file.  (I also tried making them on-the-
fly, but that's a whole 'nother story).

Nevertheless, it takes quite awhile for PIL to chunk
through all those images (currently 2880).

If I just use a straight iteration, the browser will
time-out during the installation process, which is kind
of ugly, and will no doubt freak out my potential user/
administrator.  Also, it's not absolutely clear to me
that Zope won't cut me off somehow if my request takes
too long.  (Any reassurances/warnings on this point?).

Alternatively, I can put components of the installation
process into separate threads.  This nicely drops me
back into the management interface and runs the object-building
process in the background.  However, I then have to think
about checking it to make sure it's done (probably I can
handle this, though I haven't yet -- I just check it
manually later).

The big headache though, is that with the threaded version,
I have folders disappear out of the management interface!

If I wait long enough, eventually they just start to
disappear. Same thing if I restart. So the persistence
machinery must not be working in the threaded case.

But why not?

The code is essentially the same, except that in the
non-threaded version I just call the function, whereas
in the threaded version, I create a thread to run it
with threading.Thread().  I do this at the highest level
in the loop, which means about 4 threads should be started,
which I wouldn't think would tax my system (an earlier
version with about 50 threads did cause a massive slowdown).

Both versions use ordinary 'Folder' objects and create
them using the Zope API with manage_addFolder() calls
on the parent folders.

Should I abandon threading here? (and just document
this in the INSTALL file). Should I use full-fledged
processes? Or am I just forgetting to do something?

(I've done almost no threaded programming before this --
I did run a threaded test version of this code that
generated the files on the filesystem outside of
Zope. All my tests with that version went fine.).

Thanks in advance for any ideas,
Terry

-- 
------------------------------------------------------
Terry Hancock
hancock@anansispaceworks.com       
Anansi Spaceworks                 
http://www.anansispaceworks.com 
P.O. Box 60583                     
Pasadena, CA 91116-6583
------------------------------------------------------