[Zope-dev] Runaway processes

[Stephan Richter]

On Wednesday 05 December 2007, Stephan Richter wrote:
> On Unix-like systems, we can use `os.fork()`. The advantage of this
> approach is that I can use OS system calls to kill the process. However,
> ZODB database storages cannot be shared between processes. Nikolay Kim has
> done some preliminary experiments and found that `db.open()` locks the
> system (for both, `FileStorage` and `ZeoClientStorage`). I have not
> verified these results or tried to figure out why it is hanging, but I can
> see the problem for `FileStorage`.

Okay, I spent the rest of the day testing the waters. ;-) The results are 
somewhat discouraging but the situation is not hopeless.

FileStorage - All the file handling works just right. However, the object 
index is kept in memory, and the original and forked process do not share the 
same memory space. Thus, once the child process is done doing its 
modifications to the database, the parent does not know about the updated 
index. I have done a small hack that reloads the index and it works. However, 
loading the index can take a long time for large databases. To make this 
approach feasible, we would need to find a way to describe changes in the 
index and send the result to the parent via a file.

(ZEO) ClientStorage - I could not get this to work at all, because at various 
steps in the transaction process, the code tries to allocate a lock, but 
cannot get it, which causes infinite loops.

I have attached a small package that demonstrates the behavior. You can use 
the usual bootstrap/buildout dance. Before running the zeo-based test, you 
need to start the ZEO server using "./bin/zeo-server fg".

Regards,
Stephan
-- 
Stephan Richter
CBU Physics & Chemistry (B.S.) / Tufts Physics (Ph.D. student)
Web2k - Web Software Design, Development and Training
-------------- next part --------------
A non-text attachment was scrubbed...
Name: forktest.tgz
Type: application/x-tgz
Size: 3343 bytes
Desc: not available
Url : http://mail.zope.org/pipermail/zope-dev/attachments/20071206/0290c30d/forktest.bin