[Warning] Zope/ZEO clients: subprocesses can lead to non-deterministic message loss
ATTENTION: Crosspost -- Reply-To set to 'zope-dev@zope.org' Today, I hit a nasty error. The error affects applications under Unix (and maybe Windows) which * use an "asyncore" mainloop thread (and maybe other asyncore applications) Zope and many ZEO clients belong to this class and * create subprocesses (via "fork" and "system", "popen" or friends if they use "fork" internally (they do under Unix but I think not under Windows)). The error can cause non-deterministic loss of messages (HTTP requests, ZEO server responses, ...) destined for the parent process. It also can cause the same output to be send several times over sockets. The error is explained as follows: "asyncore" maintains a map from file descriptors to handlers. The "asyncore" main loop waits for any file descriptor to become "active" and then calls the corresponding handler. When a process forks the complete state, including file descriptors, threads and memory state is copied and the new process executes in this copied state. We now have 2 "asyncore" threads waiting for the same events. File descriptors are shared between parent and child. When the child reads from a file descriptor from its parent, it steals the corresponding message: the message will not reach the parent. While file descriptors are shared, memory state is separate. Therefore, pending writes can be performed by both parent and child -- leading to duplicate writes to the same file descriptor. A workaround it to deactivate "asyncore" before forking (or "system", "popen", ...) and reactivate it afterwards: as exemplified in the following code: from asyncore import socket_map saved_socket_map = socket_map.copy() socket_map.clear() # deactivate "asyncore" pid = None try: pid = fork() if (pid == 0): # child # ... finally: if pid != 0: socket_map.update(saved_socket_map) # reactivate "asyncore" -- Dieter
[Dieter Maurer]
ATTENTION: Crosspost -- Reply-To set to 'zope-dev@zope.org'
Which I've honored.
Today, I hit a nasty error.
The error affects applications under Unix (and maybe Windows) which
* use an "asyncore" mainloop thread (and maybe other asyncore applications)
Zope and many ZEO clients belong to this class
Note a possible complication: ZEO monkey-patches asyncore, replacing its loop() function with one of its own. This is done in ZODB's ThreadedAsync/LoopCallback.py.
and
* create subprocesses (via "fork" and "system", "popen" or friends if they use "fork" internally (they do under Unix but I think not under Windows)).
It may be an issue under Cygwin, but not under native Windows, which supports no way to clone a process; file descriptors may get inherited by child processes on Windows, but no code runs by magic.
The error can cause non-deterministic loss of messages (HTTP requests, ZEO server responses, ...) destined for the parent process. It also can cause the same output to be send several times over sockets.
The error is explained as follows:
"asyncore" maintains a map from file descriptors to handlers. The "asyncore" main loop waits for any file descriptor to become "active" and then calls the corresponding handler.
There's a key related point, though: asyncore.loop() terminates if it sees that the map has become empty. This appears to have consequences for the correctness of workarounds. For example, this is Python's current asyncore loop (the monkey-patched one ZEO installs is similar in this respect): def loop(timeout=30.0, use_poll=False, map=None): if map is None: map = socket_map if use_poll and hasattr(select, 'poll'): poll_fun = poll2 else: poll_fun = poll while map: poll_fun(timeout, map) If map becomes empty, loop() exits.
When a process forks the complete state, including file descriptors, threads and memory state is copied and the new process executes in this copied state. We now have 2 "asyncore" threads waiting for the same events.
Sam Rushing created asyncore as an alternative to threaded approaches; mixing asyncore with threads is a nightmare; throwing forks into the pot too is a good working definition of hell <wink>.
File descriptors are shared between parent and child. When the child reads from a file descriptor from its parent, it steals the corresponding message: the message will not reach the parent.
While file descriptors are shared, memory state is separate. Therefore, pending writes can be performed by both parent and child -- leading to duplicate writes to the same file descriptor.
A workaround it to deactivate "asyncore" before forking (or "system", "popen", ...) and reactivate it afterwards: as exemplified in the following code:
from asyncore import socket_map saved_socket_map = socket_map.copy() socket_map.clear() # deactivate "asyncore"
As noted above, this may (or may not) cause asyncore.loop() to plain stop, in parent and/or in child process. If there aren't multiple threads, it's safe, but presumably you have multiple threads in mind, in which case behavior seems unpredictable (will the parent process's thread running asyncore.loop() notice that the map has become empty before the code below populates the map again? asyncore.loop() will or won't stop in the parent depending on that timing accident).
pid = None try: pid = fork() if (pid == 0): # child # ... finally: if pid != 0: socket_map.update(saved_socket_map) # reactivate "asyncore"
Another approach I've seen is to skip mucking with socket_map directly, and call asyncore.close_all() first thing in the child process. Of course that's vulnerable to vagaries of thread scheduling too, if asyncore is running in a thread other than the one doing the fork() call.
Tim Peters wrote: hello tim, so can we safely assume that zeo does not mix the asyncore implementation with forks or threads and hence does not suffer from the "child concurrently operating on sockets along with parent" syndrome that dieter is experiencing ? appreciate any clarifications. Regards sathya
[Dieter Maurer]
ATTENTION: Crosspost -- Reply-To set to 'zope-dev@zope.org'
Which I've honored.
Today, I hit a nasty error.
The error affects applications under Unix (and maybe Windows) which
* use an "asyncore" mainloop thread (and maybe other asyncore applications)
Zope and many ZEO clients belong to this class
Note a possible complication: ZEO monkey-patches asyncore, replacing its loop() function with one of its own. This is done in ZODB's ThreadedAsync/LoopCallback.py.
and
* create subprocesses (via "fork" and "system", "popen" or friends if they use "fork" internally (they do under Unix but I think not under Windows)).
It may be an issue under Cygwin, but not under native Windows, which supports no way to clone a process; file descriptors may get inherited by child processes on Windows, but no code runs by magic.
The error can cause non-deterministic loss of messages (HTTP requests, ZEO server responses, ...) destined for the parent process. It also can cause the same output to be send several times over sockets.
The error is explained as follows:
"asyncore" maintains a map from file descriptors to handlers. The "asyncore" main loop waits for any file descriptor to become "active" and then calls the corresponding handler.
There's a key related point, though: asyncore.loop() terminates if it sees that the map has become empty. This appears to have consequences for the correctness of workarounds. For example, this is Python's current asyncore loop (the monkey-patched one ZEO installs is similar in this respect):
def loop(timeout=30.0, use_poll=False, map=None): if map is None: map = socket_map
if use_poll and hasattr(select, 'poll'): poll_fun = poll2 else: poll_fun = poll
while map: poll_fun(timeout, map)
If map becomes empty, loop() exits.
When a process forks the complete state, including file descriptors, threads and memory state is copied and the new process executes in this copied state. We now have 2 "asyncore" threads waiting for the same events.
Sam Rushing created asyncore as an alternative to threaded approaches; mixing asyncore with threads is a nightmare; throwing forks into the pot too is a good working definition of hell <wink>.
File descriptors are shared between parent and child. When the child reads from a file descriptor from its parent, it steals the corresponding message: the message will not reach the parent.
While file descriptors are shared, memory state is separate. Therefore, pending writes can be performed by both parent and child -- leading to duplicate writes to the same file descriptor.
A workaround it to deactivate "asyncore" before forking (or "system", "popen", ...) and reactivate it afterwards: as exemplified in the following code:
from asyncore import socket_map saved_socket_map = socket_map.copy() socket_map.clear() # deactivate "asyncore"
As noted above, this may (or may not) cause asyncore.loop() to plain stop, in parent and/or in child process. If there aren't multiple threads, it's safe, but presumably you have multiple threads in mind, in which case behavior seems unpredictable (will the parent process's thread running asyncore.loop() notice that the map has become empty before the code below populates the map again? asyncore.loop() will or won't stop in the parent depending on that timing accident).
pid = None try: pid = fork() if (pid == 0): # child # ... finally: if pid != 0: socket_map.update(saved_socket_map) # reactivate "asyncore"
Another approach I've seen is to skip mucking with socket_map directly, and call asyncore.close_all() first thing in the child process. Of course that's vulnerable to vagaries of thread scheduling too, if asyncore is running in a thread other than the one doing the fork() call.
_______________________________________________ Zope-Dev maillist - Zope-Dev@zope.org http://mail.zope.org/mailman/listinfo/zope-dev ** No cross posts or HTML encoding! ** (Related lists - http://mail.zope.org/mailman/listinfo/zope-announce http://mail.zope.org/mailman/listinfo/zope )
[sathya]
so can we safely assume that zeo does not mix the asyncore implementation with forks or threads and hence does not suffer from the "child concurrently operating on sockets along with parent" syndrome that dieter is experiencing ? appreciate any clarifications.
It's normal for a ZEO application to run asyncore in its own thread. I don't really understand what Dieter is seeing, though: [Dieter]
When a process forks the complete state, including file descriptors, threads and memory state is copied and the new process executes in this copied state. We now have 2 "asyncore" threads waiting for the same events.
A problem is that it's *not* the case that a POSIX fork() clones all threads. Only the thread calling fork() exists in the child process. There's a brief but clear discussion of that here: http://www.opengroup.org/onlinepubs/009695399/functions/fork.html POSIX doesn't even have a way to *ask* that all threads be duplicated, for reasons explained there. Last I heard, Dieter was running LinuxThreads, which fail to meet the POSIX thread spec in several respects. But, AFAICT, fork() under LinuxThreads is the same as POSIX in this particular respect (since threads are distinct processes under LinuxThreads, it would be bizarre if a fork() cloned multiple processes!). I believe native Solaris threads act as Dieter describes, though (fork() clones all native Solaris threads). Dieter, can you clarify which OS(es) and thread package(s) you're using here? Do the things you're doing that call fork() (directly or indirectly) actually run from the thread running asyncore.loop()? That's the only way a POSIX fork() should end up with a clone of the thread running the asyncore loop. But then the subsequent exec (if you're doing system() or popen()) should wipe out the cloned asyncore code before the child process returns to asyncore.
hello tim, thanks for the clarification below and also the pointers to the posix behaviour of fork. The Warning about Zope/ZEO clients in the subject line certainly caused some alarm bells to go off. I am assuming now that dieters description below of using forks does not gel with the ZOPE/ZEO process model i.e there are no forks being called within the code to cause the asyncore thread to be cloned ( even if one were using a non posix compliant thread lib like native solaris it would still be a non issue ). At least thats how I see it so far ! please let me know if anybody thinks otherwise :) Regards Sathya
[sathya]
so can we safely assume that zeo does not mix the asyncore implementation with forks or threads and hence does not suffer from the "child concurrently operating on sockets along with parent" syndrome that dieter is experiencing ? appreciate any clarifications.
It's normal for a ZEO application to run asyncore in its own thread. I don't really understand what Dieter is seeing, though:
[Dieter]
When a process forks the complete state, including file descriptors, threads and memory state is copied and the new process executes in this copied state. We now have 2 "asyncore" threads waiting for the same events.
A problem is that it's *not* the case that a POSIX fork() clones all threads. Only the thread calling fork() exists in the child process. There's a brief but clear discussion of that here:
http://www.opengroup.org/onlinepubs/009695399/functions/fork.html
POSIX doesn't even have a way to *ask* that all threads be duplicated, for reasons explained there.
Last I heard, Dieter was running LinuxThreads, which fail to meet the POSIX thread spec in several respects. But, AFAICT, fork() under LinuxThreads is the same as POSIX in this particular respect (since threads are distinct processes under LinuxThreads, it would be bizarre if a fork() cloned multiple processes!). I believe native Solaris threads act as Dieter describes, though (fork() clones all native Solaris threads).
Dieter, can you clarify which OS(es) and thread package(s) you're using here? Do the things you're doing that call fork() (directly or indirectly) actually run from the thread running asyncore.loop()? That's the only way a POSIX fork() should end up with a clone of the thread running the asyncore loop. But then the subsequent exec (if you're doing system() or popen()) should wipe out the cloned asyncore code before the child process returns to asyncore.
[sathya]
thanks for the clarification below and also the pointers to the posix behaviour of fork. The Warning about Zope/ZEO clients in the subject line certainly caused some alarm bells to go off.
I am assuming now that dieters description below of using forks does not gel with the ZOPE/ZEO process model i.e there are no forks being called within the code to cause the asyncore thread to be cloned
I'm still not clear on either exactly what Dieter did, or on what kind of system. In any case, he wasn't talking about forks in Zope/ZEO directly, but about forks in *application* code. The presence (or absence) of those depends on what applications do, not on what Zope/ZEO do.
( even if one were using a non posix compliant thread lib like native solaris it would still be a non issue ).
I don't think we know enough yet to say. I've never used native Solaris threads, and they're so different from POSIX threads in this respect that I'm not going to guess about how ugly life can be with them. I'm sure there's no problem here with native Windows threads. I don't *see* a way for there to be a problem under POSIX thread semantics, but then I'm still guessing too much about what Dieter actually did. If I were you, I'd ignore this thread until it reaches a conclusion -- if there were widespread problems here, we would have heard about them before in the multiple years Zope and ZEO have been in production (and even under Solaris, I believe most people chose to use Sun's POSIX threads implementation, not the native Solaris threads).
Tim Peters wrote at 2004-6-27 04:46 -0400:
... [Dieter]
When a process forks the complete state, including file descriptors, threads and memory state is copied and the new process executes in this copied state. We now have 2 "asyncore" threads waiting for the same events.
A problem is that it's *not* the case that a POSIX fork() clones all threads. Only the thread calling fork() exists in the child process. There's a brief but clear discussion of that here:
http://www.opengroup.org/onlinepubs/009695399/functions/fork.html
POSIX doesn't even have a way to *ask* that all threads be duplicated, for reasons explained there.
Last I heard, Dieter was running LinuxThreads, which fail to meet the POSIX thread spec in several respects. But, AFAICT, fork() under LinuxThreads is the same as POSIX in this particular respect (since threads are distinct processes under LinuxThreads, it would be bizarre if a fork() cloned multiple processes!). I believe native Solaris threads act as Dieter describes, though (fork() clones all native Solaris threads).
Dieter, can you clarify which OS(es) and thread package(s) you're using here? Do the things you're doing that call fork() (directly or indirectly) actually run from the thread running asyncore.loop()?
The problem occured in a ZEO client which called "asyncore.poll" in the forked subprocess. This "poll" deterministically stole ZEO server invalidation messages from the parent. I read the Linux "fork" manual page and found: fork creates a child process that differs from the parent process only in its PID and PPID, and in the fact that resource utilizations are set to 0. File locks and pending signals are not inherited. ... The fork call conforms to SVr4, SVID, POSIX, X/OPEN, BSD 4.3 I concluded that if the only difference is in the PID/PPID and resource utilizations, there is no difference in the threads between parent and child. This would mean that the wide spread "asyncore.mainloop" threads could suffer the same message loss and message duplication. I did not observe a message loss/duplication in any application with an "asyncore.mainloop" thread. Maybe, the Linux "fork" manual page is only not precise with respect to threads and the problem does not occur in applications with a standard "asyncore.mainloop" thread. -- Dieter
[Dieter Maurer]
The problem occured in a ZEO client which called "asyncore.poll" in the forked subprocess. This "poll" deterministically stole ZEO server invalidation messages from the parent.
I'm sorry, but this is still too vague to guess what happened. - Which operating system was in use? - Which thread package? - In the ZEO client that called fork(), did it call fork() directly, or indirectly as the result of a system() or popen() call? Or what? I'd like to understand a specific failure before rushing to generalization. - In the ZEO client that called fork() (whether directly or indirectly), was fork called *from* the thread running ZEO's asyncore loop, or from a different thread?
I read the Linux "fork" manual page and found:
fork creates a child process that differs from the parent process only in its PID and PPID, and in the fact that resource utilizations are set to 0. File locks and pending signals are not inherited.
...
The fork call conforms to SVr4, SVID, POSIX, X/OPEN, BSD 4.3
If it conforms to POSIX (as it says it does), then fork() also has to satisfy the huge list of requirements I referenced before: http://www.opengroup.org/onlinepubs/009695399/functions/fork.html That page is the current POSIX spec for fork().
I concluded that if the only difference is in the PID/PPID and resource utilizations, there is no difference in the threads between parent and child.
Except that if you're running non-POSIX LinuxThreads, a thread *is* a process (there's a one-to-one relationship under LinuxThreads, not the many-to-one relationship in POSIX), in which case "no difference in threads" is trivially true.
This would mean that the wide spread "asyncore.mainloop" threads could suffer the same message loss and message duplication.
That's why all sane <wink> threading implementations do what POSIX does on a fork(). fork() and threading don't really mix well under POSIX either, but the "fork+exec" model for starting a new process is an historical burden that bristles with subtle problems in a multithreaded world; POSIX introduced posix_spawn() and posix_spawnp() for sane(r) process creation, ironically moving closer to what most non-Unix systems have always done to create a new process.
I did not observe a message loss/duplication in any application with an "asyncore.mainloop" thread.
I don't understand. You said that you *have* seen message loss/duplication in a ZEO client, and I assume the ZEO client was running an asyncore thread. If so, then you have seen loss/duplication in an application with an asyncore thread. Or are you saying that you haven't seen loss/duplication under the specific Linux flavor whose man page you quoted, but have seen it under some other (so far unidentified) system?
Maybe, the Linux "fork" manual page is only not precise with respect to threads and the problem does not occur in applications with a standard "asyncore.mainloop" thread.
That "fork" manpage is clearly missing a mountain of crucial details (or it's not telling the truth about being POSIX-compliant). fork() is historically poorly documented, though.
Tim Peters wrote: just to add my 2 cents I have been looking at zserver code, the only time fork or system (which i presume invokes execve ) calls are used are at startup to either a) run a cmdline b) daemonize heres a snip from strace output strace -o strace.txt -f -e trace=fork,execve ./runzope on a zeo instance 15298 execve("./runzope", ["./runzope"], [/* 23 vars */]) = 0 15298 execve("/var/dev/vision/python233/bin/python", ["/var/dev/vision/python233/bin/py"..., "/var/dev/vision/Zope27/lib/python which creates a child process with id 15299 at this point asyncore main loop thread has not even started so it is safe to assume that the parent does not start the asyncore loop for any servers created but happens in the forked child . which means we probably cannot have multiple asyncore mainloops running The zeoclient causes threads to be created but there are no "forks" or "system" calls as far as I can tell (or strace for that matter) Can you please point out where in the zeo code does forking occur ? I will try and duplicate this condition. -ty sathya
[Dieter Maurer]
The problem occured in a ZEO client which called "asyncore.poll" in the forked subprocess. This "poll" deterministically stole ZEO server invalidation messages from the parent.
I'm sorry, but this is still too vague to guess what happened.
- Which operating system was in use?
- Which thread package?
- In the ZEO client that called fork(), did it call fork() directly, or indirectly as the result of a system() or popen() call? Or what? I'd like to understand a specific failure before rushing to generalization.
- In the ZEO client that called fork() (whether directly or indirectly), was fork called *from* the thread running ZEO's asyncore loop, or from a different thread?
I read the Linux "fork" manual page and found:
fork creates a child process that differs from the parent process only in its PID and PPID, and in the fact that resource utilizations are set to 0. File locks and pending signals are not inherited.
...
The fork call conforms to SVr4, SVID, POSIX, X/OPEN, BSD 4.3
If it conforms to POSIX (as it says it does), then fork() also has to satisfy the huge list of requirements I referenced before:
http://www.opengroup.org/onlinepubs/009695399/functions/fork.html
That page is the current POSIX spec for fork().
I concluded that if the only difference is in the PID/PPID and resource utilizations, there is no difference in the threads between parent and child.
Except that if you're running non-POSIX LinuxThreads, a thread *is* a process (there's a one-to-one relationship under LinuxThreads, not the many-to-one relationship in POSIX), in which case "no difference in threads" is trivially true.
This would mean that the wide spread "asyncore.mainloop" threads could suffer the same message loss and message duplication.
That's why all sane <wink> threading implementations do what POSIX does on a fork(). fork() and threading don't really mix well under POSIX either, but the "fork+exec" model for starting a new process is an historical burden that bristles with subtle problems in a multithreaded world; POSIX introduced posix_spawn() and posix_spawnp() for sane(r) process creation, ironically moving closer to what most non-Unix systems have always done to create a new process.
I did not observe a message loss/duplication in any application with an "asyncore.mainloop" thread.
I don't understand. You said that you *have* seen message loss/duplication in a ZEO client, and I assume the ZEO client was running an asyncore thread. If so, then you have seen loss/duplication in an application with an asyncore thread.
Or are you saying that you haven't seen loss/duplication under the specific Linux flavor whose man page you quoted, but have seen it under some other (so far unidentified) system?
Maybe, the Linux "fork" manual page is only not precise with respect to threads and the problem does not occur in applications with a standard "asyncore.mainloop" thread.
That "fork" manpage is clearly missing a mountain of crucial details (or it's not telling the truth about being POSIX-compliant). fork() is historically poorly documented, though. _______________________________________________ Zope-Dev maillist - Zope-Dev@zope.org http://mail.zope.org/mailman/listinfo/zope-dev ** No cross posts or HTML encoding! ** (Related lists - http://mail.zope.org/mailman/listinfo/zope-announce http://mail.zope.org/mailman/listinfo/zope )
-- =================================================== CEO ZeOmega Open minds' Open Solutions Plano, Texas, USA Bangalore, India 972-731-6750 (O) 214-733-3467 (M) http://www.zeomega.com Open source content management and workflow solutions ====================================================
[sathya]
... The zeoclient causes threads to be created but there are no "forks" or "system" calls as far as I can tell (or strace for that matter) Can you please point out where in the zeo code does forking occur ? I will try and duplicate this condition.
ZEO and ZEO never fork -- they wouldn't be portable if they did (only Unixish systems have fork()). The only forking you'll find here is in startup scripts specific to Unixish systems. Dieter is talking about forks (whether direct or indirect) in *application* code, not in the Zope/ZEO cores. If application code running in a Zope/ZEO process forks, and Zope/ZEO are running asyncore in a thread, and the fork() implementation does clone all threads (which does not happen under Windows or POSIX), then of course the fork() the application code does is going to create a clone of the asyncore thread too. That said, it would indeed be helpful if Dieter revealed enough details about what his app did, and on which kind of system, so it would be possible for others to try to duplicate his results. I have no doubt that he is seeing problems, BTW -- Dieter has an excellent track record.
tim thanks for confirming it. Wont loose sleep over it now. I did not mean to sound like questioning anybodys track record. Since we have ZEO clusters in production it raised alarm bells thats all. Its good to know the problem MAY occur only if u fork in your own app code and the core zeo/zope code by itself is not contributing to it. I guess as a general rule of thumb its not a good idea to use forks in zope application or product code anyway (unless you know what you are doing.) Regards sathya
[sathya]
... The zeoclient causes threads to be created but there are no "forks" or "system" calls as far as I can tell (or strace for that matter) Can you please point out where in the zeo code does forking occur ? I will try and duplicate this condition.
[tim]
ZEO and ZEO never fork -- they wouldn't be portable if they did (only Unixish systems have fork()). The only forking you'll find here is in startup scripts specific to Unixish systems.
Dieter is talking about forks (whether direct or indirect) in *application* code, not in the Zope/ZEO cores. If application code running in a Zope/ZEO process forks, and Zope/ZEO are running asyncore in a thread, and the fork() implementation does clone all threads (which does not happen under Windows or POSIX), then of course the fork() the application code does is going to create a clone of the asyncore thread too.
That said, it would indeed be helpful if Dieter revealed enough details about what his app did, and on which kind of system, so it would be possible for others to try to duplicate his results. I have no doubt that he is seeing problems, BTW -- Dieter has an excellent track record. _______________________________________________ Zope-Dev maillist - Zope-Dev@zope.org http://mail.zope.org/mailman/listinfo/zope-dev ** No cross posts or HTML encoding! ** (Related lists - http://mail.zope.org/mailman/listinfo/zope-announce http://mail.zope.org/mailman/listinfo/zope )
[sathya]
tim thanks for confirming it. Wont loose sleep over it now. I did not mean to sound like questioning anybodys track record.
No, it didn't sound like you were. I mentioned that Dieter has an excellent track record because *I'm* giving him a hard time here <wink>. I'm sure he's seeing problems, but I'm frustrated by the lack of concrete information about how they were provoked and on which kind of system.
Since we have ZEO clusters in production it raised alarm bells thats all. Its good to know the problem MAY occur only if u fork in your own app code
A potential problem is that Unixish systems typically fork "under the covers" for things that look utterly harmless to an application writer -- like the os.system() and os.popen() Dieter mentioned. fork()ing can create horrible problems for threaded applications (which is essentially *why* POSIX only clones the thread calling fork() -- that creates problems too, but the POSIX spec I referenced before discusses all this, so I won't repeat it here).
and the core zeo/zope code by itself is not contributing to it.
I don't think they are, but there's still not enough info here to say for sure.
I guess as a general rule of thumb its not a good idea to use forks in zope application or product code anyway (unless you know what you are doing.)
A more general rule of thumb is to avoid forking a threaded application of any kind. It's easy to follow that rule on OSes that don't have fork <cough>, but unfortunately tricky to avoid on those that do have fork.
Hi Tim, Tim Peters wrote at 2004-6-27 17:06 -0400:
[Dieter Maurer]
The problem occured in a ZEO client which called "asyncore.poll" in the forked subprocess. This "poll" deterministically stole ZEO server invalidation messages from the parent.
I'm sorry, but this is still too vague to guess what happened.
Even when I sometimes make errors, my responses usually contain all relevant information.
- Which operating system was in use?
The ZEO client application mentioned above is almost independent of the operating system -- beside the fact, that is uses "fork" (and therefore requires the OS to support it). Therefore, I did not mention that the application was running on Linux 2.
- Which thread package?
The application mentioned above does not use any thread. Therefore, it is independent of the thread package. Would it use threads it were "LinuxThreads" (but it does not). There is no mystery at all that the application lost ZEO server invalidation messages. It directly follows from the fork semantics with respect to file descriptors. The problem I saw for wider Zope/ZEO client usage came alone from reading the Linux "fork" manual page which indicates (or at least can be interpreted) that child and parent have the same threads. There was no concrete observation that messages are lost/duplicated in this szenario. Meanwhile, I checked that "fork" under Linux with LinuxThreads behaves with respect to threads as dictated by the POSIX standard: the forked process has a single thread and does not inherit other threads from its parent. I will soon check how our Solaris version of Python behaves. If this, too, has only one thread, I will apologize for the premature warning...
- In the ZEO client that called fork(), did it call fork() directly, or indirectly as the result of a system() or popen() call? Or what? I'd like to understand a specific failure before rushing to generalization.
The ZEO client as the basic structure: while 1: work_to_do = get_work(...) for work in work_to_do: pid = fork() if pid == 0: do_work(work) # will not return sleep(...) "do_work" opens a new ZEO connection. "get_work" and "do_work" use "asyncore.poll" to synchronize with incoming messages from ZEO -- no "asyncore.mainloop" around. The "poll" in "do_work" has stolen ZEO invalidation messages destined for the parent such that "get_work" has read old state and returned work items already completed. That is the problem I saw. All this is easy to understand, (almost) platform independent and independant of the thread library. *Iff* a thread library lets a forked child inherit all threads then the problem I announced in this "Warning" thread can occur, as it then behaves similarly to my application above (with an automatic rather than a explicit "poll"). It may well be that there is no thread library that does this. In your words: all thread implementations may be "sane" with respect to thread inheritance... -- Dieter
[Dieter Maurer]
The problem occured in a ZEO client which called "asyncore.poll" in the forked subprocess. This "poll" deterministically stole ZEO server invalidation messages from the parent.
[Tim Peters]
I'm sorry, but this is still too vague to guess what happened.
[Dieter Maurer]
Even when I sometimes make errors, my responses usually contain all relevant information.
I agree, but for whatever reason I'm having a very hard time following this message thread.
- Which operating system was in use?
The ZEO client application mentioned above is almost independent of the operating system -- beside the fact, that is uses "fork" (and therefore requires the OS to support it).
The OS is important because the semantics of fork() depend on the OS.
Therefore, I did not mention that the application was running on Linux 2.
OK, so, e.g., the Solaris fork() semantics play no role in the actual damage you saw.
- Which thread package?
The application mentioned above does not use any thread. Therefore, it is independent of the thread package. Would it use threads it were "LinuxThreads" (but it does not).
You said the app was a ZEO client, and, if that's so, it uses multiple threads whether or not your part of the app creates threads of its own. For example, a ZEO client creates a new thread just to connect to a ZEO server. If this is a ZEO client that never connects to a ZEO server, then perhaps threads are wholly irrelevant.
There is no mystery at all that the application lost ZEO server invalidation messages. It directly follows from the fork semantics with respect to file descriptors.
I can believe that's the truth, but I confess I still don't see how.
The problem I saw for wider Zope/ZEO client usage came alone from reading the Linux "fork" manual page which indicates (or at least can be interpreted) that child and parent have the same threads. There was no concrete observation that messages are lost/duplicated in this szenario.
Good! Thanks.
Meanwhile, I checked that "fork" under Linux with LinuxThreads behaves with respect to threads as dictated by the POSIX standard: the forked process has a single thread and does not inherit other threads from its parent.
I will soon check how our Solaris version of Python behaves. If this, too, has only one thread, I will apologize for the premature warning...
Solaris offers (or imposes <0.9 wink>) choices that don't exist on other platforms. One Solaris choice is whether you link Python with native Solaris threads, or with the Sun POSIX pthreads library. Another choice is whether you call Solaris fork() or Solaris fork1() (note that Python exposes fork1() on platforms that have it -- fork1() clones only the calling threading). The dangerous combination is Solaris threads + Solaris fork(). The other 3 combinations are harmless in this respect. Note that even using Solaris threads, it doesn't follow that places where Linux calls fork() under the covers are also places Solaris calls fork() under the covers. For example, Solaris system() calls Solaris vfork() under the covers, which differs from Solaris fork() in several key respects (and also differs from Solaris fork1()). The most relevant way vfork() differs from fork() under Solaris is that vfork() only clones the calling thread.
- In the ZEO client that called fork(), did it call fork() directly, or indirectly as the result of a system() or popen() call? Or what?
The ZEO client as the basic structure:
while 1: work_to_do = get_work(...) for work in work_to_do: pid = fork() if pid == 0: do_work(work) # will not return sleep(...)
"do_work" opens a new ZEO connection. "get_work" and "do_work" use "asyncore.poll" to synchronize with incoming messages from ZEO -- no "asyncore.mainloop" around.
The "poll" in "do_work" has stolen ZEO invalidation messages destined for the parent such that "get_work" has read old state and returned work items already completed. That is the problem I saw.
Well, don't do that then <wink>.
All this is easy to understand, (almost) platform independent and independant of the thread library.
I still wouldn't say it's easy to understand. While the thread that calls fork isn't running an asyncore loop, it must still be the case that asyncore in the parent has a non-empty map -- yes? If it had an empty map, the child processes would start with a clean slate (map), and so wouldn't pick up socket traffic meant for the parent. If that's so, it looks like just clearing asyncore's map in the child (before do_work()) would solve the (main) problem.
*Iff* a thread library lets a forked child inherit all threads then the problem I announced in this "Warning" thread can occur, as it then behaves similarly to my application above (with an automatic rather than a explicit "poll").
I still don't want to rush to generalizations; as above, even on Solaris with native Solaris threads and clone-everything Solaris fork(), system() should be harmless regardless. I don't know about popen() on Solaris, though; etc etc.
It may well be that there is no thread library that does this. In your words: all thread implementations may be "sane" with respect to thread inheritance...
At least Solaris fork() with Solaris native threads is not sane in this respect. Solaris fork1() with Solaris native threads is sane, ditto any flavor of Solaris fork with Sun pthreads. And sanity is relative <wink>.
Tim Peters wrote at 2004-6-29 00:31 -0400:
... [Dieter]
Meanwhile, I checked that "fork" under Linux with LinuxThreads behaves with respect to threads as dictated by the POSIX standard: the forked process has a single thread and does not inherit other threads from its parent.
I will soon check how our Solaris version of Python behaves. If this, too, has only one thread, I will apologize for the premature warning...
The same is true for Python 2.3.3 generated on Solaris without specification of any specific thread library. It may however be that "gcc" has a preference for the "pthreads" library.
... [Dieter]
The ZEO client has the basic structure:
while 1: work_to_do = get_work(...) for work in work_to_do: pid = fork() if pid == 0: do_work(work) # will not return sleep(...)
"do_work" opens a new ZEO connection. "get_work" and "do_work" use "asyncore.poll" to synchronize with incoming messages from ZEO -- no "asyncore.mainloop" around.
The "poll" in "do_work" has stolen ZEO invalidation messages destined for the parent such that "get_work" has read old state and returned work items already completed. That is the problem I saw.
Well, don't do that then <wink>.
All this is easy to understand, (almost) platform independent and independant of the thread library.
I still wouldn't say it's easy to understand. While the thread that calls fork isn't running an asyncore loop, it must still be the case that asyncore in the parent has a non-empty map -- yes?
Sure, set up by "ZEO.ClientStorage" to receive invalidation messages in anticipation that either a "mainloop" runs or someone calls "poll".
If it had an empty map, the child processes would start with a clean slate (map), and so wouldn't pick up socket traffic meant for the parent.
If that's so, it looks like just clearing asyncore's map in the child (before do_work()) would solve the (main) problem.
That indeed was the solution for my immediate problem... However, I did not only think at my own problem but also what damage this could make for others. And I had read this imprecise "fork" manual page -- which suggested danger for all forking applications with "mainloop"... Thanks to your replies, I now have bookmarked these excellent "opengroup.org" specifications. I will consult them when I will have the next POSIX semantics question... -- Dieter
On Fri, Jun 25, 2004 at 07:23:19PM +0200, Dieter Maurer wrote:
ATTENTION: Crosspost -- Reply-To set to 'zope-dev@zope.org'
Today, I hit a nasty error.
The error affects applications under Unix (and maybe Windows) which
* use an "asyncore" mainloop thread (and maybe other asyncore applications)
Zope and many ZEO clients belong to this class
and
* create subprocesses (via "fork" and "system", "popen" or friends if they use "fork" internally (they do under Unix but I think not under Windows)).
Hm. this applies to external methods and product code that makes these calls? -- Paul Winkler http://www.slinkp.com
Paul Winkler wrote at 2004-6-25 17:41 -0400:
On Fri, Jun 25, 2004 at 07:23:19PM +0200, Dieter Maurer wrote:
ATTENTION: Crosspost -- Reply-To set to 'zope-dev@zope.org'
Today, I hit a nasty error.
The error affects applications under Unix (and maybe Windows) which
* use an "asyncore" mainloop thread (and maybe other asyncore applications)
Zope and many ZEO clients belong to this class
and
* create subprocesses (via "fork" and "system", "popen" or friends if they use "fork" internally (they do under Unix but I think not under Windows)).
Hm. this applies to external methods and product code that makes these calls?
Discussion in "zope-dev@zope.org" suggests that this problem cannot occur as described with a POSIX fork implementation. I observed the problem in a different setup and concluded from the Linux "fork" manual page only that Zope's "asyncore.mainloop" thread will suffer from the same problem. I did not observe the behaviour in this setup and it is well possible that it cannot occur. -- Dieter
ATTENTION: Crosspost -- Reply-To set to 'zope-dev@zope.org' On Friday, I reported a bug that can cause non-deterministic message loss and duplication of messages in forking applications with an "asyncore" mainloop thread. Unfortunately, the proposed workararound does not work for various reasons (as you may already have recognized and reported): * it modifies global variables without protection which is a receipe for desaster in a multi-threaded environment * it resets state that is already activated. Therefore, it is not effective in preventing the main problem. * when applied for "system", it blocks Zope until the the call returns which may be far too long. I am working at another work around. The main ideas is: * Inform "asyncore" about the actor for which its mainloop should execute. * When the actor is set, "asyncore" calls handlers only for this actor and does nothing otherwise. The problem: what is an "actor"? The most natural choice would be the process, identified via its process id. However, under Linux, the process id may not identify the process but the thread. I am still looking for an adequate, platform independent "actor" definition. -- Dieter
participants (5)
-
Dieter Maurer -
Paul Winkler -
sathya -
Tim Peters -
Tim Peters