Hi all, How does the GIL affect Zope threads? Best regards, Manuel.
--On 4. Mai 2008 09:44:17 -0400 Manuel Vazquez Acosta <mva.led@gmail.com> wrote:
Hi all, How does the GIL affect Zope threads?
Zope threads are Python threads and therefore a thread switch is not possible is some thread is holding the GIL. -aj
Andreas Jung wrote:
--On 4. Mai 2008 09:44:17 -0400 Manuel Vazquez Acosta <mva.led@gmail.com> wrote:
Hi all, How does the GIL affect Zope threads?
Zope threads are Python threads and therefore a thread switch is not possible is some thread is holding the GIL.
-aj
That would mean that the only place where real multitasking occurs is when fetching objects from ZEO and other I/O bounded tasks, isn't? Best regards, Manuel
another scenario is if you are using GIL aware Database adapters like mxODBC to access an RDBMS sam ----- Original Message ----- From: "Manuel Vazquez Acosta" <mva.led@gmail.com> To: "Andreas Jung" <lists@zopyx.com> Cc: "Zope user list" <zope@zope.org> Sent: Sunday, May 4, 2008 9:12:16 AM (GMT-0600) America/Guatemala Subject: Re: [Zope] Zope threads and the GIL Andreas Jung wrote:
--On 4. Mai 2008 09:44:17 -0400 Manuel Vazquez Acosta <mva.led@gmail.com> wrote:
Hi all, How does the GIL affect Zope threads?
Zope threads are Python threads and therefore a thread switch is not possible is some thread is holding the GIL.
-aj
That would mean that the only place where real multitasking occurs is when fetching objects from ZEO and other I/O bounded tasks, isn't? Best regards, Manuel _______________________________________________ Zope maillist - Zope@zope.org http://mail.zope.org/mailman/listinfo/zope ** No cross posts or HTML encoding! ** (Related lists - http://mail.zope.org/mailman/listinfo/zope-announce http://mail.zope.org/mailman/listinfo/zope-dev )
Basically mxODBC is a third party extension and is GIL "aware" as it yields the GIL before accessing the DB ----- Original Message ----- From: "Andreas Jung" <lists@zopyx.com> To: "sam" <python@zeomega.com>, "Manuel Vazquez Acosta" <mva.led@gmail.com> Cc: "Zope user list" <zope@zope.org> Sent: Sunday, May 4, 2008 7:36:08 PM (GMT-0600) America/Guatemala Subject: Re: [Zope] Zope threads and the GIL --On 4. Mai 2008 19:52:38 -0500 sam <python@zeomega.com> wrote:
another scenario is if you are using GIL aware Database adapters like mxODBC to access an RDBMS
What means GIL-aware in this context? -aj
--On 4. Mai 2008 21:05:55 -0500 sam <python@zeomega.com> wrote:
Basically mxODBC is a third party extension and is GIL "aware" as it yields the GIL before accessing the DB
Then it should play nicely with Python. But keep in mind that thread switches can also be blocked by some thread without having the GIL locked. We've seen such situations were a thread had to perform complex regular expression matching operations. Obviously code in Python C extension is also able to lock up the Python interpreter until the result is being passed back to the Python interpreter. Andreas
Andreas Jung wrote:
Then it should play nicely with Python. But keep in mind that thread switches can also be blocked by some thread without having the GIL locked. We've seen such situations were a thread had to perform complex regular expression matching operations. Obviously code in Python C extension is also able to lock up the Python interpreter until the result is being passed back to the Python interpreter.
Another Q: Is AccessControl GIL aware? I mean, this is probably the most used component in Zope. Maybe a performance boost is experienced if those checks can be made simultaneously. Thinking harder, I realize that a tradeoff is in place: Objects can change its security status (the security policy -- i.e, the security object -- can be changed also, but is a weird use case, isn't it?). So maybe write-lock is needed. On the other hand, for objects with a high demand rate, which are kept in memory, good results are to be expected. CacheFu deals OK with Anonymous users, but a boost is to be expected for authenticated users. Best regards, Manuel.
--On 5. Mai 2008 10:11:32 -0400 Manuel Vazquez Acosta <mva.led@gmail.com> wrote:
Andreas Jung wrote:
Then it should play nicely with Python. But keep in mind that thread switches can also be blocked by some thread without having the GIL locked. We've seen such situations were a thread had to perform complex regular expression matching operations. Obviously code in Python C extension is also able to lock up the Python interpreter until the result is being passed back to the Python interpreter.
Another Q: Is AccessControl GIL aware? I mean, this is probably the most used component in Zope. Maybe a performance boost is experienced if those checks can be made simultaneously.
Better tell us what your _real_ question/problem is. I am not getting the point what you are really interested in. -aj
Andreas Jung wrote:
--On 5. Mai 2008 10:11:32 -0400 Manuel Vazquez Acosta <mva.led@gmail.com> wrote:
Andreas Jung wrote:
Then it should play nicely with Python. But keep in mind that thread switches can also be blocked by some thread without having the GIL locked. We've seen such situations were a thread had to perform complex regular expression matching operations. Obviously code in Python C extension is also able to lock up the Python interpreter until the result is being passed back to the Python interpreter.
Another Q: Is AccessControl GIL aware? I mean, this is probably the most used component in Zope. Maybe a performance boost is experienced if those checks can be made simultaneously.
Better tell us what your _real_ question/problem is. I am not getting the point what you are really interested in.
-aj
I'm thinking in CPU-bounded parts of a request: Reindexing, Clustering, Automatic Content Extraction, and the like... We have implemented k-means clustering algorithm on top ZCTextIndex (now plan to port to zc.catalog), although this is batch operation, some other are per-query: Summarizing is one of those: Per cluster, and per document. (Although this is maybe a question for the OTS list). So I'm a little bit worried about how the resource utilization. Best regards, Manuel.
Manuel Vazquez Acosta wrote at 2008-5-4 11:12 -0400:
... That would mean that the only place where real multitasking occurs is when fetching objects from ZEO and other I/O bounded tasks, isn't?
I depends what you mean with "real multitasking". If you mean by this that several CPUs are concurrently active for your Zope process, then you are near. You can, however, also write code for expensive time consuming operations in a language different from Python and then release the GIL before you active such code. Then, this code, too, can occupy a CPU in addition to the Python activity. -- Dieter
Dieter Maurer wrote:
Manuel Vazquez Acosta wrote at 2008-5-4 11:12 -0400:
... That would mean that the only place where real multitasking occurs is when fetching objects from ZEO and other I/O bounded tasks, isn't?
I depends what you mean with "real multitasking".
If you mean by this that several CPUs are concurrently active for your Zope process, then you are near.
Exactly. Several CPUs, I would like to use.
You can, however, also write code for expensive time consuming operations in a language different from Python and then release the GIL before you active such code. Then, this code, too, can occupy a CPU in addition to the Python activity.
I'm very new to Python, and don't grasp yet the C API. Also I would like to implement almost everything in Python: I need this, b/c my of my team's skills in C are None. Best regards, Manuel.
-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1 Manuel Vazquez Acosta wrote:
Dieter Maurer wrote:
Manuel Vazquez Acosta wrote at 2008-5-4 11:12 -0400:
... That would mean that the only place where real multitasking occurs is when fetching objects from ZEO and other I/O bounded tasks, isn't? I depends what you mean with "real multitasking".
If you mean by this that several CPUs are concurrently active for your Zope process, then you are near.
Exactly. Several CPUs, I would like to use.
Generally, the rule-of-thumb for scaling Zope on a multi-CPU box is to run an appserver instance per CPU, each talking to a shared storage (ZEO, usually, but RelStorage would do as well). Tres. - -- =================================================================== Tres Seaver +1 540-429-0999 tseaver@palladion.com Palladion Software "Excellence by Design" http://palladion.com -----BEGIN PGP SIGNATURE----- Version: GnuPG v1.4.6 (GNU/Linux) Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org iD8DBQFIH7wa+gerLs4ltQ4RAndLAJwM+qs7w5hQy0tA9ZY33mJBGPeTBgCeN8fi J50GVau5mZLxpFnRyJLYCIc= =09EA -----END PGP SIGNATURE-----
participants (5)
-
Andreas Jung -
Dieter Maurer -
Manuel Vazquez Acosta -
sam -
Tres Seaver