[Zope] Re: Threading, dtml and performance?!

Paul Zwarts paz@oratrix.com
Wed, 7 Nov 2001 16:02:08 +0100


Hi all,

As a suggestion for Solaris users (Im on linux) please be sure to check
that your PYTHON thread is the culprit and NOT the database thread. They
are indeed two separate issues. Why I say this is because I don't know
much about Solaris, but on Redhat linux (kernel 2.4+) a ps -ef command
returns the status of the database thread.

What we experienced months ago was the very same issue. (as per this
thread in its history). We had thought that python was the culprit (we
used to be on Irix 6.5 which doesn't feedbak thread status). What it
turned out to be was UPDATE WAITING or COMMIT WAITING was appended to
the thread. Not python at all, but the database driver. When a stalled
thread ocurrs, zope stalls because its waiting for the thread to close.
This action is required because it supports the required sql locking
schema. Otherwise you get ancient problems of updates impinging on
updates causing bad data integrity.

We had never experienced crashes, related directly to this, but we have
had crashes. That at first made me think of this same problem, but then
watching activity in the
manage_interface/control_panel/debug/refresh+1second showed me a python
process in a loop that caused the crash. That culprit was entirely bad
programming practice on my part.

I don't know if this blurb helps anyone, but I hope I can cause some
ideas to occur per narrowing down the possibilities.

It may also be worth noting that we still run zope 2.3.3 to this date
because python2 just seems simply to be slower. Perhaps this weekend
I'll take a second stab at building a test server on python2 and the
latest zope. If I learn anything more, I'll update...

Cheers,
Paul Zwarts

-----Original Message-----
From: zope-admin@zope.org [mailto:zope-admin@zope.org] On Behalf Of Tony
McDonald
Sent: Wednesday, November 07, 2001 3:48 PM
To: Oliver Erlewein
Cc: zope@zope.org
Subject: Re: [Zope] Re: Threading, dtml and performance?!

On 7/11/01 2:20 pm, "Oliver Erlewein" <oerlewein@gmx.de> wrote:

> Hi Tony
> 
> No, I havn't solved the problem but after restarting production (after
19
> days) I noticed that the funny way the threadcount reacted. I'd say,
that the
> threads don't get killed after they have finished. I also noticed that
the
> memory use went down from 344Mb to about 26Mb. Now I've put a restart
into my
> nightly cronjob. (Are there some reasons why  Zope should not be
restarted
> dayly?????)
> 

Not that I'm aware of. I know some people do restart their servers
daily. I
had thought about doing that when I had *massive* memory bloat on a
machine
with a small amount of memory. However, we now have a machine with more
memory, but the problem disappeared with a new zope release (could have
been
as early as the 2.2 series I think, I can't be sure though).

> I've just tested the whole thing on a WinNT 4.0 PC with Zope 2.4.2
(binary
> distribution). No problems! The PC gets slower but can answer two
browser
> instances parallel.
> If I do the same thing on Solaris (intel & sun servers!) it is
impossible!

I've not tried on Intel Solaris, just Sparc. You're absolutely right
though,
nothing I have tried so far has enabled me to get more than one thread
at a
time serviced under a Solaris server.

> Normally you have sortof a break in a loop where control is given to
the other
> parallel threads. This seems not to happen on my source compile with
> Solaris. I'm not so firm on where to address this problem (or if the
problem
> was 
> already adressed elswhere e.g. python maillists).

I've brought it up on the main python list. Some people responded with
suggestions and one person said they would try to get some threading
fixes
into a python 2.1.2 release. However, I grabbed some of the source code
from
the python 2.2b1 release (which gives test_thread.py output similar to
that
on my MacOS-X machine - which *does* service more than one thread) and
compiled up a 2.1 hybrid python that gave some similar output to that of
my
MacOS-X and Solaris 2.2b1 python. I then built a 2.4.1 release from
source,
using this hybrid python and *still* found that I could only service one
thread at a time.

I don't know if I'm doing anything wrong or what, my knowledge of python
innards is virtually nil. I don't know if I've linked against the right
thread libraries or what.

> Therefore I'd really
> apreciate some help! ;-) I don't want to start thinking about a
migration
> Linux or
> even (puke) Windows.

Same here... :(

> 
> On Solaris I'm still using 2.4.0. Does it make sense to upgrade? I've
seen
> that in Zope 2.4.3b1 there was some issue with dtml-recursion?

I've had some core dumps from 2.4.0, but not had time to do a 'big M' on
them to see what's happening. We are also running 2.4.1, I've no plans
to
upgrade any further for the time being. The thread issue does not go
away
with Zope 2.4.1 though :(

> 
> Greetings from a rany Cologne!
> Oliver
> 

And a cold Newcassle!
Tone.
-- 
Dr Tony McDonald,  Assistant Director, FMCC, http://www.fmcc.org.uk/
The Medical School, Newcastle University Tel: +44 191 243 6140
A Zope list for UK HE/FE  http://www.fmcc.org.uk/mailman/listinfo/zope


_______________________________________________
Zope maillist  -  Zope@zope.org
http://lists.zope.org/mailman/listinfo/zope
**   No cross posts or HTML encoding!  **
(Related lists - 
 http://lists.zope.org/mailman/listinfo/zope-announce
 http://lists.zope.org/mailman/listinfo/zope-dev )