[Zope] Re: Re[4]: Optimizing ZOPE

[alan runyan]

> Not exactly. It's a combination of server load (long requests) and
> other objects that are read. I think that the transaction management
> of Zope have problems with request on objects that take a long time.
> So if a request takes too much time and another request want to access
> another object than it might crash.

have you tried to use the logging features of ZOPE?  z2.py
STUPID_FILE_LOGGER=var/Zope.log

I got this information from
http://www.zope.org/Members/michel/Projects/Interfaces/LOGGING.txt/

I really would like to help try to solve your problems and then we could try
to bring them up to Digital Creations.  ZOPE is a really decent
product and the problems I've seen are people not architecting ZOPE around
their problem (this is not intuitive, like some of ZOPE) but
solving their problems with ZOPE (more intuitive) disregarding how ZOPE
works (things should be abit more bullet proof).


> No. I switched off the syncing and the problems are the same. In debug
> mode of zope I get error messages on console like 'conflict competing
> write ...'. It seems that zope have bugs rendering the DTML sites to
> HTML. That problem does not break every request when this message
> occurs.

I believe the STUPID_FILE_LOGGER will give you more information on this.  I
should have brought it up earlier.  and we should have been discussing this
on the ZOPE mailing list so we could have this archived.   those conflict
competing writes usually have path associated with it.  are all of these
occuring in the same location in the ZODB?

> On Windows sometimes it breaks nothing, sometimes only this request
> (thread) breaks and sometimes the complete zope crashes and the
> process is killed or sometimes the process runs but handles no
> requests. We get this kind of error at least once a day (on good days)
> and every hour on bad days depending on load I think.

yikes, this is very bad.  Andy McKay has helped port ActiveStates website to
ZOPE and I wonder if they are experience the same problems.

> On Linux it's no problem, because the worse case is that the thread is
> killed and started again. Zope is NOT completely killed or hangs. All
> 2-3 months the process crashes, but a little shell script restarts it
> and all is fine.

there is no doubt that ZOPE has more experience and is favored to run on
*nix ,from what I understand.

> Yes. The main point is that you should use zope's capabilities as less
> as possible for such big projects. I use zope nearly only for DTML and
> user authorization. Big Files are in filesystem, database is SQL,
> backend is python, some functions of zope are patched (username
> logging of http_server.py, my own error pages for HTML errors; e.g.
> 404, 401, ...). So I don't really use all zope's possibilities, but
> it's too slow anyway.

the big files should be served from apache and not ZOPE.  that is the lesson
to be learned I believe.

> example:
> I wrote some little benchmarking tests. I do a search query with about
> 40 objects as result. These objects (complete classes with data and
> methods) are return to DTML as a list. DTML access the data to display
> with methods like 'getDate', 'getName', 'get....'. The backend
> generates these objects and sets the values with methods 'setDate',
> 'setName' ,.... . I benchmarked it and the result was that this search
> request needs 1829ms (no exact value, because of python benchmark
> script; I think it's a little bit faster than this values). Than I began
> to optimize and test some methods, functions, ... to get slow
> commands. I tried to avoid these complex objects, generate only simple
> container classes that contains only plain data. I access these values
> from DTML directly with obj.date, obj.name, obj.xxx. I left all
> methods and complex things out. I read MIME type information on
> download and not at file object generating time. I left out my python
> function 'quoteForHTML' that do the HTML Quoting for all special chars
> (I will write a python C module these days if I find no other way).
> And the result was for the same request it took only 400ms. So I have
> to optimize all my code to get it work. That's not simple. The code is
> about 11500 lines. And the object oriented handling is thrown away for
> DTML requests because of speed problems. Python and Zope are not fast
> enough for that complex backend. This code in PHP would be faster and
> does not end in crashes. PHP code is so ugly that I don't want it, but
> it is stable.
>

You say the code is 11500 lines long.  You are using German characters I
assume, I wonder if this is causing any problem?
there is a german ZOPE company beehive that you may want to get in touch
with.  they have some extensive experience and
may be able to out, http://www.beehive.de/index.html .

this is why I would like to see your problem solved.  I have used PHP in the
past and am not a big fan.  It is stable, fast, and has a larger community.
How experience with Python are you? I see you had setName getName, in python
you usually directly set/get attributes with instance.property=value and
value=instance.property, overloading __setattr__ and __getattr__ if you
wanted to do a tiny more than set/get values as well.

> Who are you exactly ? It seems that you know a lot about zope. Are you
> developing on zope ? I read you brought up ZEO, right ?

I'm just some programmer in Texas, USA.  I have been reading and playing
with ZOPE for almost two years.  I am not developing on ZOPE, but may start
in the next few months (I just lost my job this week).  I have not done
anything with ZEO besides getting it up and running in a test environment --
nothing production such as your case.  I'm CCing the ZOPE list.



~runyaga