April 2000 - Zope-Dev

RE: [Zope] Re: Zope hanging (poss. threads-related)
by Amos Latteier 13 Apr '00

13 Apr '00

> > I'm actually wondering if reducing that zombie_timeout (and > > maintenance_interval in medusa/http_server.py) would go > anywhere towards > > alleviating this problem as a temporary measure. Would > there be any reasons > > not to try this? The ZServer zombie stuff is to get rid of zombie client connections, not zombie publishing threads. These are quite different beasts. > > Everything before the call to handle() is 100%. Sometimes, > however, we don't > > get from handle() to the next stage. This is on Zope 2.1.6, > which I've been > > running with up to 100 threads, although I unfortunately > can't excercise > > that many! In general there is little reason to have so many publishing threads. You almost never need that many unless you have a bunch of requests that can take a *long* time. If, on the other hand, you are trying to provoke some kind of thread contention issue I advise you to publish resources that take a long time to return. That way you can easily pile up as many publishing theads as you want to. > > I've added this logging to the Zope 2.1.3 serving the live > site, and will > > report my findings as soon as something untoward occurs. > Maybe others who > > are experiencing hanging would also be able to do some > extra logging and > > report the results [now, there, I see Wiki would be really useful!]. > > > > In the meantime, any suggestions as to where to go next > will be keenly acted > > on! In my experience when a Zope publishing thread hangs its almost always a problem with the published resource. Maybe there's something that puts Zope in a loop that never exits, or maybe there's some DA weirdness that hangs the thead. My advice is to try and identify which requests hang using debug logging and examine the resources that those requests use. -Amos -- Amos Latteier mailto:amos@digicool.com Digital Creations http://www.digicool.com

1 0

Re: [Zope] Re: Zope hanging (poss. threads-related)
by Tres Seaver 13 Apr '00

13 Apr '00

Marcus Collins wrote: > > I'm convinced there's some deep, dark, timing problem, and that it's > thread-related... > > This morning, I had a thread hang, again on Zope 2.1.3 running with four > threads. I was watching for unfinished requests at the time, and caught this > twenty minutes after it occurred. > > I tried viewing a frameset page on the site. No problem. Then I tried > launching /manage, and suddenly *all* the threads hung. This is the first > time all the threads have hung. It's also the first time I've managed to > catch an unfinished request and try viewing framesets within half an hour, > which leads me to believe that, previously, ZServer was doing a good job of > cleaning up the hung zombie threads (recall the zombie_timeout of 30 > minutes). > > I'm actually wondering if reducing that zombie_timeout (and > maintenance_interval in medusa/http_server.py) would go anywhere towards > alleviating this problem as a temporary measure. Would there be any reasons > not to try this? > > I've added quite a lot of DebugLogger stuff to ZServer/PCGIServer.py, and > modified the log analyser accordingly, in the hopes of nailing this sucker: > > def send_response(self): > # create an output pipe by passing request to ZPublisher, > # and requesting a callback of self.log with the module > # name and PATH_INFO as an argument. > self.done=1 > # MC 2000-04-13 additional logging > DebugLogger.log('X', id(self), 'send_response: before PCGIResponse') > response=PCGIResponse(stdout=PCGIPipe(self), stderr=StringIO()) > # MC 2000-04-13 additional logging > DebugLogger.log('X', id(self), 'send_response: before HTTPRequest') > request=HTTPRequest(self.data, self.env, response) > # MC 2000-04-13 additional logging > DebugLogger.log('X', id(self), 'send_response: before handle') > handle(self.server.module, request, response) > > Everything before the call to handle() is 100%. Sometimes, however, we don't > get from handle() to the next stage. This is on Zope 2.1.6, which I've been > running with up to 100 threads, although I unfortunately can't excercise > that many! In fact, it has proven quite a mission to get Zope to hang, maybe > because of the increased latency in serving requests due to the additional > logging. > > I've added this logging to the Zope 2.1.3 serving the live site, and will > report my findings as soon as something untoward occurs. Maybe others who > are experiencing hanging would also be able to do some extra logging and > report the results [now, there, I see Wiki would be really useful!]. > > In the meantime, any suggestions as to where to go next will be keenly acted > on! > > Thanks all! > > -- Marcus There is an outstanding Collector issue related to zombies: http://classic.zope.org:8080/Collector/871/view Amos, does this suggest anything? -- ========================================================= Tres Seaver tseaver(a)digicool.com tseaver(a)palladion.com

1 0

BadRAM: Linux kernel support for broken RAM modules
by Tres Seaver 13 Apr '00

13 Apr '00

This page supplies a kernel mod to deal with flaky DIMM's (ones with known "bad spots", detectable through Memtest86 or some such). http://home.zonnet.nl/vanrein/badram/ Memtest86 might be a good diagnostic for some of the weirder stability problems reported here. http://reality.sgi.com/cbrady_denver/memtest86/ Tres. -- ========================================================= Tres Seaver tseaver(a)digicool.com tseaver(a)palladion.com

1 0

Some thoughts on splitter
by Christian Wittern 13 Apr '00

13 Apr '00

Dear Zopistas, dear Michel, I have some thoughts on splitting text, mainly out of my experience with Chinese texts. 1. The handling of text should depend on some language tagging, similar to HTML4. This ensures that multilingual documents can be treated correctly. For backward compatibility, the default (in absence of a language tag) could be English, and of course the acquisition methods should be used to provide reasonable easy ways for multilingual sites to define the language once and be done with. 2. So far, automatic splitting algorithms have been able to reach 70-90% accuracy, depending on the type of texts, vocabulary and the day of the week. There are some systematic problems, that make it unlikely to see a 100% solution. Even the very good Chasen for splitting Japanese, which is used in JSplitter can not split Japanese with 100% accuracy. The bottomline of this is, there is a need to leave a door open for human intervention in finetuning the splitting. I would therefore think in the direction of a preprocessor to the splitter, that inserts spaces at places it thinks ok to split. This should be done upon importing text in Zope or upon the first save in edits of new text. People working with the texts could then finetune the splitting to avoid blunders. The Zope splitter could then happily just split at the spaces as usual. Of course, depending on the language settings, these spaces should be filtered out before the text is presented in normal view. 3. A completely different approach would be to forget about splitting and just index into every Asian character. Jim Breen employed this method in his Japanese-English dictionary EDICT (). This allows efficient retrieval of arbitrary strings. I am not sure however, how this would adopt to the kind of dynamic indexing required in Zope. All the best, Christian Dr. Christian Wittern Chung-Hwa Institute of Buddhist Studies 276, Kuang Ming Road, Peitou 112 Taipei, TAIWAN Tel. +886-2-2892-6111#65, Email chris(a)ccbs.ntu.edu.tw

2 1

Bug in User.py
by Brian Spilsbury 13 Apr '00

13 Apr '00

There's an insufficiently guarded operation in User.py def allowed(self, parent, roles=None): ... # This is a method, grab it's self. parent=parent.im_self if hasattr(parent, 'aq_inContextOf') and not parent.aq_inContextOf(self.aq_parent.aq_parent,1): if 'Shared' in roles: # Damn, old role setting. Waaa ... I've added the hasattr(parent, 'aq_inContextOf') to make sure that the second clause is reasonable. I'm not sure of the process of submitting fixes, etc, hopefully this will go to the right place.

2 1

Re:[Zope] Re: Zope hanging (poss. threads-related)
by Tres Seaver 13 Apr '00

13 Apr '00

I would *really* like for us to capture the information flowing through the list in some less ephemeral form, so I have set up a Wiki for discussing this and other availability/stability/reliability issues at: http://www.zope.org/Members/tseaver/Projects/HighlyAvailableZope/ I made a messy first stab at putting some of the current thread into a page under TwistingInTheWindProblems. Please feel free to edit, or tell me that this is a bad idea. Tres. -- ========================================================= Tres Seaver tseaver(a)digicool.com tseaver(a)palladion.com

1 0

Finding the transaction timestamp in ZODB
by Grakka 13 Apr '00

13 Apr '00

Hi there, Can enybody explain the format of the "byte serial" on the Data.fs? the FileStorage.py reads: "8 byte serial, type stamp that matches the transaction timestamp" When I u64() the read contents, I ger a string like this one (without the quotes): "230720283629242325L" I believe this reflects some sort of timestamp, but can't figure it out how to convert it to "readable" date/time value, anyone can help? Thanks in advance.

2 1

More on DCOracle and BLOB handling
by Cheuk Cheng 12 Apr '00

12 Apr '00

First I appreciated for those who responded to my original posting. So far I still haven't found a solution but yet learned a few new things and would like to share with everyone. For those who didn't read my previous message, my problem was trying to use a Zope external method and doing a blob insertion into Oracle (8.0.5 on Solaris). I'm using the DCOracle compiled with VC6 on a NT platform. With DCOracle I could create the connection and query the database without any problem for any kind of datatypes including blob. When trying to perform a insertion, it keeps reporting a "invalid hex string" error. Now here's what I found so far. When I was searching for a solution for my problem, I came across the mxODBC driver. I created a standalone Python script that makes use of mxODBC driver and successfully inserted data blobs of any size (>10M). So I thought that I could copy the mxODBC package from python\lib\ODBC to Zope (under Shared\ODBC\) and access it via an external method. Then I found a naming conflict with the Zope odbc driver. I went ahead and renamed the odbc.pyd to win32odbc.pyd. Next I tried to import the mxODBC driver in my external method as: import Shared.ODBC from Shared.ODBC import Windows dbc = Shared.ODBC.Windows.Connect('user','password','dsn',clear_auto_commit=1) When the external method is executed, I got an error message "Error Type:AttributeError ErrorValue: Connect". If I successfully imported the module, shouldn't I have access to the connection method? Anyone has any idea on what I've done wrong? Also I would like to find out if there's anyone successfully using DCOracle with Oracle 8.0.5 (on any platform) to insert blob. Please comment. Thanks -Cheuk

1 0

Invisible instance bug with Sub ZClasses
by Jason Spisak 12 Apr '00

12 Apr '00

Zopists, In the Collector, tickets [1178], [1175], [1161] all refer to the same bug. This seems to be a pretty significant set back to anyone using ZClass to create an application.(Which is probably a high percentage) Is this holding anyone else's process up majorly? It's only a problem with 2.1.6. I haven't checked it against CVS, has anyone else? Alexandre? All my best, -- Jason Spisak 444(a)hiretechs.com

1 0

Re: [Zope-dev] Import restriction?
by Tres Seaver 12 Apr '00

12 Apr '00

mindlace <mindlace(a)imeme.net> > I'm curious as to why "Import" is limited to the local file-system. > Would I be committing some sort of serious security error if I allowed > users to upload their own exported folders, etc, through that interface? I just rejected a patch which allowed through-the-web import. See http://classic.zope.org:8080/Collector/894/view for a discussion of the security issues. Tres. -- ========================================================= Tres Seaver tseaver(a)digicool.com tseaver(a)palladion.com

1 0