April 2000 - Zope-Dev

Importing into ZODB AND updating
by Grakka 17 Apr '00

17 Apr '00

Can anyone provide help about importing an exported ZODB (ZEXP) file and UPDATE any objects that are newer to the existing ones? Right now, when ZOPE (ObjectManager.py) finds that another record with the same ID, it raises an exception "Object Is Already In Use", but I want to update it instead..... In both the ZODB and the ZEXP file, each object contains the 8-byte serial, which is the TimeStamp of the last modified date of the object, I can access the TimeStamp of the ID located in ZODB, but I do not know how to access the TimeStamp serial on the IMPORTED file, here is a chunk of code from ObjectManager.py def _checkId(self, id, object, allow_dup=0): # If allow_dup is false, an error will be raised if an object # with the given id already exists. If allow_dup is true, # only check that the id string contains no illegal chars. if not id: raise 'Bad Request', 'No id was specified' if bad_id(id) != -1: raise 'Bad Request', ( 'The id %s contains characters illegal in URLs.' % id) if id[0]=='_': raise 'Bad Request', ( 'The id %s is invalid - it begins with an underscore.' % id) if not allow_dup: if hasattr(self, 'aq_base'): self=self.aq_base ----------->>> if hasattr(self, id): ----------->>> #self._getOb(id)._p_serial has the ZODB (not the export) timestamp ----------->>> #object._p_serial yields the same result as above.. ----------->>> raise 'Bad Request', ( ----------->>> 'The id %s is invalid - it is already in use' % id)) -------------snip------------- On the parameter definition, "object" is defined: ob=self._p_jar.importFile(file) My understanding is that the importFile module first "places" all the exporte file in the ZODB database, but makes the objects "unreachable [??]" until it checks for dupes ans so on.. I know this mail is long enough but if you read this far I want to thank you in advance :) Julio

1 0

Re: [Zope-dev] compile warning
by Tres Seaver 17 Apr '00

17 Apr '00

"Dr. Ross Lazarus" <rossl(a)med.usyd.edu.au> > I've noticed this flash by before when compiling source distributions > > rossl on innana~/zope216>python wo_pcgi.py > wo_pcgi.log > ./../Components/zlib/zlib.c: In function `PyZlib_compress': > ./../Components/zlib/zlib.c:61: warning: assignment from incompatible > pointer type > rossl on innana~/zope216> > > fresh zope 2.1.6 source on redhat 6.1/intel. > AFAIK it's been there for several releases - I'd never thought to ask... > - in the light of the mysterious core dump/thread problem, does this > warning matter? Its a typo, but harmless, since the pointer being cast to the wrong type before assignment is NULL. Here's a quick patch if you'd like to squish the warning (I'll go ahead and check this in on the head of the CVS trunk):: diff -u -r1.3 zlib.c --- zlib.c 1997/08/07 18:21:15 1.3 +++ zlib.c 2000/04/17 19:59:19 @@ -58,7 +58,7 @@ "Can't allocate memory to compress data"); return NULL; } - zst.zalloc=(free_func)Z_NULL; + zst.zalloc=(alloc_func)Z_NULL; zst.zfree=(free_func)Z_NULL; zst.next_out=(Byte *)output; zst.next_in =(Byte *)input; -- ========================================================= Tres Seaver tseaver(a)digicool.com tseaver(a)palladion.com

1 0

icon_for_meta_type on Zope.org
by Jason Spisak 17 Apr '00

17 Apr '00

Zope.org authors, Is the icon_for_meta_type() on the Search Results pages an external method? I'll bet it is. :) Is that code available. I have been trying to get the icon for a given it's meta_type without success and that would really help. All my best, -- Jason Spisak 444(a)hiretechs.com

2 1

RE: [Zope-dev] Re: Zope hanging (poss. threads-related)
by Marcus Collins 17 Apr '00

17 Apr '00

> -----Original Message----- > From: Dr. Ross Lazarus [mailto:rossl@med.usyd.edu.au] > Sent: 17 April 2000 11:38 > To: zope-dev(a)zope.org > Subject: [Zope-dev] Re: Zope hanging (poss. threads-related) <snip> > I think it was worth the effort even though it didn't break anything - > one thing we really need is a reproducible way to make this problem > appear. As Amos has said, the problem may lie deep in the > bowels of the applications running on top of zserver. Given that there > was mention that freebsd installations seemed immune, we have the the > Linux thread libraries to content with as well as all the zope code > running at crash prone sites..anyone got this happening with an old > linux install? I only have access to redhat 6.1 I don't think us FreeBSD users are quite immune... So far, we've had reports of hanging/restarts/crashes on FreeBSD, Linux and Solaris. The three use vastly different thread libraries, so I don't think it's your thread libraries that are at fault. > I've added a comment to Marcus's wiki page for collecting data on this > problem - suggest anyone else with the problem does the > same...http://www.zope.org/Members/tseaver/Projects/HighlyAvai > lableZope/DiagnosingHangProblems Thanks for the feedback! Regards, -- Marcus

1 0

RE: [Zope-dev] Re: Zope hanging (poss. threads-related)
by Marcus Collins 17 Apr '00

17 Apr '00

> -----Original Message----- > From: Tres Seaver [mailto:tseaver@palladion.com] > Sent: 17 April 2000 01:20 > To: zope-dev(a)zope.org > Cc: Michel Pelletier; Marcus Collins; Tony Rossignol > Subject: Re: [Zope-dev] Re: Zope hanging (poss. threads-related) <snip> > Here are a couple of commonalities we need to look at: > > * IE 5 seems to be involved. >From combining the logs of Apache and DebugLogger, it doesn't seem restricted to IE5. The following are user agents that, it seems, initiated hung threads over the weekend: * Mozilla/4.0 (compatible; MSIE 5.0; AOL 5.0; Windows 98; DigExt) * Mozilla/4.0 (compatible; MSIE 4.01; Windows NT) * Mozilla/4.7 [en] (Win98; I) > * The management interface seems to trigger the problem (no > application-specific code involved, just manipulation of "stock" > Zope objects). Although I've experienced hanging after using the management interface, this doesn't seem to be the sole cause. There was no manage activity between Zope's last restart and our weekend's hangings. > * Framesets may be involved. Definitely. Although other threads, representing non-frameset documents, have also failed to complete, it seems the the initial hanging is *always* brought on by frameset documents. > * Perhaps premature request cancellation (likely to happen often > in the management interface) is involved. Quite possibly, although it would be incredibly hard to determine a correlation from log files... > I would like to see a description of the interaction between the > ZServer-async-select() thread and the Zope threads to which > it dispatches requests, particularly what happens when a Zope thread > tries to write to a connection which has been closed by the client > without a "clean" shutdown (e.g., attempting to write to the socket > with fail with errno == EPIPE). Do you suspect that IE 5 is not closing connections properly, or that ZServer is not handling broken connections properly? > I have a couple of testcases for this problem. > (Un)fortunately, I don't use IE5 (and don't go with girls who do! > :), and therefore they all work without a hitch. <snip> I've been exercising your test script with IE and Netscape, and have several unfinished requests, although I think at least some of these are the result of premature request cancellations. However, I did come across several instances of the following, which possibly warrant attention/explanation: request 143894896 did not end B 143894896 2000-04-17T14:58:48 GET /mctest/breakZope/test_stall?seconds:int=30 I 143894896 2000-04-17T14:58:48 0 A 143894896 2000-04-17T15:01:05 200 0 B 143894896 2000-04-17T15:01:45 GET /mctest/breakZope/test_stall?seconds:int=30 I 143894896 2000-04-17T15:01:45 0 A 143894896 2000-04-17T15:04:23 200 0 It seems strange that *the same thread*, having failed to complete the request, should begin another. This was evident from the latest hanging log data from our live site, too, but in more detail and with a twist: request 145377824 did not end I 145377824 2000-04-15T07:21:53 found_terminator: using StringIO; size=2057 B 145377824 2000-04-15T07:21:53 POST /ContentInput/Work/Generate/ConfirmSubmitContent I 145377824 2000-04-15T07:21:53 found_terminator: using StringIO; size=7300 I 145377824 2000-04-15T07:21:53 0 X 145377824 2000-04-15T07:21:53 send_response: before PCGIResponse X 145377824 2000-04-15T07:21:53 send_response: before HTTPRequest X 145377824 2000-04-15T07:21:53 send_response: before handle These must surely be two requests (see the different sizes), although I am really puzzled at how two different requests to this page (incidentally, not a frameset) could be made inside of a second, by the same person. It's also peculiar that the log begins with "I", not "B". The plot thickens... In the latest spate of hangings, a thread hung, but *some* requests continued to be served, while failed. This behaviour continued until it appeared that all requests had failed, at which point Zope was restarted (meatware process). During this time period, frameset pages were requested, and at least some failed. The above information is expanded on at http://www.zope.org/Members/tseaver/Projects/HighlyAvailableZope/MarcusColli ns/ Thanks again, -- Marcus

1 0

RE: [Zope] Re: Zope hanging (poss. threads-related)
by Tony Rossignol 17 Apr '00

17 Apr '00

More feedback here. I wasn't sure if this should warrant a new title or not so I've left it here, for those following the thread. I have, on occasion, found threads spinning for no apparent reason on our stable machine. One theory here is that the older slower machine we have that does not restart, may actually still be suffering from similar issues as the other servers but for one reason or another it does not trigger the restart process or does not get severe enough to restart. So when I see a spinner I am starting to pay attention to it. Here is some more info: top - 8:08am up 131 days, 20:18, 1 user, load average: 1.32, 1.28, 1.30 95 processes: 93 sleeping, 2 running, 0 zombie, 0 stopped CPU states: 0.1% user, 0.0% system, 0.0% nice, 0.0% idle Mem: 517528K av, 511300K used, 6228K free, 80676K shrd, 10832K buff Swap: 130748K av, 7068K used, 123680K free 153468K cached PID USER PRI NI SIZE RSS SHARE STAT LIB %CPU %MEM TIME COMMAND 31771 nobody 17 0 110M 110M 18132 R 0 96.7 21.9 537:41 python 9436 root 7 0 936 936 728 R 0 12.0 0.1 0:00 top 30717 nobody 0 0 207M 207M 18144 S 0 4.3 41.0 7:45 python 30763 nobody 0 0 207M 207M 18144 S 0 3.4 41.0 11:01 python ... strace -p 31771 - close(-1) = -1 EBADF (Bad file descriptor) close(-1) = -1 EBADF (Bad file descriptor) ... I don't know what to make of this but maybe it will help. During this time Zope was still able to serve content at almost a normal pace. -- ------------------------------- tonyr(a)ep.newtimes.com Director of Web Technology New Times, Inc. -------------------------------

1 0

compile warning
by Dr. Ross Lazarus 17 Apr '00

17 Apr '00

I've noticed this flash by before when compiling source distributions rossl on innana~/zope216>python wo_pcgi.py > wo_pcgi.log ./../Components/zlib/zlib.c: In function `PyZlib_compress': ./../Components/zlib/zlib.c:61: warning: assignment from incompatible pointer type rossl on innana~/zope216> fresh zope 2.1.6 source on redhat 6.1/intel. AFAIK it's been there for several releases - I'd never thought to ask... - in the light of the mysterious core dump/thread problem, does this warning matter? -- Dr Ross Lazarus Associate Professor and Sub-Dean for Information Technology Faculty of Medicine, Room 126A, A27, University of Sydney, Camperdown, NSW 2006, Australia Tel: (+61 2) 93514429 Mobile: +61414872482 Fax: (+61 2) 93516646 Email: rossl(a)med.usyd.edu.au

1 0

Re: Zope hanging (poss. threads-related)
by Dr. Ross Lazarus 17 Apr '00

17 Apr '00

Fired up 2.1.6 with default threads. (This particular installation used to crash at least once a day until I set single threaded mode) Assaulted it using this example code for 15 minutes or so with multiple IE5 clients behaving as badly as I could make them - no sign of core dump or a restart.....not sure if this is helpful or not. I didn't try multiple ip addresses for all the clients - all were on my workstation. I think it was worth the effort even though it didn't break anything - one thing we really need is a reproducible way to make this problem appear. As Amos has said, the problem may lie deep in the bowels of the applications running on top of zserver. Given that there was mention that freebsd installations seemed immune, we have the the Linux thread libraries to content with as well as all the zope code running at crash prone sites..anyone got this happening with an old linux install? I only have access to redhat 6.1 I've added a comment to Marcus's wiki page for collecting data on this problem - suggest anyone else with the problem does the same...http://www.zope.org/Members/tseaver/Projects/HighlyAvailableZope/Dia… Sigh. Back to single thread. Gak. Tres Seaver wrote > I have a couple of testcases for this problem. (Un)fortunately, I don't use IE5 > (and don't go with girls who do! :), and therefore they all work without a > hitch. > > First, an ExternalMethod, implemented so:: -- Dr Ross Lazarus Associate Professor and Sub-Dean for Information Technology Faculty of Medicine, Room 126A, A27, University of Sydney, Camperdown, NSW 2006, Australia Tel: (+61 2) 93514429 Mobile: +61414872482 Fax: (+61 2) 93516646 Email: rossl(a)med.usyd.edu.au

1 0

zope and writes
by mindlace 17 Apr '00

17 Apr '00

Zopistas, There've been various comments about how the ZODB isn't particularly suited towards write-intensive scenarios. What I'm wondering is whether this shouldn't be altered to be rewrite intensive scenarios. For example, if I add an object to the ZODB, I've just added the object. However, if I *change* that object, I now have two objects- as it is now, and as it was previously. Now, is this all inclusive? I'm thinking of using the ZODB to store information like a mailing list archive. The vast majority of the data- the messages themselves- would never change. However, I'd like to have meta-data, like scoring on messages, that would change. If I change a property on a DTML Document, does the ZODB store a copy of all of the document's other properties (which haven't changed?) If so, is there a way to pack on a "per-object" basis, or would I just need to use some kind of non-versioning storage? Thanks, -- ethan mindlace fremen mindlace(a)imeme.net zope -&- imap email -&- mailing list weave your web with the web at http://imeme.net

3 2

Re: [Zope-dev] Re: Zope hanging (poss. threads-related)
by Tres Seaver 16 Apr '00

16 Apr '00

Michel Pelletier <michel(a)digicool.com> wrote: > > Marcus Collins wrote: > > > > > -----Original Message----- > > > From: Tony Rossignol [mailto:tonyr@ep.newtimes.com] > > > Sent: 14 April 2000 19:19 > > > To: Marcus Collins; zope-dev(a)zope.org > > > Subject: Re: [Zope-dev] RE: [Zope] Re: Zope hanging (poss. > > > threads-related) > > > > > Thank you for starting this. I'll try to gather up information > > > I've been trying to collect here and post it in the next few days. > > > > Thanks! Maybe you could also look at extending the DebugLogger output > > > <URL:http://www.zope.org/Members/tseaver/Projects/HighlyAvailableZope/DebugL…> > > > and posting the results of any hanging there? > > > > > RE: could be zombie related - > > > > > > Where might I find more info on this? Could this zombie issue be > > > present in FCGI as well? > > > > Amos Lattier remarked in > > http://lists.zope.org/pipermail/zope-dev/2000-April/004194.html that: > > > > "The ZServer zombie stuff is to get rid of zombie client > > connections, not zombie publishing threads. These are quite > > different beasts." > > > > I'm not yet grokking the whole picture, so I can't really answer to that. > > Note that there is an outstanding issue in the Collector at > > http://classic.zope.org:8080/Collector/954/view that might be related. As > > you previously noted, there is no zombie_timeout in the FCGI server. > > What the Zombie timeout means is that after a publishing thread gets > done answering a request, the socket may not go away. This many for a a > number of reasons, the client 'hung' and is not 'putting down the phone > after the converstation is over' (so to speak) or network troubles may > prevent the connection from closing properly. This means that there is > a 'zombie' connection laying around. This zombie will probably end up > going away on its own, but if not, ZServer will kill it after a period > of time. > > The only reasorce laying around during the life of a Zombie is an tiny > little unused open socket, the Mack truck of a Zope thread that served > the request for the zombie socket does not 'hang' for that entire period > of time, but goes on after it has completed the request to serve other > requests. > > Amos is correct in that these problems are almost always at the > Application level, and not at the ZServer level. The fact that Pavlos > can prevent hanging by inserting a print statement in the asyncore loop > is suspicious, but we do not have enough information yet to point > fingers anywhere. Here are a couple of commonalities we need to look at: * IE 5 seems to be involved. * The management interface seems to trigger the problem (no application-specific code involved, just manipulation of "stock" Zope objects). * Framesets may be involved. * Perhaps premature request cancellation (likely to happen often in the management interface) is involved. I would like to see a description of the interaction between the ZServer-async-select() thread and the Zope threads to which it dispatches requests, particularly what happens when a Zope thread tries to write to a connection which has been closed by the client without a "clean" shutdown (e.g., attempting to write to the socket with fail with errno == EPIPE). I have a couple of testcases for this problem. (Un)fortunately, I don't use IE5 (and don't go with girls who do! :), and therefore they all work without a hitch. First, an ExternalMethod, implemented so:: from time import sleep def stall( secs = 30 ): """ Delay to allow testing. """ sleep( secs ) return "Slept %d seconds" % secs Called trivially from DTML method called 'test_stall':: <html> <head> <title><dtml-var document_title></title> </head> <body> <h2><dtml-var document_title></h2> <dtml-unless seconds> <dtml-call "REQUEST.set( 'seconds', 60 )"> </dtml-unless> <p><dtml-var "stall( seconds )"></p> </body></html> Exercised repeatedly through a frameset:: <html> <head> <title> Test Stalling </title> </head> <frameset rows="33%, 33%, 34%"> <frameset cols="33%, 33%, 34%"> <frame name="r1c1" src="test_stall?seconds:int=30"> <frame name="r1c2" src="test_stall?seconds:int=30"> <frame name="r1c3" src="test_stall?seconds:int=30"> </frameset> <frameset cols="33%, 33%, 34%"> <frame name="r2c1" src="test_stall?seconds:int=30"> <frame name="r2c2" src="test_stall?seconds:int=30"> <frame name="r2c3" src="test_stall?seconds:int=30"> </frameset> <frameset cols="33%, 33%, 34%"> <frame name="r3c1" src="test_stall?seconds:int=30"> <frame name="r3c2" src="test_stall?seconds:int=30"> <frame name="r3c3" src="test_stall?seconds:int=30"> </frameset> </frameset> </html> I can run this repeatedly, stopping it in mid-run, etc., all without causing any problems that I can diagnose. Anyone care to try it on their setup with IE5 as the client? Tres. -- ========================================================= Tres Seaver tseaver(a)digicool.com tseaver(a)palladion.com

1 0