- Zope-Dev - Zope lists

Some thoughts on splitter
by Sin Hang Kin 15 Apr '00

15 Apr '00

Yes! The pre-process approach is heading the right way. However, I would like to suggest not inserting spaces, but some code that does not alter the display. It seems that the unicode have set aside the code &zwnj; which is called zero width non-joiner. The code can then stored with the text and would be there for the editing and go through later processing. The index every char approach is not perferred due to this can be emulated by the previous method if needed. Also, the catalogue must use unicode for cross-encoding search. It is well known that Han have many encoding in big-5, gb2312, jis, etc. It is a good practice to convert all code to unicode, normalized it, then perform the splitting. The convertion must based on language and encoding, however, most html do not declare its language and encoding. I have seen some encoding detection code based on checking the freq used han characters which give a good guess of the encoding. Rgs, Kent Sin

3 4

RE: [Zope-dev] Extensions directory
by Chris McDonough 15 Apr '00

15 Apr '00

Well, if you make a directory in the Products folder and put a blank file in there named __init__.py, it becomes a Product. It does nothing whatsoever, but it will appear in the Products folder of the Control Panel. Then you can put an Extensions directory in that folder and use it to store external methods. You can create a number of do-nothing products that will allow you to categorize your external methods. This seems to be what you're trying to do. > -----Original Message----- > From: jpenny(a)universal-fasteners.com > [mailto:jpenny@universal-fasteners.com] > Sent: Friday, April 14, 2000 4:04 PM > To: Chris McDonough > Subject: Re: [Zope-dev] Extensions directory > > > On Fri, Apr 14, 2000 at 04:14:55PM -0400, Chris McDonough wrote: > > In CVS at least (and maybe in stable revs?), Jim F. provided the > > capability to store external methods within a Product directory's > > 'Extensions' directory, referenced by an external method in a module > > with the name "ProductName.ModuleName". So if you've got > an extension > > directory that has lots and lots of external methods from > lots and lots > > of Products, you can break them up and store them in their > respective > > product Extension dirs. > > Thanks, (I think). I am not real sure that I want to build Products, > though. What I am doing just does not feel reusable enough to justify > the effort. >

1 0

Software Carpentry entries now on-line
by gvwilson＠nevex.com 15 Apr '00

15 Apr '00

First-round entries in the Software Carpentry design competition are now available on-line at: http://www.software-carpentry.com/entries/index.html Our thanks to everyone who entered; we look forward to some lively discussion on the "sc-discuss" list. Best regards, Greg Wilson Software Carpentry Project Coordinator

1 0

>2GB Data.fs files on FreeBSD
by R. David Murray 14 Apr '00

14 Apr '00

OK, I've seen it bandied about that Linux doesn't support large files but FreeBSD does, and to run Zope on FreeBSD if you want a large database file. So how do you do that? On my BSDI box I started getting "File too large" errors when the file exceeded some magic number around 2GB. I moved the file to a FreeBSD box hoping to be able to trim it there, but I get an error when I try to start Zope: Traceback (innermost last): File "/usr/local/zope/Zope-2.1.2-src/z2.py", line 436, in ? exec "import "+MODULE in {} File "<string>", line 1, in ? File "/usr/local/zope/Zope-2.1.2-src/lib/python/Zope/__init__.py", line 109, in ? DB=ZODB.FileStorage.FileStorage(Globals.BobobaseName) File "/usr/local/zope/Zope-2.1.2-src/lib/python/ZODB/FileStorage.py", line 301, in __init__ self._pos, self._oid, tid = read_index( File "/usr/local/zope/Zope-2.1.2-src/lib/python/ZODB/FileStorage.py", line 1231, in read_index if file_size < start: raise FileStorageFormatError, file.name ZODB.FileStorage.FileStorageFormatError: /usr/local/zope/Zope-2.1.2-src/var/Data.fs So it looks like there is a problem using Zope with a large database no matter what the platform. Has anyone figured out how to fix this? My main goal is to be able to get the last transactions out of this database, after which I'll be able to shrink it well under the 2BG limit...I'm going to try just truncating the last few bytes to see if I can get Zope to open it and let me recover most of the transactions. (Side note: tranalyzer dies before it processes the whole file, probably some sort of memory resource limit on my server.) But given the number of people who have said "use FreeBSD if you want big files", I'm really wondering about this. What if later I have an application where I really need a >2GB database? --RDM

4 8

icon_for_meta_type on Zope.org
by Jason Spisak 14 Apr '00

14 Apr '00

Zopists, Is the icon_for_meta_type used to return serach results and their meta_type icon an external method? I am trying to get the icon for a given meta_type trough mapping. Here's the code. The code pasted in from Zoe.org gives me a name error on icon_for_meta_type: <dtml-with "_(meta_types=filtered_meta_types(AUTHENTICATED_USER))"> <dtml-in meta_types mapping> <a href="<dtml-var URL>?<dtml-var QUERY_STRING>+selected_meta_type=<dtml-var name>"> <dtml-var "icon_for_meta_type(REQUEST,meta_type=name)"></a> </dtml-in> </dtml-with> Anyone know how to get the icon back for the meta_type in this situation? All my best, -- Jason Spisak 444(a)hiretechs.com

1 0

selected_meta_types in SubOjbects tab
by Jason Spisak 14 Apr '00

14 Apr '00

Zopistas, Is there any reason why I would be able to access from a ZClass instance the management options like the management tabs: <dtml-with "_(manage_options=filtered_manage_options())"> <dtml-in manage_options mapping> blah, blah, blah... But then not be able to access from a ZClass instance the subobject tab's "selected_meta_types()" in the same way: <dtml-with "_(selected=selected_meta_types())"> <dtml-in selected mapping> It fails with this error: Traceback (innermost last): File /home/jason/Zope-2.1.6/lib/python/ZPublisher/Publish.py, line 214, in publish_module File /home/jason/Zope-2.1.6/lib/python/ZPublisher/Publish.py, line 179, in publish File /home/jason/Zope-2.1.6/lib/python/Zope/__init__.py, line 202, in zpublisher_exception_hook (Object: CatalogAware) File /home/jason/Zope-2.1.6/lib/python/ZPublisher/Publish.py, line 165, in publish File /home/jason/Zope-2.1.6/lib/python/ZPublisher/mapply.py, line 160, in mapply (Object: CommissionEmployeeClass_addForm) File /home/jason/Zope-2.1.6/lib/python/ZPublisher/Publish.py, line 102, in call_object (Object: CommissionEmployeeClass_addForm) File /home/jason/Zope-2.1.6/lib/python/OFS/DTMLMethod.py, line 150, in __call__ (Object: CommissionEmployeeClass_addForm) File /home/jason/Zope-2.1.6/lib/python/DocumentTemplate/DT_String.py, line 502, in __call__ (Object: CommissionEmployeeClass_addForm) File /home/jason/Zope-2.1.6/lib/python/OFS/DTMLMethod.py, line 146, in __call__ (Object: ui_start) File /home/jason/Zope-2.1.6/lib/python/DocumentTemplate/DT_String.py, line 502, in __call__ (Object: ui_start) File /home/jason/Zope-2.1.6/lib/python/OFS/DTMLMethod.py, line 146, in __call__ (Object: drawers) File /home/jason/Zope-2.1.6/lib/python/DocumentTemplate/DT_String.py, line 502, in __call__ (Object: drawers) File /home/jason/Zope-2.1.6/lib/python/DocumentTemplate/DT_With.py, line 133, in render (Object: _(selected=selected_meta_types())) File /home/jason/Zope-2.1.6/lib/python/DocumentTemplate/DT_Util.py, line 335, in eval (Object: _(selected=selected_meta_types())) (Info: selected_meta_types) File <string>, line 0, in ? NameError: (see above) Does anyone know of a way to push that into the namespace? All my best, -- Jason Spisak 444(a)hiretechs.com

1 0

RE: [Zope-dev] Extensions directory
by Chris McDonough 14 Apr '00

14 Apr '00

In CVS at least (and maybe in stable revs?), Jim F. provided the capability to store external methods within a Product directory's 'Extensions' directory, referenced by an external method in a module with the name "ProductName.ModuleName". So if you've got an extension directory that has lots and lots of external methods from lots and lots of Products, you can break them up and store them in their respective product Extension dirs. > -----Original Message----- > From: jpenny(a)universal-fasteners.com > [mailto:jpenny@universal-fasteners.com] > Sent: Friday, April 14, 2000 3:06 PM > To: zope-dev(a)zope.org > Subject: [Zope-dev] Extensions directory > > > Background: > I find that I am writing quite a bit of External python code that is > quite task specific; probably far too task specific to try to > package. For example, I have an (internal only) page that shows > part consumption on a per machine basis. There are hundreds > of machines, > and thousands of parts, so the graphs must be produced dynamically. > > This is pretty easy to do. (I use ploticus, a great open source > tool, http://www.sgpr.org/.) But I can see, down the road, being > requested to produce many different sorts of graphs. And since > the data stream will have differing types and number of arguments, > it is not clear that it would be easy to write a Product, even if > I had the inclination. > > But, I hate to have many scripts in a single flat directory. > This can be > both a mess to maintain and can slow things down on some file-systems. > > So, is there any strong reasons that the Extensions directory needs > to be flat (i.e. not have subdirectories)? > > Jim > > _______________________________________________ > Zope-Dev maillist - Zope-Dev(a)zope.org > http://lists.zope.org/mailman/listinfo/zope-dev > ** No cross posts or HTML encoding! ** > (Related lists - > http://lists.zope.org/mailman/listinfo/zope-announce > http://lists.zope.org/mailman/listinfo/zope ) >

1 0

Extensions directory
by jpenny＠universal-fasteners.com 14 Apr '00

14 Apr '00

Background: I find that I am writing quite a bit of External python code that is quite task specific; probably far too task specific to try to package. For example, I have an (internal only) page that shows part consumption on a per machine basis. There are hundreds of machines, and thousands of parts, so the graphs must be produced dynamically. This is pretty easy to do. (I use ploticus, a great open source tool, http://www.sgpr.org/.) But I can see, down the road, being requested to produce many different sorts of graphs. And since the data stream will have differing types and number of arguments, it is not clear that it would be easy to write a Product, even if I had the inclination. But, I hate to have many scripts in a single flat directory. This can be both a mess to maintain and can slow things down on some file-systems. So, is there any strong reasons that the Extensions directory needs to be flat (i.e. not have subdirectories)? Jim

1 0

RE: [Zope] Re: Zope hanging (poss. threads-related)
by Marcus Collins 14 Apr '00

14 Apr '00

> -----Original Message----- > From: Amos Latteier [mailto:Amos@digicool.com] > Sent: 13 April 2000 20:53 > To: 'Tres Seaver'; Marcus Collins > Cc: zope-dev(a)zope.org > Subject: RE: [Zope] Re: Zope hanging (poss. threads-related) <snip> > The ZServer zombie stuff is to get rid of zombie client > connections, not zombie publishing threads. These are quite > different beasts. Thanks for the clarification. I'm still on my way to understanding how it all fits together... > > > Everything before the call to handle() is 100%. Sometimes, > > > however, we don't get from handle() to the next stage. This is > > > on Zope 2.1.6, which I've been running with up to 100 threads, > > > although I unfortunately can't excercise that many! > In general there is little reason to have so many publishing > threads. > You almost never need that many unless you have a bunch of > requests that can take a *long* time. > > If, on the other hand, you are trying to provoke some kind of > thread contention issue I advise you to publish resources that take > a long time to return. That way you can easily pile up as many > publishing theads as you want to. Normally, I'd estimate that 20 threads would be sufficient for our site, and that's what we want to run. We have some search queries that may take a number of seconds to complete. However, in this instance, I was deliberately attempting to hang threads. The pages of our site on which I've noted hanging occurring take much less than a second to publish; no hanging problems have been noted on pages which take significantly longer (say, 10 or 20 seconds in the extreme) to publish, although possibly that's a statistical probability. > > > I've added this logging to the Zope 2.1.3 serving the live > > > site, and will report my findings as soon as something untoward > > > occurs. Maybe others who are experiencing hanging would also be > > > able to do some extra logging and report the results [now, > > > there, I see Wiki would be really useful!]. I've described at http://www.zope.org/Members/tseaver/Projects/HighlyAvailableZope/DiagnosingH angProblems/DebugLogger how I've set up DebugLogger to give a greater granularity to the logging of requests. Perhaps, if other users also discover threads hanging, they could post their results there? Since extending the logging, I have found that the hanging occurs less frequently; when I reduced the logging again, the hanging recurred -- specifically, when I reduced the logging in send_response() in PCGIServer.py. The logs then indicated that the request failed sometime between found_terminator() ending and the return from HTTPRequest() in send_response(). Note that this is _different_ to my previous posting, where I indicated that the request failed before handle() returned. Both of the reports are true, so it seems we haven't managed to nail it down quite as well as I thought... The hangings that I've now recorded in testing occurred with Zope 2.1.6 when NUMBER_OF_THREADS was set at 10. It's also hung with 4 and with 20 threads. To be fair, though, since adding additional logging, I've really had to pound it (20-30 simultaneous users, abnormally heavy usage) to get it to hang. The hangings on the live site (Zope 2.1.3) have occurred under light load (one or two users), first with 20 threads, and now with 4. > In my experience when a Zope publishing thread hangs its > almost always a problem with the published resource. Maybe there's > something that puts Zope in a loop that never exits, or maybe > there's some DA weirdness that hangs the thead. > My advice is to try and identify which requests hang using > debug logging and examine the resources that those requests use. The specific resources on which this hanging has occurred are: - /manage_*, but not yet when using manage_main alone (again, perhaps a statistical probability) - a frameset page on our site, which uses: * ZSQL methods calling a MySQL database (MySQL 3.22.27/32; ZMySQLDA 1.1.3 patched with MySQLdb 0.1.0; db connection pool_size equal to NUMBER_OF_THREADS) * GUF 1.2.2 (using the database) * SiteAccess 1.0.1. A single thread at a time hangs. Recently, however, when /manage was accessed shortly after a thread hung, ZServer became completely unresponsive to http or pcgi (didn't check ftp...). I'll be putting this and further details up at http://www.zope.org/Members/tseaver/Projects/HighlyAvailableZope/MarcusColli ns as soon as I get a chance. Unfortunately, that's not likely to be before Monday. Thanks All. -- Marcus

1 0

RE: [Zope] Re: Zope hanging (poss. threads-related)
by Marcus Collins 14 Apr '00

14 Apr '00

[following suit and moving this over to zope-dev.org] > -----Original Message----- > From: Tres Seaver [mailto:tseaver@palladion.com] > Sent: 13 April 2000 19:55 > To: Marcus Collins; zope-dev(a)zope.org > Subject: Re: [Zope] Re: Zope hanging (poss. threads-related) > Hmm, _I_ was the one making that argument. I guess I wasn't > really urging everyone to quit posting to the list -- rather, I > hoped that the community could "harvest" the information from the > list and massage it into a useful format for problem solving (e.g., > to make it possible to notice connections bewteen pieces > of information which appear on the list at fairly disparate times). > Most of the pages I put up there to start with wree attempts to > categorize types of problems, and also potential sources of them. OK, I'm afraid I misunderstood you at first. I've seen the fault of my ways, repented, and quite agree with you and with Michel, that the Wiki is in fact quite suited to this. I'll be putting up bits and pieces there as soon as I can get a chance (and, I confess, figure out Wiki :-). > Perhaps is was a caching issue? I clicked the link, saw that > I was not logged in, logged in, returned to the page, and had to > force a reload to see the "Edit this page" links. This was indeed the problem; certainly operator failure not to realise it... Thanks. Now back to documenting the problem! -- Marcus

2 1