March 2000 - Zope-Dev

ZCataloging imported data
by Jason Spisak 06 Mar '00

06 Mar '00

R. David Murray: I'm not suer if you got a reply on this one yet, but I have experience here so I'll hope it helps. Michael P. is the real ZCatalog guru and this information is a sum of what he imparted to me. <snip> he first problem I ran into was during the import. I wrote an external method that read the data from a tab delimited file and used the data to build ZClass instances. I tried to create them all in one folder in one transaction. This failed miserably at somewhere around 1500 records. I got an error about an 'frexp' call being out of range, somewhere in ZCatalog's BTree methods </snip> I loaded 35,000 records from a tab delimited file using an external method. I never got an 'frexp' error so I can't help with that. I did create them all in one folder though and was told that there is effectively no hard limit to the amount of items that can be contained in a folder. <snip> So I tried loading the records in batches, and at first that seemed to work. Then I got an out of memory error. </snip> I tried batches too. But only after I got an out of memory, and then a disk full error. I was told to take the subtransaction threshold in the indexnig tab in the Catalog and push it up to 1000000. That way a subtransaction would only committ to disk after quite a few records had gone in. I was indexing some text that was 1000 words long and it was committing on every single one. You have to figure out what you are loading and make the transactions commit after a reasonable number of records so that you don't exceed your resources. <snip> Now we find that when we enter certain keywords (the examples we have found so far are 'well' and 'fire', which you probably don't need to know) on one of these by-hand ZClass instences, they are *not* found by a ZCatalog search on the keywords field index. Other words entered into the keywords field do cause the record to be found ('wells', for example). </snip> Try finding C/C++ ;-) 'bill' doesn't show up either. There are certain words that are hard coded to be skipped in the indexing program. You can change these because this is Open Source and you get it all. I think it's a C program but I dno't nkow which one. Michael what was that one again? This is not to be confused with the 'stop words' in the Catalog.py. Those can be chanegd too, but at the risk of bloating you indexes. Also '+' and '-' (dashes and minuses) get skipped. To get speed and small manageable size some sacrifices have to be made, but you can always work around them because this is Open Source. I also am using a 'keyword' index for C/C++ and items that I know are important to the usefulness of the Catalog. I hope this helped. All my best, -- Jason Spisak 444(a)hiretechs.com

1 0

Editing PropertySheet Definitions
by James W. Howe 06 Mar '00

06 Mar '00

Is there a way (easy or hard) to edit the contents of a property sheet? What I want to be able to do are things like rename a property, change the meta type, etc. At a minimum I just want to be able to change the name of a property but I don't like the idea of deleting the property and adding it back. If I do that, it appears at the end of the list and I then have to create my own custom form to display the properties in a good sequence. Speaking of which, it would be nice if there were a way to reorder properties so you could control the sequence in a propertysheet without handcoding a custom form. Thanks James W. Howe mailto:jwh@allencreek.com Allen Creek Software, Inc. pgpkey: http://ic.net/~jwh/pgpkey.html Ann Arbor, MI 48103

1 0

Re: [Zope-dev] Using Zope in a client-server system
by Itai Tavor 06 Mar '00

06 Mar '00

Tres Seaver wrote: >> Michel Pelletier wrote: >> >> > Itai Tavor wrote: >> <snip> >> > >> >> There are a couple of features that I am not sure how to >>implement though: >> >> >> >> - I need to display constantly-updating information on the >> >> workstations. I could do this by reloading the window every second, >> >> but that seems ugly - I'd like to only update certain fields, which I >> >> can do using layers and JavaScript, but I need to get the updated >> >> data from the server. I guess that can be done with Java, but I'd >> >> like to keep the system simple - a browser with HTML and JavaScript >> >> only on the client side, DTML, Zope with Python Methods, and External >> >> Methods, and an SQL database on the server, no Java or any other >> >> complications. >> > >> > Ah, well, you see you've hit one of the disadvantages of using a >> > stateless protocol like HTTP to solve your problem. There is no good >> > general purpose solution that I know of to address this issue. >> >> HTTP not being bi-directional is a bigger problem than it being >> stateless... I planned to use a session product that will identify >> each workstation and keep a record of exactly what is happening on >> it. But the biggest problem is that the browser / web server setup is >> client event driven. >> >> >> - I also need to push data to the workstations - based on events >> >> detected at the server, I need to bring up a message on a >> >> workstation, and remove it in response to another event. I really >> >> don't want to refresh a frame every second for this, and I also would >> >> hate to have to write a Java applet to do it. >> > >> > This is the identical problem to the one above. >> > >> >> Any comments or suggestions regarding the above problems, the choice >> >> of Zope for this project, or the project in general would be >> >> appreciated. >> > >> > If you want to go far enough, you can try and use XML-RPC (also, >> > stateless, but not client-bound to a certain model) and something like >> > wxPython to build your own GUI that can at least mantain some sort of >> > persistent or bi-directional communication between your clients and >> > servers. But this is a big task. >> >> Programming my own GUI is already farther than I'd like to go... and >> if I do, would it still be a good idea to use Zope on the server >> side? If I use Zope, wouldn't I still have to use HTTP? How otherwise >> can I get the client to communicate with the server? (Complete >> ignorance of XML-RPC implied here.) >> >> > I think if you maybe rethink your problem you can compromise on >> > something that does not require the server pushing data to the client. >> >> Yeah, without actually failing to deliver any of the features the >> client is expecting... the only way I can think of is simply to >> reload certain frames every second... seems like a very ugly solution. >> >> I need to quote on time and cost for this project, and to do that I >> have to figure out the tools I'd be using. I'm sure a Zope/web >> browser solution will be the quickest, but if I go that way and get >> stuck, I won't be in a very enviable position. I'm sure that most >> programmers would immediately start coding Java or VB at this >> point... am I making a mistake trying to apply Zope to a task it's >> not good for? > > By default, browsers aren't servers: they don't know from > bind()/listen()/accept(), which is what push technology requires (unless you > have a long-running, bi-directional connection between browser and >server, which > HTTP doesn't allow). Polling is the only "native" technique for refresh, and > suffers from _horrible_ scalability problems. > > Options in order of simplicity: > > 1. Pull instead of push (ick!) My sentiments exactly. > 2. As Hung Jung suggests, build the "push model" stuff separately from the > browser, using Python or Java to implement HTTP (and perhaps XML-RPC?) > Integration with the browser is difficult: clients will have to have > some mechanism for registering their mini-server with your server; > if you want the broser's display to update when the push event comes in, > you may be stuck. Right. Mixing two GUI's - a browser for most of the interface and a separate app for pushed data - would be real ugly. So I either have to work inside the browser, as in option 3 below, or code my own GUI. Registering the mini-servers would not be a problem. This project will be running on a small number of workstations, all located in one location. It's not designed for public or wide-area use and won't even be connected to the internet. So, the problem with this solution is how to get the pushed data into the GUI. > 3. Code a lightweight socket server applet in Java, persuade everyone in > the world to trust you to run it in their "applet sandbox", and then > "register" it with your server. Again, because of the physical layout of the project, trust won't be an issue, which makes this solution rather attractive. > 4. Write a CORBA callback object in Java, and register it with your server; > the server then pushes events to your object (scalability is again an > issue, as the server blocks for each client being pushed). Note that > adding CORBA into Zope is a non-trivial exercise, for the >moment at least; > see my "notes":http://www.zope.org/Members/tseaver/CosNaming on the > issues. I read your notes, and I think I'd like to stay away from COBRA - I'd use it if I decided to go totally overboard and code both the server and client in Java. But with Zope on the server end, XML-RPC seems like the best way to communicate. > 5. Write a CORBA event-channel applet (CosEventPushConsumer) in Java, set > up a CosEvent/CosNotification channel, subscribe the applet to it, and > push events to the channel from your server (decouples your server from > event-delivery hassles/blockage, scales MUCH better than direct >callbacks). Scalability is not much of a problem for me. I wouldn't like the server to block too much but that shouldn't be much of a problem either - with 10 clients on a 100baseT network, and 4 Zope threads on a fast server, I can't imagine running into any performance problems. But I do want to keep the number of technologies and separate apps involved to a minimum, and this seems fairly complicated. Out of the various options suggested it seems to me that the main options to consider are: - Use a browser for the main GUI, and either compromise the requirements to eliminate the need for push or code an applet to display push data in the browser. - Write my own GUI. The second option raises some new questions: - Would it still be a good idea to use Zope for the server, or should I code the server as well (a full Java solution comes to mind using COBRA for two-way communication). How would I handle push in Zope? Assuming I code a push client that speaks XML-RPC, how do I get Zope to send data outside the context of an http transaction? Actually, this same question is relevant for the solution of using a Java applet within the browser. - Two good candidates for writing the GUI in are Java and wxPython. Using wxPython would have the advantage of keeping everything in Python. On the other hand, there's an XML-RPC server for Java, but not for Python - I guess I would use HTTP if I use wxPython. Can anyone suggest pros/cons for these solutions? I have one other question related to the use of Zope in this project: The server has to monitor a serial line and respond to received data by pushing messages to the clients. What would be the best way to achieve this? An external python process reading the serial line and calling ZPublisher.Client to triger a DTML method that will do the rest of the work? P.S. I know this is getting way off topic for the Zope group. I really appreciate all of you contributing your experience. Itai -- Itai Tavor -- "Je sautille, donc je suis." -- itavor(a)vic.bigpond.net.au -- - Kermit the Frog -- -- "What he needs now is understanding... and a confederate victory" -- -- Dr. Jacobi, Twin Peaks --

6 5

RESPONSE.write
by Toby Dickenson 06 Mar '00

06 Mar '00

I've tracked down the problems in using RESPONSE.write with ZServer. The symptoms of this bug are an incorrect Content-Length: 0 header, corrupt output, and no support for HTTP/1.1 clients. The replacement files, relative to 2.1.4, are at http://www.zope.org/Members/htrd/RESPONSE.write/howto (also reported in the collector) Toby Dickenson tdickenson(a)geminidataloggers.com

1 0

Access to bug collector
by Duncan Booth 06 Mar '00

06 Mar '00

I noticed recently that the scale tag for images doesn't work. Checking the collector reveals that someone else reported this bug, and posted a patch back in December last year, however the item is still marked as under review. I understand there are only a limited number of hours each day, so it may take a while for patches to make their way into a released product, but would it not be possible for someone to review items a bit more often and at least make them visible even if the patch hasn't yet been applied. In this particular instance I guess the patch is simply to change two '==' into '!='. -- Duncan Booth duncan(a)dales.rmplc.co.uk int month(char *p){return(124864/((p[0]+p[1]-p[2]&0x1f)+1)%12)["\5\x8\3" "\6\7\xb\1\x9\xa\2\0\4"];} // Who said my code was obscure? http://dales.rmplc.co.uk/Duncan

1 0

Re: [Zope-dev] Re - [Zope-dev] ZClass ID bug
by Rik Hoekstra 05 Mar '00

05 Mar '00

You may want to take a look at this doc and see if it helps: <http://www.zope.org/Members/AlexR/ZClassIDBug> [RH] Yep, that seems to the same problem. I must have missed it before. Thanks Rik

1 0

Re - [Zope-dev] ZClass ID bug
by Alexandre Ratti 05 Mar '00

05 Mar '00

Hi Rik, You may want to take a look at this doc and see if it helps: <http://www.zope.org/Members/AlexR/ZClassIDBug> Alexandre At 12:00 02/03/2000 -0800, you wrote: >From: "Rik Hoekstra" <rik.hoekstra(a)inghist.nl> >To: <zope-dev(a)zope.org> >Subject: [Zope-dev] ZClass ID bug >Date: Thu, 2 Mar 2000 16:45:03 +0100 > >I have a ZClass based on CatalogAware (no problem) and on >DTMLDocument/DTMLMethod (it does not matter which). A number of instances >are added in a batch. > >All this works well, but the id representation of the resulting instances >are not uniform and sometimes they're just wrong. > >- In a normal manage_main view of their Container, the ids will show without >problem. >- 'objectIds' (and 'objectItems') also returns the right values, but >objectValues returns a <string> (i.e. an object) instead. This cannot be >brought to show its value (at least not by me) >- All the same absolute_url also returns <http://bla/bla/<string>. This >makes ZClasses subclassed from DTMLDocument/DTMLMethod unusable, at least >for my purposes. >I solved the problem by making a ZClasses based on objectManager, which is >not ideal, but workable for me. >Has anyone else seen this problem? >Is there a solution/patch around. i have no idea how to debug this? >I will post this to the Collector as well, but I thought this would be of an >interest to the list as well. >Rik

1 0

Pointer or reference object
by Jason Spisak 05 Mar '00

05 Mar '00

Zopsters, This is very out of the blue, but has anyone thought about tackling the 'reference' question by using a phantom Zope object? I don't know nearly enough Python to answer this question, but maybe someone can. Is it possible to create a 'Reference' object that really stores the url of another object and maquerades as the other object's meta_type. Anything that is asked of the 'Reference' object, just get's passed along to the real object in it's place? Obviously whenever you delete an object any 'References' with that object's url stored in them get deleted too(ZCatalog could hold these). Sort of like a reflective mirror. You see the object in the mirror but you aren't actually looking at the object, you are looking at the mirror. A ZMirror. All my best, -- Jason Spisak 444(a)hiretechs.com

2 1

Re: [Zope-dev] Using Zope in a client-server system
by Sin Hang Kin 05 Mar '00

05 Mar '00

Making zope a proxy server on the client's machine is very good. I always dreamed to have zope installed on all my computers as a cache proxy and it can support pre-fetch, ad-filter, search, and merging data between different machines. But this take big changes.

2 1

[Zope] Logging users in Zope 2.x (second try)
by Hannu Krosing 04 Mar '00

04 Mar '00

How should I configure logging to get usernames into Z2.log I first hoped to get it from Apache's access_log, but it saves username only when doing user authentication itself. Then I thought that Zope at least would log it and migrated the site to Zope 2.1.3 to get logging support. Unfortunately it does still not log usernames and I have not found the place where I could turn it on. Also the format is buggy: Apache access_log: 193.40.215.135 - - [11/Feb/2000:08:29:55 +0200] "GET / HTTP/1.1" 200 2673 Zope Z2.log 127.0.0.1- - [11/Feb/2000:19:04:46 +0300] "GET /misc_/OFSP/dtmlmethod.gif" 200 1000 the bug being the missing space after the IP address. what I was hoping for would have looked like: 127.0.0.1 - superuser [11/Feb/2000:19:04:46 +0300] "GET /manage" 200 1000 ------- Hannu

2 2