- Zope - Zope lists

Re: [Zope] Generic SQL insert
by Michael Shulman 15 Apr '06

15 Apr '06

"Robert (Jamie) Munro" <jamie(a)textmatters.com> wrote: > I'd much rather have a dictionary of fields and values, and just throw > it at the DB, not having to make those queries for every table. I have > acheived it like so: I have achieved a similar result, although for update methods in my case, in what I think is a slightly better way (although probably people on this list will tell me I'm wrong). Rather than having a single generic ZSQL method, I have one ZSQL method for each table, but intsead of writing them all by hand I have a single script which generates/regenerates them from inspection of the columns in the database, like so: # select some arbitrary record from the table into table_data, then: fields = table_data.data_dictionary() arguments = "" comma = "" print "update %s set" % table types = {'s':"nb", 'd':"nb", 'i':"int", 'n':"nb"} for field in fields.keys(): arguments = arguments + field + " " type = types[fields[field]["type"]] print " %s%s = <dtml-if expr=\"%s is None\">null<dtml-else><dtml-sqlvar %s type=\"%s\" optional></dtml-if>" % (comma, field, field, field, type) comma = "," print " where %s = <dtml-sqlvar %s type=\"int\">" % (id_field, id_field) # update_zsql is the ZSQL "update" method I am creating/updating update_zsql.manage_edit('','database_name',arguments,printed) This way, all the columns are assigned the correct type, but the calling script doesn't have to handle types or casts, since all the types are assigned by inspecting the actual database. Also, this way the update/insert will fail if any fields are missing. Actually, given how object-oriented Zope is in general, I was surprised to find that its database interface (ZSQL methods) still requires the user to write SQL code essentially by hand, rather than using an object-relational mapper like SQLObject. There appears to be a toolkit called sqlos for using SQLObject with Zope 3, but I haven't yet tried it out. > > mydict = {"field1":"value1" , "field2":"value2" ,...} > (fields,values)=zip(*myDict.items()) > context.genericInsert(table='table name',fields=fields,values=values) > > Where generic insert is the following ZSQL method: > insert into "<dtml-var table>" > (<dtml-in expr="fields">"<dtml-var sequence-item>"<dtml-if > sequence-end><dtml-else>,</dtml-if></dtml-in>) > values (<dtml-in expr="values"><dtml-sqlvar sequence-item > type=string><dtml-if sequence-end><dtml-else>,</dtml-if></dtml-in>); > > with parameters: > * table - table name > * fields - list of fieldnames > * values - list of values in the same order > > What do other people think of this? Is it a really bad idea? > > Robert Munro

1 0

zope 2.8.6 on Mac Intel
by manuel spuhler 15 Apr '06

15 Apr '06

Hi, does anyone know how-to compile Zope 2.8.6 on a OS X system with an Intel processor?

6 20

Re: zope 2.8.6 on Mac Intel: Resolved
by manuel spuhler 14 Apr '06

14 Apr '06

Hello David, sorry, I meant you before, but hey, hello too Dieter ;-) I finally manage to install and run Zope 2.8.6 on Mac Intel after installing the Universal package of python 2.4.3 for os x (http://www.python.org/download/releases/2.4.3/) Then I did the usual unix stuff: $./configure --with-python=/usr/local/bin/python --prefix=/opt/Zope2.8 && make &&make install $ /opt/Zope2.8/bin/mkzopeinstance.py Even if this Python version is not supported, I didn't notice (yet;-) any problem. As far as I know, it's the 1st Zope 2.8.6 and Plone 2.11 on Mac Intel and it's at least as fast as on Linux !!! Thank you all for your help and happy zoping! Manuel Zope Version (Zope 2.8.6-final, python 2.4.3, darwin) Python Version 2.4.3 (#1, Mar 30 2006, 11:02:16) [GCC 4.0.1 (Apple Computer, Inc. build 5250)] System Platform darwinope Version (Zope 2.8.6-final, python 2.4.3, darwin) Python Version 2.4.3 (#1, Mar 30 2006, 11:02:16) [GCC 4.0.1 (Apple Computer, Inc. build 5250)] System Platform darwin

1 0

How to update list from dtml?
by Jaroslav Lukesh 14 Apr '06

14 Apr '06

Dear Zopistas, How to do in DTML something similar in python: foo[1] = 3.5 In DTML this does not work: <dtml-call "REQUEST.set('foo[1]', 3.5)"> Note, that I have list foo[,,,,,,,] already defined Many thanks, J. Lukesh

4 4

Getting www.domain.com:8080/manage to go through Apache
by michael nt milne 14 Apr '06

14 Apr '06

Hi I've got various virtual hosts running but am having a few issues getting the administration site on www.domain.com:8080/manage to go through Apache. I want to route this all through SSL. What I can get is the admin for the *individual site* but not all the sites listed under a Zope instance. I'm trying to use <location></location> to put a request for www.domain.com/manage through to www.domain.com:8080/manage in Zope and then back out through port 443. I haven't set up the SSL certificates yet. Here's what I've got so far which isn't working quite as planned. Anyone achieved this at all? Thanks NameVirtualHost xx.x.xxx.xxx:80 <VirtualHost xx.x.xxx.xxx:80> ServerName www.domain.net <location /manage> RewriteEngine On RewriteRule ^/(.*) http://www.domain.co.uk:8080/manage/VirtualHostBase/https/www.domain.net:44… </location> </VirtualHost> -- michael

1 0

Memory Leak in Session Data Container
by Andy Yates 14 Apr '06

14 Apr '06

Good day everyone! I hope your weekend was as good as mine. I didn't have to work and I got to watch my son's first soccer (football) game! For the past couple of years (I think) the Session Data Container has been leaking memory. During this time several leaks have been patch but at least for our servers these fixes have only slowed down the process. As our use of Zope grows and traffic to our site increases so does the memory usage of Zope. Unless Zope is restarted it will eventually use all physical memory and swap space and crash the server. Over the past month the problem has once again grown to the point where we must restarted Zope daily, sometimes even every 8 hours. I don't know why this problem does not seem more wide spread. My guess is because our site has a large load and we make extensive use of session variables. Fortunately, I have an easy way to reproduce the problem and even better I've found a work around. My hope is that the work around will lead somebody more familiar with the innards of Zope to where the problem is located. TEST CASE To reproduce the problem I used the following: VA Linux Server w/ 1Gig Fedora Core 3 Zope 2.7.8-final, python 2.3.4, linux2 Python 2.3.4 (#1, Feb 2 2005, 12:11:53) [GCC 3.4.2 20041017 (Red Hat 3.4.2-6.fc3)] top and/or snmp to watch memory usage Step 1: In Zope I created the following script: session = context.REQUEST.SESSION s= 'Andy' * 102400; session['sessiontest1']=s The size of 's' depends purely on how fast you would like Zope to eat memory. On my server this will consume all physical memory in about 1000 calls. Step 2: Changed the session_data object timeout to 1 minute. This is just to cause the session objects to expire in 1 minute to hurry the test along. Changed the Max subobjects to 0 (unlimited). I don't know if this is needed, but it is how we have our production servers set. Step 3: Use ab (apache bench) to call the script 1000 times using 10 concurrent threads. This causes Zope to consume over 90% of the memory as one would expect. The transient object container (/temp_folder/session_data) will now show 1009 objects. I don't know where the 6-9 extra come from.? Step 4: Wait 1-2 minutes for the objects to expire. Note: Use a browser or ab to access some other resource in Zope and or the script a number of times. This is needed to "tickle" the transient object container's expiration/garbage collection system. (You've got to keep those buckets moving) As expected the transient object container now shows 0 objects. However, Zope never releases the memory. You can wait minutes, hours, days what ever all the while tickling Zope. Zope will NEVER release the memory until it is restarted. Oh, and new requests just use more memory. Zope is NOT just hanging on to it for future use. THE WORK ARROUND I created a script called onDelete Parameter List sdo, toc sdo.clear() Then in /temp_folder/session_data I set the Script to call when objects are deleted to /onDelete Now when the tests are run the same amount of memory is consumed, but it is released back to the system when the session objects expire. CONCLUSION To me it looks like data placed in the session object are not getting deleted when the session object expires. Anything placed in a session object is essentially leaked. When the onDelete script explicitly removes the contents of the session data object the memory leak stops. Comment? Questions? Fixes? ;-) Thanks! Andy Yates

3 3

import & export error on.2.9.2 version of zope has been solved
by shivayogi kumbar 14 Apr '06

14 Apr '06

The problem with import and export of zope application from zope2.7.3 to zope2.9.2 has been solved.But when I am sending some request to the server that time It's giving error as follows "Error Type: ImportError Error Value: import of "name of the application" is unauthorized getting this type of error" Is it because of the change in the version of ZODB, in zope2.7.3 they have used the 3.2.4c version ZODB , and in ZOPE2.9.2 used 3.6.0 version ZODB. Sir "Lennart Regebro" has suggested me to copy and paste the data.fs file from Zope2.7.3 where actully my application has developed.Please help me out -- regards Shiv

2 1

Re: [Zope] TAL page whitespace removal
by Sascha Welter 14 Apr '06

14 Apr '06

(Thu, Apr 13, 2006 at 12:00:08PM -0400) zope-request(a)zope.org wrote/schrieb/egrapse: > Date: Wed, 12 Apr 2006 19:03:29 +0100 > From: "Robert (Jamie) Munro" <jamie(a)textmatters.com> ... > gzip will add enormous processing overhead to the server. Your server must be a C=64 or something. I'm running Zope on an Apple Powermac G3 B&W at 350Mhz -- not really an up-to-date powerhouse. Gzipping most of my COREBlog pages shows no noticeable difference in processor load - even on such a lame machine. I'm not even using apaches gzip-encoding module, but the Zope builtin stuff. Being able to get things faster to the clients frees load off the server, the payoff for gzipping content is mucho worth it in my experience. The next thing that paid off huge was implementing handling of If-Modified-Since headers. Bandwidth use for the Googlebot went down from from about 50MB per month to around 1MB. That helps getting processor load and disk load down too. Other search engines and even some "real" clients get 304's too. Regards, Sascha -- http://betabug.ch/blogs/ch-athens

1 0

Strategies for testing generated sql?
by Paul Winkler 14 Apr '06

14 Apr '06

This is a very general question: I'm looking for ideas on how to effectively test dynamically generated sql queries. Both unit and functional test ideas welcome. We have a bunch of views (in the zope 3 / Five sense, but that's not really relevant) which typically construct a Shared.DC.ZRDB.DA.nvSQL instance, passing it a dictionary to use as the "namespace" mapping (I'm simplifying, there's more indirection than that). We have a bunch of tests for this stuff, but typically they just assert that the generated sql query exactly matches some expected string. This strikes me as only a minimal useful regression test when it passes, but it suffers from a number of drawbacks: * When a test fails, it can be very difficult to spot what changed. I'm getting really tired of staring at stuff like (contrived example, but some of my test failures actually look pretty close to this): AssertionError: "select aaaa, aabb, aacc, aadd, aaee, aaff, aagg, bbaa, bbbb, bbcc, bbdd, bbee, bbff, bbgg, ccaa, ccbb, cccc, ccdd, ccee, ccff, ccgg, ddaa, ddbb, ddcc, dddd, ddee, ddff, ddgg, eeaa, eebb, eecc, eedd, eeee, eeff, eegg, ffaa, ffbb, ffcc, ffdd, ffee, ffff, ffgg, ggaa, ggbb, ggcc, ggdd, ggee, ggff, gggg, case when bbcc != '' then bbcc when aaddd != '' then aaddd end as bbcc from blarftable b where (b.aacc = fred and b.aaee != joe) order by aabb" != "select aaaa, aabb, aacc, aadd, aaee, aaff, aagg, bbaa, bbbb, bbcc, bbdd, bbee, bbff, bbgg, ccaa, ccdd, cccc, ccbb, ccee, ccff, ccgg, ddaa, ddbb, ddcc, dddd, ddee, ddff, ddgg, eeaa, eebb, eecc, eedd, eeee, eeff, eegg, ffaa, ffbb, ffcc, ffdd, ffee, ffff, ffgg, ggaa, ggbb, ggcc, ggdd, ggee, ggff, gggg, case when bbcc != '' then bbcc when aaddd != '' then aaddd end as bbcc from blarftable b where (b.aacc = fred and b.aaee != joe) order by aabb" Gee thanks! * I think it would be a lot more useful to be able to reliably parse out and make assertions about some of the sub-clauses. So I could write tests that make assertions like: - this query is selecting columns a, b, c, d in that order. - the "where" clause should compare column X against value Y etc. I could of course do some ad-hoc stuff with string methods and maybe regexes, but I have to wonder if there's some sql-specific library that I could leverage. Googling suggests that there's a parser in gadfly (of course), and an example sql parser in pyparsing. Worth the effort or should I stick with simple string stuff? Functionally, we are missing some important testables: * we have no way to verify that these queries are syntactically correct sql. * we have no way to verify that these queries behave as expected against a sample data set. How do people test this sort of thing? Do you go whole-hog and fire up MySQL or whatever? Or use gadfly? sqlite? or what? Are there other tools I should know about? -- Paul Winkler http://www.slinkp.com

4 6

newbie needs advice on where to use zope
by Sells, Fred 13 Apr '06

13 Apr '06

I realize that to a man with a hammer, every problem is a nail, but... I've been tasked to produce ~50page contracts with a little dynamic information inserted into the appropriate fields. This will run on a linux server and must generate PDF files. These contracts actually consist of multiple packets, with different signature and distribution requirements. Parts of the contract will vary from state to state. I originally did this job a year ago using xml, xslt and fop. While that worked well, it required that all changes had to come to me and then I would edit the xml. While that does lead to job security, it's not the type of job I would like to secure ;) I am receiving the boilerplate in OpenOffice format. and was considering using the python UNO bridge to openoffice or a mailmerge script. Although I have not used OpenOffice before, it is one of our company standards. Part of the problem is that the dynamic data resides in a legacy system that can only be accessed via HTTP. Also these contracts will be done individually, as required at our various field facilities. Furthermore I have little freedom to change appearance, due to various ~legal issues. So after all that rambling, Is Zope a viable tool or am I killing a gnat with a cannon? I have installed both zope and plone in the past, but never used them for more than playing around, so I am concerned about the learning curve. One thing that did appeal to me was that there needs to be a formal approval cycle for changes to the boilerplate, and I seem to recall that Zope is good at that. Are there any prebuilt freebie zope applications that would give me a head start, or any good starting points in general. I'm very familiar with Python. --------------------------------------------------------------------------- The information contained in this message may be privileged and / or confidential and protected from disclosure. If the reader of this message is not the intended recipient, you are hereby notified that any dissemination, distribution or copying of this communication is strictly prohibited. If you have received this communication in error, please notify the sender immediately by replying to this message and deleting the material from any computer. ---------------------------------------------------------------------------

2 1