April 2006 - Zope - Zope lists

Structure for large schema
by David Pratt 17 Apr '06

17 Apr '06

Hi. I had asked this question on zope3 about a week ago or so an had no responses. I am hoping I can receive some general guidance on this issue. I am trying to determine the best structure for storing a large schema where some attributes are lists or dictionaries. There are about 70 attributes in the schema and I am trying to choose a structure that will not necessarily have to hold a bunch of empty space. I have a base schema of approximately 30 attributes and others that subclass from it with the largest being about 70. I had originally thought of RDF at the onset and built a datastore with rdflib using a relational database. I chose this option because when ZODB gets larger it takes plenty of RAM. Problem here is the number of accesses to gather a complete object. It is pretty efficient from a storage perspective since if an object does not have particular attributes, you are not storing them and all items in the store are unique. I was not duplicating one piece of data. But say you wanted to present a page consisting of 20 items or do a search. Gathering this up takes much time when you are hitting a disk how many times to gather up just a single object so query times were unacceptable and loading rdf from outside sources also took a very long time. The data store grows into millions of records so you better have a pretty sweet rdb server with lots of RAM also. I had dismissed a relational database on its own since the data does not lend to a row and I may want to add to the schema at some point in time which could mean some pretty ugly business this way. But then I saw the vertical example in the examples folder of SQLAlchemy that can do something to create dynamic fields as necessary to potentially avoid this kind of hassle. The ZODB provides the flexibility and Generations could work well for future updates so this looks very good but how efficient is it if 15% of the attributes have data and 85% do not? I have also been experimenting with hybrid pickle / rdb storage so that the attributes that will receive the most attention are stored as fields and the full record is stored as pickle that is unpickled for views and data entry. In any case. Thought I'd ask since I am concerned about the efficiency of storage and speed of access both. If rdf access was fast then it would be great but this had not been the case. I just thought there may be some other ideas on this or someone could advise on the efficiency of ZODB when in some cases, uses will be selective about which attributes are important to them. Regards, David

2 3

Generic SQL insert
by Robert (Jamie) Munro 17 Apr '06

17 Apr '06

Whenever I'm using SQL databases in zope, I always seem to have to make a ZSQL instance for inserting into every table in my database, and they are all nearly the same - they just have a list of all the fields in the database in the parameters, then they say: insert into [table] ([list of fields]) values ([list of <dtml-sqlvar>s]) I'd much rather have a dictionary of fields and values, and just throw it at the DB, not having to make those queries for every table. I have acheived it like so: mydict = {"field1":"value1" , "field2":"value2" ,...} (fields,values)=zip(*myDict.items()) context.genericInsert(table='table name',fields=fields,values=values) Where generic insert is the following ZSQL method: insert into "<dtml-var table>" (<dtml-in expr="fields">"<dtml-var sequence-item>"<dtml-if sequence-end><dtml-else>,</dtml-if></dtml-in>) values (<dtml-in expr="values"><dtml-sqlvar sequence-item type=string><dtml-if sequence-end><dtml-else>,</dtml-if></dtml-in>); with parameters: * table - table name * fields - list of fieldnames * values - list of values in the same order What do other people think of this? Is it a really bad idea? Robert Munro

4 3

Generic SQL insert & Rakun XML Application Platform
by Evrim Ozcelik 17 Apr '06

17 Apr '06

Hi, Have you tried Rakun Project for this problem. It supports MySQL, PostgreSQL and SQLite for now. But it can handle all kind of SQL databases. Please take a look at its web site: http://www.rakun.org/ For all kind of questions about it, you can contact me >Content-Type: text/plain; charset="US-ASCII" > >Great idea. Not to be recommended in general. > >This works because every field is textual, and you are >sql-quoting by using type=string. > >Here are the problems: >1) if someone reads this and does not use the type=string >tag, or equivalent, they will be wide open to sql injection. >2) OR, they can pass a list of type with each variable. >3) If you have to handle casts, then you will have to pass >a list of cast-types, as well. > >So, you have essentially moved the problem from making at >least one insertion call per table to a single insertion method >that requires the creation of two, three, or four lists. This does >not self-evidently require less work. > >You can no longer inspect the method to see if it is correct. >You have to look to each call-point to determine what is actually >being used. Just as bad, your application goes happily on its way if you >are missing (non-key) variables. > >Keep zsql methods a simple as possible. Use as few tricks as >possible. Your goal is self-evident correctness, not the minimization >of typing. >jim penny zope-bounces(a)zope.org wrote on 04/13/2006 02:23:22 PM: > Whenever I'm using SQL databases in zope, I always seem to have to make > a ZSQL instance for inserting into every table in my database, and they > are all nearly the same - they just have a list of all the fields in the > database in the parameters, then they say: > > insert into [table] ([list of fields]) values ([list of <dtml-sqlvar>s]) > > I'd much rather have a dictionary of fields and values, and just throw > it at the DB, not having to make those queries for every table. I have > acheived it like so: > > mydict = {"field1":"value1" , "field2":"value2" ,...} > (fields,values)=zip(*myDict.items()) > context.genericInsert(table='table name',fields=fields,values =values) > > Where generic insert is the following ZSQL method: > insert into "<dtml-var table>" > (<dtml-in expr="fields">"<dtml-var sequence-item>"<dtml-if > sequence-end><dtml-else>,</dtml-if></dtml-in>) > values (<dtml-in expr="values"><dtml-sqlvar sequence-item > type=string><dtml-if sequence-end><dtml-else>,</dtml-if></dtml-in>); > > with parameters: > * table - table name > * fields - list of fieldnames > * values - list of values in the same order > > What do other people think of this? Is it a really bad idea? > > Robert Munro > _______________________________________________ > Zope maillist - Zope(a)zope.org > http://mail.zope.org/mailman/listinfo/zope > ** No cross posts or HTML encoding! ** > (Related lists - > http://mail.zope.org/mailman/listinfo/zope-announce > http://mail.zope.org/mailman/listinfo/zope-dev ) >

1 0

CMF 2.0.0 released
by Jens Vagelpohl 16 Apr '06

16 Apr '06

-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1 The CMF developer community and Zope Corporation are pleased to announce the release of version 2.0.0 of the Zope Content Management Framework (CMF) as the first stable release from the CMF 2.0 development branch. What is the CMF? The Zope Content Management Framework provides a set of services and content objects useful for building highly dynamic, content-oriented portal sites. As packaged, the CMF generates a site much like the zope.org site. The CMF is intended to be easily customizable, in terms of both the types of content used and the policies and services it provides. What's new in this release? In addition to countless bug fixes and minor improvements, this release features many noteworthy additions, such as: - GenericSetup-integration as the replacement for the CMFSetup technology started in the CMF 1.5-series - the use of Zope 3-style interfaces throughout, bridged back to Zope 2-interfaces where needed for backwards compatibility - An experimental set of Five-style browser views for CMFDefault and CMFCalendar, which will evolve into the standard views for CMF 2.1 - widespread integration with Zope 3/Five technologies, and the folding-in of the previous CMFonFive addon-product - pluggable TypeInformation objects - new-style Actions Where do I get it? Download it from http://www.zope.org/Products/CMF/CMF-2.0.0 Points of interest include: - Windows ZIP file: http://www.zope.org/Products/CMF/CMF-2.0.0/CMF-2.0.0.zip - Unix tar/gzip archive: http://www.zope.org/Products/CMF/CMF-2.0.0/CMF-2.0.0.tar.gz - Release notes: http://www.zope.org/Products/CMF/CMF-2.0.0/README.txt - Change history: http://www.zope.org/Products/CMF/CMF-2.0.0/CHANGES.txt - Installation instructions: http://www.zope.org/Products/CMF/CMF-2.0.0/INSTALL.txt - CMF Release Roadmap: http://www.zope.org/Products/CMF/docs/roadmap Where do I go to learn more? The CMF mailing list (zope-cmf(a)zope.org) has many participants who are active in supporting the CMF. ...to report bugs? The CMF Collector at http://www.zope.org/Collectors/CMF is the place to report bugs (please search for existing reports of your issue first!) - --------- Jens Vagelpohl jens(a)dataflake.org -----BEGIN PGP SIGNATURE----- Version: GnuPG v1.4.1 (Darwin) iD8DBQFEQhVPRAx5nvEhZLIRAmilAJ9YY0YeqZK5SSFG88+ooWYNoqV4iACeJLJx lbGZzmDOcn1EnuiNAEkgtCY= =nyIK -----END PGP SIGNATURE-----

2 1

Re: [Zope] Generic SQL insert
by Michael Shulman 15 Apr '06

15 Apr '06

"Robert (Jamie) Munro" <jamie(a)textmatters.com> wrote: > I'd much rather have a dictionary of fields and values, and just throw > it at the DB, not having to make those queries for every table. I have > acheived it like so: I have achieved a similar result, although for update methods in my case, in what I think is a slightly better way (although probably people on this list will tell me I'm wrong). Rather than having a single generic ZSQL method, I have one ZSQL method for each table, but intsead of writing them all by hand I have a single script which generates/regenerates them from inspection of the columns in the database, like so: # select some arbitrary record from the table into table_data, then: fields = table_data.data_dictionary() arguments = "" comma = "" print "update %s set" % table types = {'s':"nb", 'd':"nb", 'i':"int", 'n':"nb"} for field in fields.keys(): arguments = arguments + field + " " type = types[fields[field]["type"]] print " %s%s = <dtml-if expr=\"%s is None\">null<dtml-else><dtml-sqlvar %s type=\"%s\" optional></dtml-if>" % (comma, field, field, field, type) comma = "," print " where %s = <dtml-sqlvar %s type=\"int\">" % (id_field, id_field) # update_zsql is the ZSQL "update" method I am creating/updating update_zsql.manage_edit('','database_name',arguments,printed) This way, all the columns are assigned the correct type, but the calling script doesn't have to handle types or casts, since all the types are assigned by inspecting the actual database. Also, this way the update/insert will fail if any fields are missing. Actually, given how object-oriented Zope is in general, I was surprised to find that its database interface (ZSQL methods) still requires the user to write SQL code essentially by hand, rather than using an object-relational mapper like SQLObject. There appears to be a toolkit called sqlos for using SQLObject with Zope 3, but I haven't yet tried it out. > > mydict = {"field1":"value1" , "field2":"value2" ,...} > (fields,values)=zip(*myDict.items()) > context.genericInsert(table='table name',fields=fields,values=values) > > Where generic insert is the following ZSQL method: > insert into "<dtml-var table>" > (<dtml-in expr="fields">"<dtml-var sequence-item>"<dtml-if > sequence-end><dtml-else>,</dtml-if></dtml-in>) > values (<dtml-in expr="values"><dtml-sqlvar sequence-item > type=string><dtml-if sequence-end><dtml-else>,</dtml-if></dtml-in>); > > with parameters: > * table - table name > * fields - list of fieldnames > * values - list of values in the same order > > What do other people think of this? Is it a really bad idea? > > Robert Munro

1 0

zope 2.8.6 on Mac Intel
by manuel spuhler 15 Apr '06

15 Apr '06

Hi, does anyone know how-to compile Zope 2.8.6 on a OS X system with an Intel processor?

6 20

Re: zope 2.8.6 on Mac Intel: Resolved
by manuel spuhler 14 Apr '06

14 Apr '06

Hello David, sorry, I meant you before, but hey, hello too Dieter ;-) I finally manage to install and run Zope 2.8.6 on Mac Intel after installing the Universal package of python 2.4.3 for os x (http://www.python.org/download/releases/2.4.3/) Then I did the usual unix stuff: $./configure --with-python=/usr/local/bin/python --prefix=/opt/Zope2.8 && make &&make install $ /opt/Zope2.8/bin/mkzopeinstance.py Even if this Python version is not supported, I didn't notice (yet;-) any problem. As far as I know, it's the 1st Zope 2.8.6 and Plone 2.11 on Mac Intel and it's at least as fast as on Linux !!! Thank you all for your help and happy zoping! Manuel Zope Version (Zope 2.8.6-final, python 2.4.3, darwin) Python Version 2.4.3 (#1, Mar 30 2006, 11:02:16) [GCC 4.0.1 (Apple Computer, Inc. build 5250)] System Platform darwinope Version (Zope 2.8.6-final, python 2.4.3, darwin) Python Version 2.4.3 (#1, Mar 30 2006, 11:02:16) [GCC 4.0.1 (Apple Computer, Inc. build 5250)] System Platform darwin

1 0

How to update list from dtml?
by Jaroslav Lukesh 14 Apr '06

14 Apr '06

Dear Zopistas, How to do in DTML something similar in python: foo[1] = 3.5 In DTML this does not work: <dtml-call "REQUEST.set('foo[1]', 3.5)"> Note, that I have list foo[,,,,,,,] already defined Many thanks, J. Lukesh

4 4

Getting www.domain.com:8080/manage to go through Apache
by michael nt milne 14 Apr '06

14 Apr '06

Hi I've got various virtual hosts running but am having a few issues getting the administration site on www.domain.com:8080/manage to go through Apache. I want to route this all through SSL. What I can get is the admin for the *individual site* but not all the sites listed under a Zope instance. I'm trying to use <location></location> to put a request for www.domain.com/manage through to www.domain.com:8080/manage in Zope and then back out through port 443. I haven't set up the SSL certificates yet. Here's what I've got so far which isn't working quite as planned. Anyone achieved this at all? Thanks NameVirtualHost xx.x.xxx.xxx:80 <VirtualHost xx.x.xxx.xxx:80> ServerName www.domain.net <location /manage> RewriteEngine On RewriteRule ^/(.*) http://www.domain.co.uk:8080/manage/VirtualHostBase/https/www.domain.net:44… </location> </VirtualHost> -- michael

1 0

Memory Leak in Session Data Container
by Andy Yates 14 Apr '06

14 Apr '06

Good day everyone! I hope your weekend was as good as mine. I didn't have to work and I got to watch my son's first soccer (football) game! For the past couple of years (I think) the Session Data Container has been leaking memory. During this time several leaks have been patch but at least for our servers these fixes have only slowed down the process. As our use of Zope grows and traffic to our site increases so does the memory usage of Zope. Unless Zope is restarted it will eventually use all physical memory and swap space and crash the server. Over the past month the problem has once again grown to the point where we must restarted Zope daily, sometimes even every 8 hours. I don't know why this problem does not seem more wide spread. My guess is because our site has a large load and we make extensive use of session variables. Fortunately, I have an easy way to reproduce the problem and even better I've found a work around. My hope is that the work around will lead somebody more familiar with the innards of Zope to where the problem is located. TEST CASE To reproduce the problem I used the following: VA Linux Server w/ 1Gig Fedora Core 3 Zope 2.7.8-final, python 2.3.4, linux2 Python 2.3.4 (#1, Feb 2 2005, 12:11:53) [GCC 3.4.2 20041017 (Red Hat 3.4.2-6.fc3)] top and/or snmp to watch memory usage Step 1: In Zope I created the following script: session = context.REQUEST.SESSION s= 'Andy' * 102400; session['sessiontest1']=s The size of 's' depends purely on how fast you would like Zope to eat memory. On my server this will consume all physical memory in about 1000 calls. Step 2: Changed the session_data object timeout to 1 minute. This is just to cause the session objects to expire in 1 minute to hurry the test along. Changed the Max subobjects to 0 (unlimited). I don't know if this is needed, but it is how we have our production servers set. Step 3: Use ab (apache bench) to call the script 1000 times using 10 concurrent threads. This causes Zope to consume over 90% of the memory as one would expect. The transient object container (/temp_folder/session_data) will now show 1009 objects. I don't know where the 6-9 extra come from.? Step 4: Wait 1-2 minutes for the objects to expire. Note: Use a browser or ab to access some other resource in Zope and or the script a number of times. This is needed to "tickle" the transient object container's expiration/garbage collection system. (You've got to keep those buckets moving) As expected the transient object container now shows 0 objects. However, Zope never releases the memory. You can wait minutes, hours, days what ever all the while tickling Zope. Zope will NEVER release the memory until it is restarted. Oh, and new requests just use more memory. Zope is NOT just hanging on to it for future use. THE WORK ARROUND I created a script called onDelete Parameter List sdo, toc sdo.clear() Then in /temp_folder/session_data I set the Script to call when objects are deleted to /onDelete Now when the tests are run the same amount of memory is consumed, but it is released back to the system when the session objects expire. CONCLUSION To me it looks like data placed in the session object are not getting deleted when the session object expires. Anything placed in a session object is essentially leaked. When the onDelete script explicitly removes the contents of the session data object the memory leak stops. Comment? Questions? Fixes? ;-) Thanks! Andy Yates

3 3