- Zope - Zope lists

RE: No memory leak! (was: Re: [Zope] Memory Leak in Session Data Container)
by Andy Yates 18 Apr '06

18 Apr '06

Leak, bug, improper configuration, what ever. ;-) With the onDelete in place for 1 week now I checked the temporary Cache Parameters as you requested. Since I put the onDelete on this server memory use has still grown, but at a much slower rate. I think it will go about 10 days like this instead of 1 before using up all system memory. For whatever reason we have 7 caches on this server. I do not see where this is configured. /Control_Panel/Database/temporary/manage_cacheParameters Total number of objects in the database 363761 Total number of objects in memory from all caches 34952 Target number of objects in memory per cache 5000 I flushed the cache and about 1 hour after flushing, memory use has not gone down at all. During that hour objects in all caches went from 0 to 15341. Objects in the transient object container are still holding steady at 2000-3000. Before the onDelete they grew without bound. /Control_Panel/Database/temporary/manage_cacheParameters Total number of objects in the database 369885 Total number of objects in memory from all caches 15341 Target number of objects in memory per cache 5000 It seems odd to me that the total objects in the db are growing so large. If I understand you correctly a workaround would be to change the number of objects per cache from 5000 to something more reasonable. Where is this configured and what is reasonable? 5, 50, 500. I do not see an item for this in the config file. Should there be any cache at all on an in-memory temp db? Andy Yates > -----Original Message----- > From: Dieter Maurer [mailto:dieter@handshake.de] > Sent: Friday, April 14, 2006 7:54 AM > To: Andy Yates; zope(a)zope.org > Subject: No memory leak! (was: Re: [Zope] Memory Leak in Session Data Container) > > Dieter Maurer wrote at 2006-4-11 22:14 +0200: > >Andy Yates wrote at 2006-4-10 12:43 -0500: > >> ... > >>Fortunately, I have an easy way to reproduce the problem and even better > >>I've found a work around. My hope is that the work around will lead > >>somebody more familiar with the innards of Zope to where the problem is > >>located. > > > >Congratulations! > > > >I will look into this within the next days (in Zope 2.8.1, however). > > I checked this: > > There is no memory leak related to session handling in Zope 2.8.1! > > > ... > >The ZODB cache may keep your session objects (and thereby its > >content) alive. > > Almost surely, this is what happened in Andy's test. > > > Here is a description how I checked for the existence of a memory > leak: > > I instrumented the pickles in "tempstorage.TemporaryStorage.TemporaryStorage" > such that I can find out the number of pickles stored > (similar to the reference counts in "Control_Panel/DebugInfo"). > > I performed a mass test, creating thousands of sessions > (similar to Andy's test). > > The pickle number stored by "TemporaryStorage" grew as expected > (as it stores the sessions). > > I forced all session objects maintained by "/temp_folder/session_data" > to be discarded. > > Creating a single new session caused almost all "TemporaryStorage" > pickles to be released. > > However, the ZODB caches still contained about 20.000 session > objects. It was very surprising (and contrary to my previous > message) that the storage flushed most of its pickles > while references to them were still in the ZODB caches. > This looks like a bug (but not a memory leak). > > I flushed the ZODB caches. > > I repeated the test and verified that the number of > pickles remaining in "TemporaryStorage" does not increase. > > > Results: > > * no memory leak! > > * maybe something that looks like a memory leak caused by the ZODB caches > which may hold up to 20.000 (with standard configuration) no > longer used objects > > A more senseful configuration can work around this. > > > -- > Dieter

2 1

DELETE Objects
by Erik Billing 18 Apr '06

18 Apr '06

I want to create a python script that deletes a specific file from a folder. For now, the user who is calling the script does not have the Delete objects permission on the folder, only the file which he should be able to delete. I have previously used the delObjects method on the folder to remove the file, but this requires the Delete obejcts permission on the folder itself. I would like to do something like container[entry_id].DELETE() but the DELETE method requires 3 arguments, and I dont know what to give it? Or is it better to go in the delObjects on the container and use a proxy to solve the permission problems? Best regards /Erik

3 2

Export html
by BKJ＠telefonica.net 18 Apr '06

18 Apr '06

Hello, I have an archetype, I want to export the html that I see in the view mode to a rtf (or similar) format. I tried to use the product "RTFExport" but it doesn't export from HTML (or I don't know how to do this). I have found the product "PloneOOoTransforms" , does anybody have experience with this product? Can this product export the html of the view mode to another format. Thanks (I apologize If I've been using the wrong list to post this question) TERRA -->

1 0

rss republishing
by David Bear 18 Apr '06

18 Apr '06

I have a plone intranet site running on box A. I have a public web server, running apache on box B. I would like to have a plone topic published as an rss feed, which Box B republishes for clients. I don't want the clients to know the pages originate on box A. I would like them to appear from box b -- and even cached for a period of time. Are there any zope products that might help with these? or an apache module if there apache experts here. I might even entertain an mod-python module if something like it existed. -- David Bear What's the difference between private knowledge and public knowledge?

2 1

what process does Zope run as?
by Lars Hohmuth 18 Apr '06

18 Apr '06

What process does Zope run as in Windows XP? I finally tracked down the 10053 errors in Plone's MailHost down to port blocking in one of McAfee's virus scanners. But neither enabling port 25 for python.exe, java.exe, or pythonservice.exe seems to work. Getting rid of the port 25 blocking does, but that gives my sysadmin fits. Thanks, Lars

2 1

Install 2.9.2 fails with "error: must supply either home or prefix/exec-prefix -- not both"
by Jerry Westrick 17 Apr '06

17 Apr '06

Hello all.... I've done a clean install SUSE 10.1 RC1. (Yes I know it's new, and probably the problem) I've downloaded Zope 2.9.2 And executed the installation as described in the readme.txt. I've tried it with both --prefix and without. Any Ideas where to look would be appreceated... Here the result of the installation: log2:/backup/install/zope # tar -zxf Zope-2.9.2.tgz log2:/backup/install/zope # cd Zope-2.9.2 log2:/backup/install/zope/Zope-2.9.2 # ./configure Configuring Zope installation Testing for an acceptable Python interpreter... Python version 2.4.2 found at /usr/bin/python The optimum Python version (2.4.2) was found at /usr/bin/python. log2:/backup/install/zope/Zope-2.9.2 # make /usr/bin/python install.py -q build Dependencies/ExtensionClass-Zope-2.9.2/ExtensionClass/_ExtensionClass.c: In function ?PyExtensionClass_Export_?: Dependencies/ExtensionClass-Zope-2.9.2/ExtensionClass/_ExtensionClass.c:768: warning: cast from pointer to integer of different size log2:/backup/install/zope/Zope-2.9.2 # make install /usr/bin/python install.py -q build /usr/bin/python install.py -q install --skip-build --home "/usr/local/Zope-2.9.2" error: must supply either home or prefix/exec-prefix -- not both make: *** [install] Error 1 log2:/backup/install/zope/Zope-2.9.2 #

5 10

LocalFS under Zope2-7-7/windows
by Giuseppe Bonelli 17 Apr '06

17 Apr '06

I all, I am migrating some apps to Zope2-7-7 under windows and the LocalFS product generate an error wen starting zope. The error at the end of the traceback is: File "C:\Programmi\Zope-2.7.7-final\lib\python\Signals\WinSignalHandler.py", line 227, in registerHandler hevent = win32event.CreateEvent(sa, 0, 0, event_name) TypeError: The object is not a PySECURITY_ATTRIBUTES object Any pointer on fixing this? Can someone plese confirm that LocalFS works fine under Unixes? Is there any other solution for serving files out from filesystem directories under Zope-2-7-7/win? TIA __peppo

2 1

Structure for large schema
by David Pratt 17 Apr '06

17 Apr '06

Hi. I had asked this question on zope3 about a week ago or so an had no responses. I am hoping I can receive some general guidance on this issue. I am trying to determine the best structure for storing a large schema where some attributes are lists or dictionaries. There are about 70 attributes in the schema and I am trying to choose a structure that will not necessarily have to hold a bunch of empty space. I have a base schema of approximately 30 attributes and others that subclass from it with the largest being about 70. I had originally thought of RDF at the onset and built a datastore with rdflib using a relational database. I chose this option because when ZODB gets larger it takes plenty of RAM. Problem here is the number of accesses to gather a complete object. It is pretty efficient from a storage perspective since if an object does not have particular attributes, you are not storing them and all items in the store are unique. I was not duplicating one piece of data. But say you wanted to present a page consisting of 20 items or do a search. Gathering this up takes much time when you are hitting a disk how many times to gather up just a single object so query times were unacceptable and loading rdf from outside sources also took a very long time. The data store grows into millions of records so you better have a pretty sweet rdb server with lots of RAM also. I had dismissed a relational database on its own since the data does not lend to a row and I may want to add to the schema at some point in time which could mean some pretty ugly business this way. But then I saw the vertical example in the examples folder of SQLAlchemy that can do something to create dynamic fields as necessary to potentially avoid this kind of hassle. The ZODB provides the flexibility and Generations could work well for future updates so this looks very good but how efficient is it if 15% of the attributes have data and 85% do not? I have also been experimenting with hybrid pickle / rdb storage so that the attributes that will receive the most attention are stored as fields and the full record is stored as pickle that is unpickled for views and data entry. In any case. Thought I'd ask since I am concerned about the efficiency of storage and speed of access both. If rdf access was fast then it would be great but this had not been the case. I just thought there may be some other ideas on this or someone could advise on the efficiency of ZODB when in some cases, uses will be selective about which attributes are important to them. Regards, David

2 3

Generic SQL insert
by Robert (Jamie) Munro 17 Apr '06

17 Apr '06

Whenever I'm using SQL databases in zope, I always seem to have to make a ZSQL instance for inserting into every table in my database, and they are all nearly the same - they just have a list of all the fields in the database in the parameters, then they say: insert into [table] ([list of fields]) values ([list of <dtml-sqlvar>s]) I'd much rather have a dictionary of fields and values, and just throw it at the DB, not having to make those queries for every table. I have acheived it like so: mydict = {"field1":"value1" , "field2":"value2" ,...} (fields,values)=zip(*myDict.items()) context.genericInsert(table='table name',fields=fields,values=values) Where generic insert is the following ZSQL method: insert into "<dtml-var table>" (<dtml-in expr="fields">"<dtml-var sequence-item>"<dtml-if sequence-end><dtml-else>,</dtml-if></dtml-in>) values (<dtml-in expr="values"><dtml-sqlvar sequence-item type=string><dtml-if sequence-end><dtml-else>,</dtml-if></dtml-in>); with parameters: * table - table name * fields - list of fieldnames * values - list of values in the same order What do other people think of this? Is it a really bad idea? Robert Munro

4 3

Generic SQL insert & Rakun XML Application Platform
by Evrim Ozcelik 17 Apr '06

17 Apr '06

Hi, Have you tried Rakun Project for this problem. It supports MySQL, PostgreSQL and SQLite for now. But it can handle all kind of SQL databases. Please take a look at its web site: http://www.rakun.org/ For all kind of questions about it, you can contact me >Content-Type: text/plain; charset="US-ASCII" > >Great idea. Not to be recommended in general. > >This works because every field is textual, and you are >sql-quoting by using type=string. > >Here are the problems: >1) if someone reads this and does not use the type=string >tag, or equivalent, they will be wide open to sql injection. >2) OR, they can pass a list of type with each variable. >3) If you have to handle casts, then you will have to pass >a list of cast-types, as well. > >So, you have essentially moved the problem from making at >least one insertion call per table to a single insertion method >that requires the creation of two, three, or four lists. This does >not self-evidently require less work. > >You can no longer inspect the method to see if it is correct. >You have to look to each call-point to determine what is actually >being used. Just as bad, your application goes happily on its way if you >are missing (non-key) variables. > >Keep zsql methods a simple as possible. Use as few tricks as >possible. Your goal is self-evident correctness, not the minimization >of typing. >jim penny zope-bounces(a)zope.org wrote on 04/13/2006 02:23:22 PM: > Whenever I'm using SQL databases in zope, I always seem to have to make > a ZSQL instance for inserting into every table in my database, and they > are all nearly the same - they just have a list of all the fields in the > database in the parameters, then they say: > > insert into [table] ([list of fields]) values ([list of <dtml-sqlvar>s]) > > I'd much rather have a dictionary of fields and values, and just throw > it at the DB, not having to make those queries for every table. I have > acheived it like so: > > mydict = {"field1":"value1" , "field2":"value2" ,...} > (fields,values)=zip(*myDict.items()) > context.genericInsert(table='table name',fields=fields,values =values) > > Where generic insert is the following ZSQL method: > insert into "<dtml-var table>" > (<dtml-in expr="fields">"<dtml-var sequence-item>"<dtml-if > sequence-end><dtml-else>,</dtml-if></dtml-in>) > values (<dtml-in expr="values"><dtml-sqlvar sequence-item > type=string><dtml-if sequence-end><dtml-else>,</dtml-if></dtml-in>); > > with parameters: > * table - table name > * fields - list of fieldnames > * values - list of values in the same order > > What do other people think of this? Is it a really bad idea? > > Robert Munro > _______________________________________________ > Zope maillist - Zope(a)zope.org > http://mail.zope.org/mailman/listinfo/zope > ** No cross posts or HTML encoding! ** > (Related lists - > http://mail.zope.org/mailman/listinfo/zope-announce > http://mail.zope.org/mailman/listinfo/zope-dev ) >

1 0