RE: [ZODB-Dev] RE: [Zope] Analyzing ZODB objects
From: Dieter Maurer [mailto:dieter@handshake.de]
Forget about this approach. It might come with Python 3 or Python 4, but it is unlikely. Python is a high level language hiding memory usage; you want precise information about memory usage. I doubt you will find enough arguments and use cases to get this into Python. [...] Do you really care about the size of objects in memory? We no longer live in 1980 when memory has been a scarce resource.
Well, we still run out of memory, and then it's useful to know why. :) Caring about memory usage is caring about (one kind of) scalability. I realize I can't change this aspect of Python, and it's not that critical to know if you can deduce memory usage from pickle sizes (like you mention). [...]
I had a similar problem (the ZODB grew far too fast) and I wanted to understand why.
I extended Zope's "Undo" information to include the transaction size. This allowed me to see precisely which transactions were larger than expected.
I extended the "fsdump" utility to include the (pickle) sizes of the object records contained in a transaction and to restrict the range of dumped transactions.
This has been enough to analyse the problem: ZCatalog's Metadata records caused a transaction size to grow from an expected few hundred bytes to about 500 kB.
Any chance of seeing these changes in the core? [...] The remainder of my use cases can be currently best supported by using a debugger + DocFinder, I agree, although I think there's still a need for an admin tool that lets you easily view and browse (multiple) objects, independent of if they catalogaware or has an ZMI interface. I'll think so more about it. Thanks for all the help. -- Bjorn
On Monday 27 October 2003 01:45, Bjorn Stabell wrote:
Do you really care about the size of objects in memory? We no longer live in 1980 when memory has been a scarce resource.
Well, we still run out of memory, and then it's useful to know why. :) Caring about memory usage is caring about (one kind of) scalability.
How would you debug this problem in an application that didnt use ZODB? Why does ZODB make a difference? -- Toby Dickenson
Bjorn Stabell wrote at 2003-10-27 09:45 +0800:
From: Dieter Maurer [mailto:dieter@handshake.de] .... I had a similar problem (the ZODB grew far too fast) and I wanted to understand why.
I extended Zope's "Undo" information to include the transaction size. This allowed me to see precisely which transactions were larger than expected.
I extended the "fsdump" utility to include the (pickle) sizes of the object records contained in a transaction and to restrict the range of dumped transactions.
This has been enough to analyse the problem: ZCatalog's Metadata records caused a transaction size to grow from an expected few hundred bytes to about 500 kB.
Any chance of seeing these changes in the core?
I cannot speak about the core... But, I can make the extensions available. -- Dieter
On Tue, Oct 28, 2003 at 07:44:05PM +0100, Dieter Maurer wrote:
I extended the "fsdump" utility to include the (pickle) sizes of the object records contained in a transaction and to restrict the range of dumped transactions.
This has been enough to analyse the problem: ZCatalog's Metadata records caused a transaction size to grow from an expected few hundred bytes to about 500 kB.
But, I can make the extensions available.
I'd like to see your fsdump version; I've done that teensy mod I posted to my version, which is interesting, but yours probably blows mine out of the water. Take care, -- Christian Robottom Reis | http://async.com.br/~kiko/ | [+55 16] 261 2331
Christian Robottom Reis wrote at 2003-10-28 23:44 -0200:
On Tue, Oct 28, 2003 at 07:44:05PM +0100, Dieter Maurer wrote:
I extended the "fsdump" utility to include the (pickle) sizes of the object records contained in a transaction and to restrict the range of dumped transactions.
This has been enough to analyse the problem: ZCatalog's Metadata records caused a transaction size to grow from an expected few hundred bytes to about 500 kB.
But, I can make the extensions available.
I'd like to see your fsdump version ...
Patch attached. The patch also enhances "FileStorage" and lets its "UndoSearch._readnext" provide information about the transaction size. This is very valuable when you want to spot strange transaction sizes via Zope's "Undo" tab (much more convenient than "fsdump"). -- Dieter
On Wed, Oct 29, 2003 at 08:55:18PM +0100, Dieter Maurer wrote:
The patch also enhances "FileStorage" and lets its "UndoSearch._readnext" provide information about the transaction size. This is very valuable when you want to spot strange transaction sizes via Zope's "Undo" tab (much more convenient than "fsdump").
Jeremy, the fsdump part of this patch seems quite non-intrusive, and the UndoSearch bit is probably harmless too. Can you take a look and see if it's something you'd accept in ZODB3? (Perhaps after fixing it to PEP-8 formatting and 76-char wrapping :-) Take care, -- Christian Robottom Reis | http://async.com.br/~kiko/ | [+55 16] 261 2331
Hi Zopers, I'm trying to build a semi-real time game with a Flash client and a Zope server. Flash allows to kinds of server communication, connected sockets and http. Using Zope with the latter is relatively easy, but implies that the clients poll the server all the time, just in case there is something for them. Also, creating http connections all the time is relatively slow and costs resources. It would be better to use a socket based approach. I tried the Swocket project (http://swocket.sourceforge.net) and it works fine, but programming against it is by far not as elegant as programming in Zope and debugging is very hard indeed. So I am thinking of a socket server. The first option would be to have something like swocket run in parallel with Zope with a protocol to call zope methods over xml-rpc or something. The other option would be to create a socket server in zope, maybe in a static variable of a product (I understand that they are not flushed by Zope). The first option is probably easier, but you still have the xml-rpc overhead (though less of a need for polling with the clients). The second one sounds complicated. Any ideas on this or similar projects? I found an irc product that might be of some help. Haven't tried it though. Douwe Osinga http://douweosinga.com
On Wed, 2004-01-07 at 04:47, Douwe Osinga wrote:
Hi Zopers,
I'm trying to build a semi-real time game with a Flash client and a Zope server. Flash allows to kinds of server communication, connected sockets and http. Using Zope with the latter is relatively easy, but implies that the clients poll the server all the time, just in case there is something for them. Also, creating http connections all the time is relatively slow and costs resources. It would be better to use a socket based approach.
You might want to take a look at Twisted. http://twistedmatrix.com/products/twisted HTH, Dylan
Hi Zopers,
I'm trying to build a semi-real time game with a Flash client and a Zope server. Flash allows to kinds of server communication, connected sockets and http. Using Zope with the latter is relatively easy, but implies that the clients poll the server all the time, just in case there is something for them. Also, creating http connections all the time is relatively slow and costs resources. It would be better to use a socket based approach.
You might want to take a look at Twisted.
I did and it looks impressive. But it is not Zope. I would prefer to keep my scripts development in Zope. Also, I have Zope hosting. But thanks for the link, like I said, it looks impressive. Douwe
Douwe Osinga wrote at 2004-1-7 13:47 +0100:
I'm trying to build a semi-real time game with a Flash client and a Zope server. Flash allows to kinds of server communication, connected sockets and http. Using Zope with the latter is relatively easy, but implies that the clients poll the server all the time, just in case there is something for them. Also, creating http connections all the time is relatively slow and costs resources. It would be better to use a socket based approach.
You will need to implement a new server type (analogous to the FTP server, probably). -- Dieter
Hi Zopers, I have a musing about how Zope sites are set-up which I would like to share and would very much like comments on. Typically, a web page has a header, a body and a footer. In most Zope instances, this is translated into standard_html_header, standard_html_footer, while an index_html calls these and does something interesting in between. I don't like this setup too much, because it explicitly repeats the calling of standard_html_* for every page. If you want to add a parameter to for example standard_html_header (say the css to use or similar), you'll have to change all pages (yes, you can use default parameters, but that is not the point). So I usually end up with a structure like this: root index_html standard_html_footer standard_html_header mainbody folder1 mainbody folder2 mainbody where index_html calls header, mainbody and footer. This works great, except for that you need a folder for any page that needs to display the structure. If you want add a method to the root folder that is applicable to everypage, then this method needs to call the structure again. One could of course create a method that takes as a parameter the contents (mainbody) of a page and than calls the structure methods, but that works clumsy too. I have been thinking in subclassing python script in a way that it calls index_html and inserts the result of the script in the position of the call to mainbody. Do other people recognize this pattern? Other solutions? I'd love to hear. Douwe Osinga http://douweosinga.com
On Thu, 2004-01-15 at 06:52, Douwe Osinga wrote:
Hi Zopers,
I have a musing about how Zope sites are set-up which I would like to share and would very much like comments on.
Typically, a web page has a header, a body and a footer. In most Zope instances, this is translated into standard_html_header, standard_html_footer, while an index_html calls these and does something interesting in between.
I don't like this setup too much, because it explicitly repeats the calling of standard_html_* for every page. If you want to add a parameter to for example standard_html_header (say the css to use or similar), you'll have to change all pages (yes, you can use default parameters, but that is not the point).
[snip] Have you looked at using Page Templates? Typically, that you define a master template with a content slot. You then merely use the master macro and fill the content slot. Later, if you decide to spruce up each page with new gizmos, you do that all in master. Cheers, // m
This is a typical issue, and the best workaround I can give you is "write your own Zope objects". With this approach you can define what are going to be your content primitives and how their look and feel will be, according to its contents and the context. This is in fact means that you can have as many index_htmls as the number of objects you define. In CMF you do this in a very friendly way by by creating portal types. But if you don't want to trip into canalled waterways, you can write a product on your own. Ausum ----- Original Message ----- From: "Douwe Osinga" <dmo@oberon.nl> To: <zope@zope.org> Sent: Thursday, January 15, 2004 7:52 AM Subject: [Zope] Zope Patterns
Hi Zopers,
I have a musing about how Zope sites are set-up which I would like to share and would very much like comments on.
Typically, a web page has a header, a body and a footer. In most Zope instances, this is translated into standard_html_header, standard_html_footer, while an index_html calls these and does something interesting in between.
I don't like this setup too much, because it explicitly repeats the calling of standard_html_* for every page. If you want to add a parameter to for example standard_html_header (say the css to use or similar), you'll have to change all pages (yes, you can use default parameters, but that is not the point). So I usually end up with a structure like this:
root index_html standard_html_footer standard_html_header mainbody folder1 mainbody folder2 mainbody
where index_html calls header, mainbody and footer. This works great, except for that you need a folder for any page that needs to display the structure. If you want add a method to the root folder that is applicable to everypage, then this method needs to call the structure again.
One could of course create a method that takes as a parameter the contents (mainbody) of a page and than calls the structure methods, but that works clumsy too.
I have been thinking in subclassing python script in a way that it calls index_html and inserts the result of the script in the position of the call to mainbody.
Do other people recognize this pattern? Other solutions?
I'd love to hear.
Douwe Osinga http://douweosinga.com
_______________________________________________ Zope maillist - Zope@zope.org http://mail.zope.org/mailman/listinfo/zope ** No cross posts or HTML encoding! ** (Related lists - http://mail.zope.org/mailman/listinfo/zope-announce http://mail.zope.org/mailman/listinfo/zope-dev )
On Thu, Jan 15, 2004 at 01:52:22PM +0100, Douwe Osinga wrote:
Hi Zopers,
I have a musing about how Zope sites are set-up which I would like to share and would very much like comments on.
Typically, a web page has a header, a body and a footer. In most Zope instances, this is translated into standard_html_header, standard_html_footer, while an index_html calls these and does something interesting in between.
or we use page templates and have a main_template instead of standard_html_*...
I don't like this setup too much, because it explicitly repeats the calling of standard_html_* for every page.
I don't see how this is true given your structure below.
If you want to add a parameter to for example standard_html_header (say the css to use or similar), you'll have to change all pages (yes, you can use default parameters, but that is not the point). So I usually end up with a structure like this:
root index_html standard_html_footer standard_html_header mainbody folder1 mainbody folder2 mainbody
where index_html calls header, mainbody and footer. This works great, except for that you need a folder for any page that needs to display the structure.
What's wrong with that? It's a pretty sensible way to do it IMHO. Admittedly, cataloging is a bit of a pain because you want to catalog each folder's mainbody but have the search result point you to the folder, not to mainbody.
If you want add a method to the root folder that is applicable to everypage, then this method needs to call the structure again.
What does "call the structure" mean?
Do other people recognize this pattern? Other solutions?
Write your own content type as a Product. At a certain point it's easier than maintaining a pile of scripts & methods. -- Paul Winkler http://www.slinkp.com Look! Up in the sky! It's KABUKI WARRIOR DESTRUCTO-FINALE! (random hero from isometric.spaceninja.com)
Typically, a web page has a header, a body and a footer. In most Zope instances, this is translated into standard_html_header, standard_html_footer, while an index_html calls these and does something interesting in between.
or we use page templates and have a main_template instead of standard_html_*... Yes, but that doesn't make much of a difference in the end.
I don't like this setup too much, because it explicitly repeats the calling of standard_html_* for every page.
I don't see how this is true given your structure below. Not in the structure I describe below. More if I add a login method somewhere and I want to call the login method directly, the login method has to take care of rendering itself including the template in whatever form correctly.
What's wrong with that? It's a pretty sensible way to do it IMHO. Admittedly, cataloging is a bit of a pain because you want to catalog each folder's mainbody but have the search result point you to the folder, not to mainbody. Nothing really wrong, just that you end up with a lot of folders and some folders aren't really folders and just contain a mainbody script. It's odd and doesn't feel too good.
If you want add a method to the root folder that is applicable to everypage, then this method needs to call the structure again.
What does "call the structure" mean? Well, let's say I add a search method in the root, which would allow searching in subfolders or something. Then I either end up with complicated acquisition and context issues, or I just put a script in the root that calls the template or standard_html* stuff.
Do other people recognize this pattern? Other solutions?
Write your own content type as a Product. At a certain point it's easier than maintaining a pile of scripts & methods. Yes, I do that usually. I just wondered what other people thought.
Regards, Douwe Osinga http://douweosinga.com
participants (9)
-
Ausum Studio -
Bjorn Stabell -
Christian Robottom Reis -
Dieter Maurer -
Douwe Osinga -
Dylan Reinhardt -
Mark McEahern -
Paul Winkler -
Toby Dickenson