cpu load 99 percent a lot in plone evinironment
We are running a seperate zope instance that contains a number of plone portal. Zope version 2.8.2 plone version 2.1.1 python 2.3.5 since a few weeks the instance 'hangs', When i check the processes usually python is at 99%, but i've also had time where zope was not responding but the server load was almost zero (deadlocking ?). I've installed deadlockdebugger and there are some entries of 'slow' pages in there, but i've checked these pages and they do not contain any 'big' queries or possible slow processes. The page loops over a results set from mysql, this set contains 400 - 500 results. I've noticed that if it loops over 500 results instead of i.e. 20 the page takes almost 3 to 4 seconds longer to load. I've never seen this kind of behaviour before (i have seen it with bad queries and such, but thats not the case here). Will a zope update possibly solve this problem, or can someone inform me about the deadlockdebugger output and HOW is should read it? Obiplein 6-10 NL-1094 RB T +31 (0)20 462 4910 F +31 (0)20 462 4919 M +31 (0)62 666 8489 E marko@mm-pro.nl w w w . m u l t i m e d i a - p r o f e s s i o n a l s . n l
Hi Marko, Let me know databse size of your plone environment(server) . Marko Kruijer <marko@mm-pro.nl> Sent by: zope-bounces@zope.org 02/05/2007 06:50 PM To zope@zope.org cc Subject [Zope] cpu load 99 percent a lot in plone evinironment We are running a seperate zope instance that contains a number of plone portal. Zope version 2.8.2 plone version 2.1.1 python 2.3.5 since a few weeks the instance 'hangs', When i check the processes usually python is at 99%, but i've also had time where zope was not responding but the server load was almost zero (deadlocking ?). I've installed deadlockdebugger and there are some entries of 'slow' pages in there, but i've checked these pages and they do not contain any 'big' queries or possible slow processes. The page loops over a results set from mysql, this set contains 400 - 500 results. I've noticed that if it loops over 500 results instead of i.e. 20 the page takes almost 3 to 4 seconds longer to load. I've never seen this kind of behaviour before (i have seen it with bad queries and such, but thats not the case here). Will a zope update possibly solve this problem, or can someone inform me about the deadlockdebugger output and HOW is should read it? Obiplein 6-10 NL-1094 RB T +31 (0)20 462 4910 F +31 (0)20 462 4919 M +31 (0)62 666 8489 E marko@mm-pro.nl w w w . m u l t i m e d i a - p r o f e s s i o n a l s . n l _______________________________________________ Zope maillist - Zope@zope.org http://mail.zope.org/mailman/listinfo/zope ** No cross posts or HTML encoding! ** (Related lists - http://mail.zope.org/mailman/listinfo/zope-announce http://mail.zope.org/mailman/listinfo/zope-dev ) +=========================================================+ This message may contain confidential and/or privileged information. If you are not the addressee or authorized to receive this for the addressee, you must not use, copy, disclose or take any action based on this message or any information herein. If you have received this message in error, please advise the sender immediately by reply e-mail and delete this message. Thank you for your cooperation. +=========================================================+
Marko Kruijer wrote at 2007-2-5 14:20 +0100:
... The page loops over a results set from mysql, this set contains 400 - 500 results. I've noticed that if it loops over 500 results instead of i.e. 20 the page takes almost 3 to 4 seconds longer to load.
I've never seen this kind of behaviour before (i have seen it with bad queries and such, but thats not the case here).
You probably should profile your long running request to find out where the time is spent. -- Dieter
-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1 On 9 Feb 2007, at 09:29, Marko Kruijer wrote:
I do notice that my suspected page is slow, but that's a page template, and there is not very much out of the ordinary going on there. Just a loop in tal over a sql result set.
"Just a loop in tal over a sql result set" can be extremely slow, depending on the size of your result set. You may not realize that every time you touch one of those resultset record objects the security machinery gets in the game to ensure the executing user is allowed to see the rendered result. This is true for every single field you're trying to show on the page. Workarounds I have used in this situation (eons ago) involved retrieving the resultset in trusted code (inside a filesystem product or an External Method) and converting it to a sequence of simple types, like a sequence of dictionaries, before handing it back to the page template. Those simple types don't cause security assertions and the rendering is sped up immensely. jens -----BEGIN PGP SIGNATURE----- Version: GnuPG v1.4.5 (Darwin) iD8DBQFFzDM/RAx5nvEhZLIRAuXpAJ9hm/EDDx57u/0naxs/U0DZvBsK4ACgrShI WT5pj0hbBlSf3g1mmXQR5ck= =8n7A -----END PGP SIGNATURE-----
Jens Vagelpohl wrote:
On 9 Feb 2007, at 09:29, Marko Kruijer wrote:
I do notice that my suspected page is slow, but that's a page template, and there is not very much out of the ordinary going on there. Just a loop in tal over a sql result set.
"Just a loop in tal over a sql result set" can be extremely slow, depending on the size of your result set. You may not realize that every time you touch one of those resultset record objects the security machinery gets in the game to ensure the executing user is allowed to see the rendered result. This is true for every single field you're trying to show on the page.
I've been having similar speed problems with a page that loops through a lot of SQL records. From what I can tell from the profiler, it is the security machinery that is at fault.
Workarounds I have used in this situation (eons ago) involved retrieving the resultset in trusted code (inside a filesystem product or an External Method) and converting it to a sequence of simple types, like a sequence of dictionaries, before handing it back to the page template. Those simple types don't cause security assertions and the rendering is sped up immensely.
There seems to be a built in method - .dictionaries() that does exactly this. Unfortunatley, neither it, nor a simple external method I wrote to call my ZSQL methods seemed to help me. Robert (Jamie) Munro
Hi Robert (Jamie?), Robert (Jamie) Munro wrote:
I've been having similar speed problems with a page that loops through a lot of SQL records. From what I can tell from the profiler, it is the security machinery that is at fault.
I'm doing the same sort of stuff, and routinely have the same sorts of problems. Two other things that I've done that help are: - - Use the database to handle pagination, such that a single page of data is returned - Plone/Zope batching can be very slow with huge data sets (which to be fair it was never designed for). This is easy with PostgreSQL and there is a work around for SQL Server. - Instead of simply embedding my dataset (sometimes multiple datasets) in the page, use AHAH to pull in a panel of data into the existing page as a pure Zope transaction. This makes the UI a lot more fluid by avoiding multiple redundant Plone page loads (e.g. during pagination, or when doing drill-down type navigation). AHAH does not seem to be much discussed here - it's worth reading: - http://microformats.org/wiki/rest/ahah & http://www.gizax.it/ahahsection/ (I have based my work on the script in the second link) Short version: uses AJAX-like technique of rewriting the HTML using JavaScript, but returns pure HTML as opposed to XML or JSON, so trivial to implement, and can degrade gracefully. HTH -- Regards, PhilK 'let's hear it for the vague blur'
Philip Kilner wrote:
Hi Robert (Jamie?),
Most people call me Jamie, but Robert is fine, and it's what it says on all my official documents.
AHAH does not seem to be much discussed here - it's worth reading: -
http://microformats.org/wiki/rest/ahah & http://www.gizax.it/ahahsection/
(I have based my work on the script in the second link)
Short version: uses AJAX-like technique of rewriting the HTML using JavaScript, but returns pure HTML as opposed to XML or JSON, so trivial to implement, and can degrade gracefully.
Very useful. I've based my system on the first link because I didn't need to use POSTs, I just add ?foo=bar to the end of the URL and get it as normal. I added the ability to load a list of urls and element ids one after the other, then only activate the submit button at the bottom of the page when the whole form was loaded. This means the top of the page loads quickly, and the rest loads section by section before people scroll down to it. I was thinking I would have to do something like this, and your links made it really easy. Robert (Jamie) Munro
Hi Jamie, Robert (Jamie) Munro wrote:
Very useful. I've based my system on the first link because I didn't need to use POSTs, I just add ?foo=bar to the end of the URL and get it as normal. I added the ability to load a list of urls and element ids one after the other, then only activate the submit button at the bottom of the page when the whole form was loaded. This means the top of the page loads quickly, and the rest loads section by section before people scroll down to it.
I was thinking I would have to do something like this, and your links made it really easy.
I'm really pleased to hear it. I came across AHAH as I was investigating AJAX. I'd been dreading using AJAX, partly because it would mean a good deal of reworking of existing applications, partly because of the learning curve. For my applications, AHAH has turned out to be a better solution, and it's a bonus that it is so easy to implement. -- Regards, PhilK 'let's hear it for the vague blur'
participants (6)
-
Dieter Maurer -
Jens Vagelpohl -
Marko Kruijer -
Philip Kilner -
Robert (Jamie) Munro -
Sunil_Sagar＠external.mckinsey.com