[Zope] Problems with *BIG* SQL queries

[Mayers, Philip J]

I've got some problems performing and returning some large SQL queries in
our Zope app. The queries are reasonably quick if done straight to the SQL
server, and return about 3000 to 5000 rows - the resulting HTML (generated
with a dtml-in over the results) is about 600-900kbytes.

But, the queries seem to take a *long* time - Zope is spinning at about 60%
CPU in the thread responsible for sending the HTTP request back. Why? I've
just timed it at 1:30 for 3914 rows and 586kb of HTML.

The *really* odd thing is that using "test" on the same SQL query from the
management interface, it takes about 10 seconds. Now, admittedly it only
shows 50 rows, but surely it can't be the batching?

And no, I don't want to batch the results up. Any ideas? Is it just the dtml
processing that's slowing it down? If so, how can I speed it up
(PythonScript to generate the HTML?)

Any advice appreciated...

Regards,
Phil

+----------------------------------+
| Phil Mayers, Network Support     |
| Centre for Computing Services    |
| Imperial College                 |
+----------------------------------+