On Thu, Apr 08, 2004 at 08:02:05PM -0400, Shane Hathaway wrote:
On 04/08/04 12:16, Paul Winkler wrote:
On Wed, Mar 24, 2004 at 01:32:18PM -0500, Shane Hathaway wrote:
Jeremy has suggested that object pre-fetching could be added to ZODB.
This is much on my mind currently. Any thoughts on what an API for pre-fetching might look like?
Well, thinking about it some more, it seems like it might be just as easy to run a special thread just for fetching blobs. One queue should tell the blob thread what to get, and another queue should return the results. I think that would be a nice design.
ok... i'll mull that over.
The use case that most concerns me is: If you have an Image or File object with a very long Pdata chain, you're likely to A) thrash the ZEO client cache if it's not big enough, and B) spend a looooong time waiting for all the objects in the chain to load. At least, this is my theory on what's slowing things down - I will be trying to verify this today. See the bottom of this page: http://www.slinkp.com/code/zopestuff/blobnotes
In fact, Zope puts large files (the threshold is around 256K - 512K) into a temporary file before serving them, to free up application threads. It's a tremendous handicap.
If you're referring to the code in ZServerHTTPResponse.write: AFAICT, since Image.File calls RESPONSE.write() for each Pdata in its chain, we'll actually get a temporary file for each Pdata rather than one big one for the whole file. There may be (probably are) performance issues in there, but right now I'd be really happy to get ClientStorage a bit closer to the speed of FileStorage for this Pdata gunk. -- Paul Winkler http://www.slinkp.com