[ZODB-Dev] Speeding up ZODB (was "redis cache for RelStorage")

[Matthias]

Am 06.05.2011, 17:12 Uhr, schrieb Paul Winkler <slinkp at gmail.com>:

> On Fri, May 06, 2011 at 12:19:39PM +0200, Pedro Ferreira wrote:
>> A quick check with nethogs shows values of network usage oscillating
>> between 100 and 200 KB/s. But I guess that if I were loading an
>> excessive amount of data, this value would be higher, no?
>
> So throughput isn't the problem.
>
> I don't think you've mentioned what the latency is like between your
> ZEO server and clients.   What's the ping time look like?
>
> You mentioned pages that load lots of objects (200 objects each
> referring to a "handful" of sub-objects).  If they're not in the
> client cache, each of those will require a separate fetch from the ZEO
> server.
>
> As an example, let's say your average object size is only 1k, but due
> to network latency it takes 5ms per fetch.  Let's say your slow page
> loads 1000 objects, and 100 of them are not in the cache. That's only
> 100k of data, but you've spent 500ms waiting on the network.

It would be cool if you could give a hint to ZEO somehow to prefetch a  
certain set of objects along with their subobjects and then return  
everything in one batch. This way you avoid all the round-trips when you  
discover you want to retrieve a suboject.

-Matthias