[ZODB-Dev] Making ZODB / ZEO faster

[Erik Dahl]

Guys,

Thanks for all the great feedback.  Still processing it but here are  
somethings we will try.

RelStorage - in our app context to see if there it helps / hurts. will  
report back results.  Quick tests show some improvement.  We will also  
look at tuning out current ZEO setup.  Last time I looked there was  
only the invalidation queue.  I poked around a bit for tuning docs and  
didn't see any.  Can someone point me to them?

zc.catalogqueue - we are adding fullish indexing to our objects (they  
aren't text pages so not truly a "full" index).  hopefully moving  
indexing out to a separate process will keep the impact of the new  
index low and help with our current conflict issues.

Our slow loading object was a persistent with a regular list inside of  
the main pickle.  Objects that the list pointed to were persistent  
which I believe means that hey will load separately.  In general we  
have tried to make our persistent objects reasonably large to lower  
the amount of load round trips.  I haven't actually checked its size  
yet but it will be interesting to see.

-EAD



On Dec 4, 2009, at 6:21 PM, Shane Hathaway wrote:

> Jim Fulton wrote:
>> I find this a bit confusing.  For the warm numbers, It looks like  
>> ZEO didn't
>> utilize a persistent cache, which explains why the ZEO numbers are  
>> the
>> same for hot and cold. Is that right?
>
> Yes.  It is currently difficult to set up ZEO caches, which I  
> consider an issue with this early version of zodbshootout.   
> zodbshootout does include a sample test configuration that turns on  
> a ZEO cache, but it's not possible to run that configuration with a  
> concurrency level greater than 1.
>
>> What poll interval are you using for relstorage in the tests?
>> Assuming an application gets reasonable cache hit rates, I don't  
>> see any
>> meaningful difference between ZEO and relstorage in these numbers.
>
> You are entitled to your opinion. :-)  Personally, I have observed a  
> huge improvement for many operations.
>
>>>> Second, does the test still write and then read roughly the same
>>>> amount of data as before?
>>> That is a command line option.  The chart on the web page shows  
>>> reading and
>>> writing 1000 small persistent objects per transaction,
>> Which is why I consider this benchmark largely moot. The database is
>> small enough
>> to fit in the servers disk cache.  Even the slowest access times are
>> on the order of .5
>> milliseconds. Disk accesses are typically measured in 10s of
>> milliseconds.  With magnetic
>> disks, for databases substantially larger than the server's ram, the
>> network component
>> of loading objects will be noise compared to the disk access.
>
> That's why I think solid state disk is already a major win,  
> economically, for large ZODB setups.  The FusionIO cards in  
> particular are likely to be at least as reliable as any disk.  It's  
> time to change the way we think about seek time.
>
> Shane
>