[Grok-dev] Re: Scalability of annotations?

[Martijn Faassen]

Wichert Akkerman wrote:
> Previously Sebastian Ware wrote:
>> Merry Christmas... no snow in Stockholm, but there are plenty of  
>> Christmas trees :)
>>
>> Anyway, I am wondering a bit about the scalability of annotations. If  
>> I use annotations as an edit history, will I run into scalability  
>> problems after say 100k edits? Say that each edit requires 5kb of data.
> 
> If memory serves me correctly annotations are lazily loaded btrees, so
> they should scale quite well. 

I just examined the source and that's correct; keys and values are 
stored in BTrees. I don't think BTrees are very relevant here though, as 
Grok's Annotation base class uses annotation factories, and this 
generates a single key on the fly.

That said, an annotation object is a persistent object like any others 
in the ZODB. That means that changing it shouldn't normally conflict 
with other objects. You're free to use advanced data structures like 
BTrees inside of a particular annotation to store edit history objects 
as independently persistent objects, to further avoid conflicts. (these 
BTrees are unrelated to the BTrees inside the annotation mechanism; it'd 
just be a way you could construct the application)

> There is of course no substitute for a benchmark to test your specific scenario.

Fully agreed.

Regards,

Martijn