[ZODB-Dev] zeopack requirements?

[Jim Fulton]

On Dec 17, 2007, at 9:39 AM, Alan Runyan wrote:

> Makes sense.
>
> Is zc.FileStorage an alternative packing strategy?

Yes. I expect to roll this work into 3.9.  It is a separate package  
now so I can deploy it with 3.8.  It is very much a work in progress.


>  Does it use less memory?


No, it uses quite a bit more. :)

Memory is pretty cheap.  The current packing approach beats the $#@!  
out of the disk.  This new approach tries to be much kinder by storing  
cross-reference data in memory rather than massively seeking to read  
cross-reference data from disk.  Also, on linus systems, it uses  
posix_fadvise to avoid trashing the disk cache.

It 2-3 times as fast as the old pack algorithm and I think it will be  
much kinder to the server and other processes on the system.

It does most of the work in a sub-process so it can take better  
advantage of multiple CPUs and, although it uses more memory, it will  
to a much better job of returning the memory to the OS.

I'm not aware that reducing memory is a goal.  If it is a goal, we can  
look at having the packing code actually use a database for the data  
it needs.  This would slow down packing quite a bit, but could limit  
the amount of memory used.

Another thing I looked at was adding an option to skip GC.  This seems  
to speed packing and reduce memory usage quite a bit.  I plan to add  
this as an database configuration option.  I'd like to be able to  
decide whether to do GC at pack time.  The pack API doesn't support  
this.  I think I'll eventually resort to using the extension mechanism  
to add additional pack methods. :)

Jim

--
Jim Fulton
Zope Corporation