[ZODB-Dev] PGStorage

[Jim Fulton]

On Jan 23, 2008, at 9:59 AM, Benji York wrote:

> Flavio Coelho wrote:
>> Actually what I am trying to run away from is the "packing  
>> monster" ;-)
>
> Jim has done a great deal of work on packing (that will go into 3.9  
> I presume)

and is available now in zc.FileStorage.

> that should make your pack 3 to 6 times faster (depending on if you  
> do garbage collection at pack time or not).

And consumes twice as much memory, depending on your settings.

For testing, I used a 20G database containing catalog data whos size  
was cut in half by packing.  It used around 900MB of memory for  
packing with garbage collection. :(  Without GC, it used much less.

Another major benefit of my new packing code is that it does most of  
the work in a separate process, which allows it to take advantage of  
multiple processors.

>> I want to be able to use an OO database without the inconvenience  
>> of having
>> it growing out of control and then having to spend hours packing the
>> database every once in a while. (I do a lot of writes in my DBs).  
>> Do this
>> Holy grail of databases exist? :-)
>
> Why not put the pack in cron?

My new packing code helps a lot, but packing is still very disruptive.

IMO, something that packed incrementally, with disk being freed along  
the way, would be a big improvement. This isn't possible with  
FileStorage.

Jim

--
Jim Fulton
Zope Corporation