[ZODB-Dev] Large ZODB packing and progress

[Shane Hathaway]

On 02/25/2011 04:49 AM, Jim Fulton wrote:
> On Fri, Feb 25, 2011 at 5:44 AM, Martijn Pieters<mj at zopatista.com>  wrote:
>> Last night we used our two-phase pack to start packing the largest
>> Oracle RelStorage ZODB we run. Some statistics first:
>>
>> * Never packed before in it's 2 year existence.
>> * Has more than 4.5 million transactions, 52 million object states.
>> * Packing to 60 days ago means we'll loose 4 million transactions, and
>> 41.5 million states.
>>
>> Packing ran through the night, but I had to abort the pack this
>> morning as I am still skittish about holding commit locks for packing
>> during normal operations.
>
> I don't know about relstorage, but for FileStorage, commit locks are
> only held in the very last stage of packing after the vast majority of
> the work is done.  Even then, the commit lock is released periodically
> to let new transactions proceed.

Whereas FileStorage packs by creating a new copy of the database, 
RelStorage deletes individual rows.  Unfortunately, SQL delete 
statements tend to be surprisingly expensive.

To get closer to what FileStorage does, RelStorage could theoretically 
copy rows into new tables instead of deleting rows.  I've pondered doing 
that, but it would take some major work, I think.

Shane