[ZODB-Dev] Pack problem

[Jim Fulton]

On Mon, Jun 30, 2014 at 4:40 AM, Alessandro Pisa
<alessandro.pisa at gmail.com> wrote:
> On 30 June 2014 10:32, Frédéric Iterbeke <Frederic.Iterbeke at ugent.be> wrote:
>> Op 30/06/2014 9:30, Alessandro Pisa schreef:
>>
>>> I have a ~70Gb Data.fs that does not pack anymore.
>>> When I pack it it creates a ~8GB Data.fs.pack, then it evaluates thi
>>> condition:
>>>
>>> -https://github.com/zopefoundation/ZODB/blob/3.9.5/src/ZODB/FileStorage/fspack.py#L410
>>> as True, removes, the Data.fs.pack and returns.
>>>
>>> The same happens with ZODB-3.10.5 and using pack-gc = false.
>>> In every permutation I tried the produced Data.fs.pack files have the
>>> same checksum.
>>>
>>> Does anybody have some hints?
>>>
>>> What I am trying to do:
>>>   - comment the Data.fs.pack removing.
>>>   - use that file as a new Data.fs
>>
>> Well, 70 Gb Data.fs is pretty big. But not impossible afaik.
>>
>> If you set pack-gc = false it's normal that nothing is removed.
>>
>
> Setting or unsetting it doesn't change the produced Data.fs.pack (it
> has the same md5sum).
> Anyway the pack time considerably reduces (from 4 hours to 25 minutes).

That's because the GC (and even pack algorithm) built into FileStorage
is very inefficient.

When you pack a file storage with GC enabled, you are really doing 2
things:

1. Removing non-current database records (as of the pack time).
    This is properly called "packing".

2. Removing objects that are no longer reachable (from any records,
    current or non-current) from the root object.  If packing doesn't
    remove any objects, neither will packing.

Jim

-- 
Jim Fulton
http://www.linkedin.com/in/jimfulton