[ZODB-Dev] RelStorage 1.5.0b1 dry-run > two phase pack, better pack lock behaviour

[Maurits van Rees]

Op 22-02-11 17:25, Martijn Pieters schreef:
> Hi,
>
> I was already investigating the possibility to split the RelStorage
> packing process up into smaller chunks.
>
> Due to the expected load on the Oracle cluster during a pack, we'll
> have to run the pack at night and want to be absolutely certain that
> database is ready for normal site operations again the next day. With
> a 40+GB database (hasn't been packed for it's entire run, more than 2
> years now) we are not confident packing will be done in one night.

I wonder it it may help to set pack-gc to false during the first pack. 
According to the docs this is faster, though it of course leaves more 
unused objects behind.  Set pack-gc to the default true value for 
subsequent packs.  Theoretically this should make sure that the first 
pack will finish on time and leave an already smaller database; after 
the second pack the database is at its smallest.

> To at least get a handle on how much work the packing is going to be,
> and to have a nice stopping point, I looked at splitting pre-pack and
> pack operations out into two separate steps. To my delight I saw that
> the 1.5.0 beta already implements basically running only the pre-pack
> phase (the --dry-run option). From there I created the attached patch,
> one that renames the dry-run op into a 'prepack only' option, and adds
> another option to skip the pre-pack and just use whatever is present
> in the pack tables.
>
> I haven't yet actually run this code, but the change isn't big. I
> didn't find any relevant tests to update. Anyone want to venture some
> feedback?

Makes sense to me.  The patch looks good too, though I have not tried it.


-- 
Maurits van Rees
Web App Programmer at Zest Software: http://zestsoftware.nl
Personal website: http://maurits.vanrees.org/