[ZODB-Dev] New Berkeley based storages available

[Barry A. Warsaw]

Hi all,

Well at long last, I have new rewrites of the Berkeley storages
available as a beta release (2.0beta1).  Hopefully these versions will
pan out better than the earlier versions.  I plan to include these
with ZODB 3.2 when it's released, but I doubt it's worth including the
old versions of the storages, since they had so many problems.  Note
that the table schemas for these storages are different, so you'd need
to use copyTransactionsFrom() to upgrade.

You can download the distutils package from SourceForge:

    http://sourceforge.net/projects/zodb

see also the Berkeley storage Wiki for more information:
    
    http://www.zope.org/Wikis/ZODB/BerkeleyStorage

As before, there are both Full and Minimal storages.  Full.py provides
support for undo and versions, but requires a pack operation to remove
old object revisions and cyclic garbage.  Minimal.py keeps only
current revisions of objects, but still needs to be packed to remove
cycles.  Both storages implement reference counting garbage
collection, and both storages support a new autopacking strategy (see
below) that should eliminate most needs for explicit packing.

A quick overview of what's new:

- No more lock exhaustion problems.  By substituting application level
  locks for the BerkeleyDB locking subsystem, we've been able to avoid
  the static lock table exhaustion problem.  Briefly, because ZODB
  transactions can touch a potentially unlimited number of objects,
  and because of the locking semantics for BerkeleyDB BTrees, it will
  always be possible to run out of locks.  But because access to the
  storages is serialized through application locks anyway, we really
  don't need the BerkeleyDB locks, so we can shut off that subsystem.

  Note that this isn't (yet) an officially supported configuration for
  BerkeleyDB, but from what we understand, Sleepycat has at least one
  large customer using it this way, without problem, so we think we're
  pretty safe this way.  I'd keep an eye out for any problems here
  though, until there's been more use.

- No more temporary commit log files.  We've switched to an optimistic
  write strategy for updates to the database, so we don't need the
  temporary commit log files anymore.  This should improve performance
  and disk i/o.

- No more huge memory growth on packing.  All bookkeeping for pack now
  uses BerkeleyDB BTrees where possible, so this should make memory
  usage during packing much more manageable.

- Recoverability.  If the storage process crashes during the two phase
  commit, the storages should be able to recover the data and either
  abort or commit the transaction that crashed, as appropriate.

- Autopacking.  Full needs to be packed occasionally to get rid of old
  object revisions and cyclic garbage.  Minimal needs to be packed
  occasionally to get rid of cyclic garbage (it has no old object
  revisions).  Both storages expose an explicit pack() method, but
  they both also implement autopacking in a background thread.

  For Full, autopacking only removes old object revisions, but you can
  configure it to do a full gc'ing "classic" pack every nth autopack.
  You also configure how often autopack should run and how far in the
  past it should pack to.  For example, you could set up autopack to
  run every hour, packing away any object revisions older than 6
  hours, and doing a full gc'ing pack once per day.

  Minimal also supports a background autopack thread, but it only does
  gc'ing autopacks.  You configure how often this thread should wake
  up and pack the storage.

- Configurability.  Both storages support some easier configurability
  by passing in a `config' object to their constructors.  Through this
  you can configure such things as the BerkeleyDB cache size and log
  directory, the autopacking strategy, and the checkpoint frequency
  (more checkpoints means faster BerkeleyDB recovery if necessary).

These storages should be implementation complete, and they pass all
the ZODB unit tests.  I will be doing more testing of the storages
with Zope, and I am interesting in any feedback you might provide.
They should work just fine with ZODB 3.1, ZEO 2, and any recent Zope
version.

Enjoy,
-Barry