[Zope-Annce] ZODB 3.1.2 beta 1 released!

[Jeremy Hylton]

I'm pleased to announce a beta release of ZODB 3.1.2 and ZEO 2.0.2.  The
are three major bug fixes in this release:

- A new pack() implementation for FileStorage that fixes recently
  discovered data corruption problems.
- Atomic processing of invalidations to guarantee data consistency.
- Fix for Missing.Value to prevent segfaults during comparison.

This release corresponds exactly to Zope 2.6.2 beta 1, which will be
released in the near future.

The release is available in the usual place
http://www.zope.org/Products/ZODB3.2
It has been tested with Python 2.1.3, 2.2.2, and 2.3b1 on Windows and
Linux.  Binary installers are available for Windows for 2.1.3 and 2.2.2.

I am particularly eager to get feedback on the new pack implementation. 
It is a substantial rewrite of the garbage collector.  It fixes several
reported bugs, but I'd like to see substantial user testing to make sure
we didn't add too many news ones.

I've included the complete news file below.

Jeremy

What's new in ZODB3 3.1.2 beta 1?
=================================

ZODB
----

Invalidations are now processed atomically.  Each transaction will see
all the changes caused by an earlier transaction or none of them.
Before this patch, it was possible for a transaction to see invalid
data because it saw only a subset of the invalidations.  This is the
most likely cause of reported BTrees corruption, where keys were
stored in the wrong bucket.  When a BTree bucket splits, the bucket
and the bucket's parent are both modified.  If a transaction sees the
invalidation for the bucket but not the parent, the BTree in memory
will be internally inconsistent and keys can be put in the wrong
bucket.  The atomic invalidation fix prevents this problem.

A number of minor reference count bugs in the object cache were
fixed.  That's the cPickleCache.c file.

It was possible for a transaction that failed in tpc_finish() to lose
the traceback that caused the failure.  The transaction code was fixed
to report the original error as well as any errors that occur while
trying to recover from the original error.

ZEO
---

A ZEO client will not read from its cache during cache verification.
This fix was necessary to prevent the client from reading inconsistent
data.

The isReadOnly() method of a ZEO client was fixed to return the false
when the client is connected to a read-only fallback server.

The sync() method of ClientStorage and the pending() method of a zrpc
connection now do both input and output.

The short_repr() function used to generate log messages was fixed so
that it does not blow up creating a repr of very long tuples.

Storages
--------

FileStorage has a new pack() implementation that fixes several
reported problems that could lead to data loss.

Two small bugs were fixed in DemoStorage.  undoLog() did not handle
its arguments correctly and pack() could accidentally delete objects
created in versions.

Fixed trivial bug in fsrecover that prevented it from working at all.

FileStorage will use fsync() on Windows starting with Python 2.2.3.

FileStorage's commit version was fixed.  It used to stop after the
first object, leaving all the other objects in the version.

BTrees
------

Trying to store an object of a non-integer type into an IIBTree
or OIBTree could leave the bucket in a variety of insane states.  For
example, trying

    b[obj] = "I'm a string, not an integer"

where b is an OIBTree.  This manifested as a refcount leak in the test
suite, but could have been much worse (most likely in real life is that
a seemingly arbitrary existing key would "go missing").

When deleting the first child of a BTree node with more than one
child, a reference to the second child leaked.  This could cause
the entire bucket chain to leak (not be collected as garbage
despite not being referenced anymore).

Other minor BTree leak scenarios were also fixed.

Other
-----

Comparing a Missing.Value object to a C type that provide its own
comparison operation could lead to a segfault when the Missing.Value
was on the right-hand side of the comparison operator.  The Missing
class was fixed so that its coercion and comparison operations are
safe. 

Tools
-----

Four tools are now installed by setup.py: fsdump.py, fstest.py,
repozo.py, and zeopack.py.