[Zodb-checkins] CVS: ZODB3/bsddb3Storage - README.txt:1.2 MANIFEST:NONE README:NONE
Barry Warsaw
barry@wooz.org
Tue, 5 Nov 2002 18:08:02 -0500
Update of /cvs-repository/ZODB3/bsddb3Storage
In directory cvs.zope.org:/tmp/cvs-serv1143/bsddb3Storage
Added Files:
README.txt
Removed Files:
MANIFEST README
Log Message:
Merging the Berkeley storage's bdb-nolocks branch back into the trunk
for ZODB 3.2.
=== ZODB3/bsddb3Storage/README.txt 1.1 => 1.2 ===
--- /dev/null Tue Nov 5 18:08:02 2002
+++ ZODB3/bsddb3Storage/README.txt Tue Nov 5 18:07:31 2002
@@ -0,0 +1,404 @@
+BerkeleyDB Storages for ZODB
+============================
+
+Please see the LICENSE.txt file for terms and conditions.
+
+
+Introduction
+------------
+
+This package contains implementations for ZODB storages based on
+Sleepycat Software's BerkeleyDB and the PyBSDDB3 extension module.
+These storages save ZODB data to a bunch of BerkeleyDB tables, relying
+on Berkeley's transaction machinery to provide reliability and
+recoverability.
+
+Note that the Berkeley based storages are not "set and forget". The
+underlying Berkeley database technology requires maintenance, careful
+system resource planning, and tuning for performance. You should have
+a good working familiarity with Berkeley DB in general before trying
+to use these storages in a production environment. It's a good idea
+to read Sleepycat's own documentation, available at
+
+ http://www.sleepycat.com
+
+See also our operating notes below.
+
+
+Contents
+--------
+
+Inside the bsddb3Storage package, there are four storage
+implementations:
+
+- Full.py is a complete storage implementation, supporting
+ transactional undo, versions, application level conflict
+ resolution, and automatic reference counting garbage collection.
+ Packing this storage is only required in order to get rid of old
+ object revisions. Full storage provides some experimental
+ support for cyclic garbage collection.
+
+- Minimal.py is an implementation of an undo-less, version-less
+ storage, which implements a reference counting garbage
+ collection strategy to remove unused objects. It is still
+ possible for garbage objects to persist in the face of object
+ cycles, although a future release will integrate a cyclic
+ garbage detector.
+
+- Packless.py is another, older implementation of an undo-less,
+ version-less, reference counting storage that also obviates the
+ need for packing, except in the presence of cyclic garbage.
+ Packless has two limitations which may make it less desirable
+ than Minimal.py (which will eventually replace Packless):
+
+ * Packless uses its own temporary commit log file, which can
+ cause more disk I/O than Minimal.py
+
+ * Packless relies on the BerkeleyDB locking subsystem, so for
+ very large transactions, you may run out of Berkeley locks.
+
+- Autopack.py is a new implementation of an undo-less,
+ version-less storage that attempts to maximize write performance
+ through the use of optimistic updates and sequential record
+ appends. Autopack is also a prototype for an automatically
+ packing storage, making it truly packless even for cyclic
+ garbage collection.
+
+
+Compatibility
+-------------
+
+As of this writing (23-Aug-2002) it is recommended that at least
+Python 2.1.3 or Python 2.2.1 be used with these storages. Full,
+Minimal, Autopack storages should also work with Python 2.3 (as yet
+unreleased).
+
+Some testing has been conducted with both Zope 2.5.1, Zope 2.6, and
+the Zope 3 code base. These storages have primarily been tested on
+Linux.
+
+It is recommended that you use at least BerkeleyDB 4.0.14 and PyBSDDB
+3.4.1. Earlier versions of both packages had bugs that could crash or
+hang your application.
+
+
+Requirements
+------------
+
+You must install Sleepycat BerkeleyDB and Robin Dunn's PyBSDDB package
+separately.
+
+To obtain the latest source release of BerkeleyDB, see the Sleepycat
+Software site
+
+ http://www.sleepycat.com
+
+To obtain the latest source release of Robin Dunn's PyBSDDB package,
+see
+
+ http://pybsddb.sourceforge.net
+
+Install both BerkeleyDB and PyBSDDB as per the instructions that come
+with those pacakges. For BerkeleyDB, it's generally wise to accept
+the default configure options and do a "make install" as root. This
+will install BerkeleyDB in /usr/local/BerkeleyDB.4.0
+
+Note that because Berkeley installs itself in a non-standard location,
+the dynamic linker ld.so may not be able to find it. This could
+result in link errors during application startup. For systems that
+support ld.so.conf, it is highly recommended that you add
+/usr/local/BerkeleyDB.4.0/lib to that file and run ldconfig. An
+alternative approach is given below.
+
+PyBSDDB comes with a standard distutils-based setup script which will
+do the right thing.
+
+If you've extended your ld.so.conf file as above, you can build
+PyBSDDB like so::
+
+ % python setup.py build_ext -i
+
+Otherwise, here's the build command I've used with some success for
+the PyBSDDB distribution::
+
+ % python setup.py build_ext -i --berkeley-db=/usr/local/BerkeleyDB.4.4/ --lflags="-Xlinker -rpath -Xlinker /usr/local/BerkeleyDB.4.4/lib"
+
+Then install the package like so::
+
+ % python setup.py install
+
+When you can run the tests which ship with PyBSDDB, you'll know you've
+been successful at both BerkeleyDB and PyBSDDB installation.
+
+
+Using bsddb3Storage with Zope
+-----------------------------
+
+By default, Zope uses a FileStorage as its backend storage. To tell
+Zope to use an alternate storage, you need to set up a custom_zodb.py
+file.
+
+There is a sample custom_zodb.py file in the docs/ subdirectory,
+shipped with this release. The easiest way to get started with one of
+the Berkeley storages is to copy custom_zodb.py file to your
+SOFTWARE_HOME directory (your main Zope dir) and edit its contents to
+specify which storage you want to use. If you use an INSTANCE_HOME
+setup, you'll want to copy the file to the INSTANCE_HOME directory
+instead and do the same.
+
+If you choose to edit the contents of the custom_zodb.py file, you can
+change the "env" string to point to a different environment directory
+for BerkeleyDB. BerkeleyDB will store its support tables and log
+files in this directory. The contents of this directory can become
+quite large, even if your data needs are relatively modest (see
+"BerkeleyDB Log Files" below).
+
+You can also set up some tuning paramaters in the custom_zodb.py file.
+See the comments in that file and in the BerkeleyBase.py file for
+details.
+
+By default, the environment path is set in custom_zodb.py to a
+subdirectory of your Zope's var subdirectory. You may change this to
+any path that you have write permissions on. If the environment
+directory doesn't exist, it will be created when you first run Zope
+with one of the storages. It is recommended that you choose an
+environment directory which does not contain any other files.
+Additionally, you should not use BerkeleyDB on remotely mounted
+filesystems such as NFS.
+
+
+Using bsddb3Storage with ZEO
+----------------------------
+
+The Berkeley storages are compatible with ZEO. For general
+information on how to use alternate storage implementations with ZEO,
+see the "start.txt" file in the ZEO release documentation.
+
+
+Using Berkeley storage outside of Zope
+--------------------------------------
+
+ZODB applications that use the Berkeley storages need to take care to
+close the database gracefully, otherwise the underlying database could
+be left in a corrupt, but recoverable, state.
+
+By default, all the Berkeley storages open their Berkeley databases
+with the DB_RECOVER flag, meaning if recovery is necessary
+(e.g. because you didn't explicitly close it the last time you opened
+it), then recover will be run automatically on database open. You can
+also manually recover the database by running Berkeley's db_recover
+program.
+
+The upshot of this is that a database which was not gracefully closed
+can usually be recovered automatically, but this could greatly
+increase the time it takes to open the databases. This can be
+mitigated by periodically checkpointing the BerkeleyDB, since recovery
+only needs to take place from the time of the last checkpoint (the
+database is always checkpointed when it's closed).
+
+You can configure the Berkeley storages to automatically checkpoint
+the database every so often, by using the BerkeleyConfig class. The
+"interval" setting determines how often, in terms of ZODB commits,
+that the underlying database will be checkpointed. See the class
+docstring for BerkeleyBase.BerkeleyConfig for details.
+
+
+BerkeleyDB files
+----------------
+
+After Zope is started with one of the Berkeley storages, you will see
+a number of different types of files in your BerkeleyDB environment
+directory. There will be a number of "__db*" files, a number of
+"log.*" files, and several files which have the prefix ``zodb_``. The
+files which have the ``zodb_`` prefix are the actual BerkeleyDB
+databases which hold the storage data. The "log.*" files are
+write-ahead logs for BerkeleyDB transactions, and they are very
+important. The "__db*" files are working files for BerkeleyDB, and
+they are less important. It's wise to back up all the files in this
+directory regularly. BerkeleyDB supports "hot-backup". Log files
+need to be archived and cleared on a regular basis (see below).
+
+
+BerkeleyDB log files
+--------------------
+
+BerkeleyDB is a transactional database system. In order to maintain
+transactional integrity, BerkeleyDB writes data to log files before
+the data is committed. These log files live in the BerkeleyDB
+environment directory unless you take steps to configure your
+BerkeleyDB environment differently. There are good reasons to put the
+log files on a different disk than the data files:
+
+- The performance win can be huge. By separating the log and data
+ files, Berkeley can much more efficiently write data to disk. We
+ have seen performance improvements from between 2.5 and 10 times for
+ write intensive operations. You might also want to consider using
+ three separate disks, one for the log files, one for the data files,
+ and one for the OS swap.
+
+- The log files can be huge. It might make disk space management
+ easier by separating the log and data files.
+
+The log file directory can be changed by setting the "logfile"
+attribute on the config object given to the various storage
+constructors. Set this to the directory where BerkeleyDB should store
+your log files. Note that this directory must already exist.
+
+For more information about BerkeleyDB log files, recoverability and
+why it is advantageous to put your log files and your database files
+on separate devices, see
+
+ http://www.sleepycat.com/docs/ref/transapp/reclimit.html.
+
+You can reclaim some disk space by occasionally backing up and
+removing unnecessary BerkeleyDB log files. Here's a trick that I use::
+
+ % db_archive | xargs rm
+
+Be sure to read the db_archive manpages first!
+
+
+Tuning BerkeleyDB
+-----------------
+
+BerkeleyDB has lots of knobs you can twist to tune it for your
+application. Getting most of these knobs at the right setting is an
+art, and will be different from system to system. We're still working
+on recommendations with respect to the Full storage, but for the time
+being, you should at least read the following Sleepycat pages:
+
+ http://www.sleepycat.com/docs/ref/am_conf/cachesize.html
+ http://www.sleepycat.com/docs/ref/am_misc/tune.html
+ http://www.sleepycat.com/docs/ref/transapp/tune.html
+ http://www.sleepycat.com/docs/ref/transapp/throughput.html
+
+As you read these, it will be helpful to know that the bsddb3Storage
+databases all use BTree access method.
+
+One thing we can safely say is that the default BerkeleyDB cache size
+of 256KB is way too low to be useful. The Berkeley storages
+themselves default the cache size to 128MB which seems about optimal
+on a 256MB machine. Be careful setting this too high though, as
+performance will degrade if you tell Berkeley to consume more than the
+available resources. You can change the cache size by setting the
+"cachesize" attribute on the config object to the constructor.
+
+
+Archival and maintenance
+------------------------
+
+Log file rotation for Berkeley DB is closely related to database
+archival.
+
+BerkeleyDB never deletes "old" log files. Eventually, if you do not
+maintain your Berkeley database by deleting "old" log files, you will
+run out of disk space. It's necessary to maintain and archive your
+BerkeleyDB files as per the procedures outlined in
+
+ http://www.sleepycat.com/docs/ref/transapp/archival.html
+
+It is advantageous to automate this process, perhaps by creating a
+script run by "cron" that makes use of the "db_archive" executable as
+per the referenced document. One strategy might be to perform the
+following sequence of operations:
+
+- shut down the process which is using BerkeleyDB (Zope or the ZEO
+ storage server).
+
+- back up the database files (the files prefixed with "zodb").
+
+- back up all existing BerkeleyDB log files (the files prefixed
+ "log").
+
+- run ``db_archive -h /the/environment/directory`` against your
+ environment directory to find out which log files are no longer
+ participating in transactions (they will be printed to stdout one
+ file per line).
+
+- delete the log files that were reported by "db_archive" as no longer
+ participating in any transactions.
+
+"Hot" backup and rotation of log files is slightly different. See the
+above-referenced link regarding archival for more information.
+
+
+Disaster recovery
+-----------------
+
+To recover from an out-of-disk-space error on the log file partition,
+or another recoverable failure which causes the storage to raise a
+fatal exception, you may need to use the BerkeleyDB "db_recover"
+executable. For more information, see the BerkeleyDB documentation at
+
+ http://www.sleepycat.com/docs/ref/transapp/recovery.html
+
+
+BerkeleyDB temporary files
+--------------------------
+
+BerkeleyDB creates temporary files in the directory referenced by the
+$TMPDIR environment variable. If you do not have a $TMPDIR set, your
+temp files will be created somewhere else (see
+http://www.sleepycat.com/docs/api_c/env_set_tmp_dir.html for the
+tempfile decision algorithm used by BerkeleyDB). These temporary
+files are different than BerkeleyDB "log" files, but they can also
+become quite large. Make sure you have plenty of temp space
+available.
+
+
+Linux 2GB Limit
+---------------
+
+BerkeleyDB is effected by the 2GB single-file-size limit on 32-bit
+Linux ext2-based systems. The Berkeley storage pickle database (by
+default named "zodb_pickle"), which holds the bulk of the data for the
+Berkeley storages is particularly susceptible to large growth.
+
+If you anticipate your database growing larger than 2GB, it's
+worthwhile to make sure your system can support files larger than 2GB.
+Start with your operating system and file system. Most modern Linux
+distributions have large file support.
+
+Next, you need to make sure that your Python executable has large file
+support (LFS) built in. Python 2.2.1 is automatically configured with
+LFS, but for Python 2.1.3 you will need to rebuild your executable
+according to the instructions on this page:
+
+ http://www.python.org/doc/2.1.3/lib/posix-large-files.html
+
+IMPORTANT NOTE: If any of your BerkeleyDB files reaches the 2GB limit
+before you notice the failure situation, you will most likely need to
+restore the database environment from a backup, putting the restored
+files on a filesystem which can handle large files. This is due to
+the fact that the database file which "hit the limit" on a 2GB-limited
+filesystem will be left in an inconsistent state, and will probably be
+rendered unusable. Be very cautious if you're dealing with large
+databases.
+
+
+For More Information
+--------------------
+
+Information about ZODB in general is kept on the ZODB Wiki at
+
+ http://www.zope.org/Wikis/ZODB
+
+Information about the Berkeley storages in particular is at
+
+ http://www.zope.org/Wikis/ZODB/BerkeleyStorage
+
+The email list zodb-dev@lists.zope.org are where all the
+discussion about the Berkeley storages should take place.
+Subscribe or view the archives at
+
+ http://lists.zope.org/mailman/listinfo/zodb-dev
+
+
+
+..
+ Local Variables:
+ mode: indented-text
+ indent-tabs-mode: nil
+ sentence-end-double-space: t
+ fill-column: 70
+ End:
=== Removed File ZODB3/bsddb3Storage/MANIFEST ===
=== Removed File ZODB3/bsddb3Storage/README ===