[Zodb-checkins] CVS: StandaloneZODB/bsddb3Storage - README:1.8
Guido van Rossum
guido@python.org
Tue, 12 Feb 2002 17:33:39 -0500
Update of /cvs-repository/StandaloneZODB/bsddb3Storage
In directory cvs.zope.org:/tmp/cvs-serv13020/bsddb3Storage
Modified Files:
README
Log Message:
More changes merging the StandaloneZODB-1.0 release branch into the
trunk. I'm *almost* done with the merge; the only file not yet merged
is FileStorage.py.
=== StandaloneZODB/bsddb3Storage/README 1.7 => 1.8 ===
+Berkeley (bsddb3) Storage for ZODB/Zope
+Please see the LICENSE.txt file for terms and conditions.
Introduction
@@ -9,6 +10,19 @@
tables, relying on Berkeley's transaction machinery to provide
reliability and recoverability.
+ WARNING: We do not consider Berkeley storage to be production
+ quality yet. The primary outstanding issue is poor performance
+ due to the underly Berkeley database's inability to efficiently
+ handle large `blobs'. ZODB is designed to store the pickles of
+ persistent objects, and these pickles can sometimes be quite
+ large; pickles on the order of 10's of KB or even several MB are
+ not unheard of.
+
+ We are currently working with Sleepycat to resolve this issue, but
+ as of yet have not arrived at a satisfactory solution. Be aware
+ that if your persistent objects have pickles over 4K in size
+ (which is likely), you will experience suboptimal performance.
+
Contents
@@ -16,10 +30,10 @@
implementations:
- Packless.py is an implementation of an undo-less, version-less
- storage that obviates the need for packing, except in the face
- of cyclic garbage. It uses a reference counting garbage
- collection strategy to clean up garbage objects in the normal
- case. This storage was released in 1.0 beta 1.
+ storage that obviates the need for packing, except in the
+ presence of cyclic garbage. It uses a reference counting
+ garbage collection strategy to clean up garbage objects in the
+ normal case. This storage was released in 1.0 beta 1.
- Minimal.py is a new implementation of an undo-less, version-less
storage, however it currently only does reference counting
@@ -28,7 +42,7 @@
storage, and provides a more robust temporary commit log, this
is the wave of the future. Because of resource constraints
though, it is unlikely that Minimal will get much attention for
- the 1.0 release.
+ the 1.0 final release.
- Full.py is a complete storage implementation, supporting undo,
versions, and automatic reference counting garbage collection.
@@ -44,9 +58,9 @@
http://www.zope.org/Wikis/ZODB/TransactionalUndo
Old style undo is /not/ supported in the Full.py storage, so
- unless you have a newer version of ZODB, you will not be able to
- perform undos. We will not be adding old style undo support to
- Full.py storage.
+ unless you have a newer version of Zope (2.5 should be fine),
+ you will not be able to perform undos. We will not be adding
+ old style undo support to Full.py storage.
As of 1.0 beta 3, Full storage also supports application level
conflict resolution. This is explained at
@@ -58,8 +72,7 @@
The Full and Minimal storages have been tested with Python 2.1.1
but should work for any Python 2.x version. It may or may not
- work with Python 1.5.2. It may or may not work with Python 2.2,
- but it will be tested against Python 2.2 before the final release.
+ work with Python 1.5.2. It should work well with Python 2.2.
It's possible they work with Python 1.5.2, but it's also not
likely that Python 1.5.2 will be explicitly supported. Full
@@ -73,14 +86,16 @@
Requirements
- You must install Sleepcat BerkeleyDB and Robin Dunn's PyBSDDB
- package separately. As of this writing, BerkeleyDB is at version
- 3.3.11 and PyBSDDB is at 3.3.1. It is highly recommended that you
- use at least these versions of the package, as we've seen locking
- (and other) problems with earlier releases.
+ You must install Sleepycat BerkeleyDB and Robin Dunn's PyBSDDB
+ package separately. The Berkeley storages are known to work with
+ BerkeleyDB 3.3.11 and PyBSDDB 3.3.1. As of this date, Sleepycat
+ has released BerkeleyDB 4.0.14 but PyBSDDB still doesn't
+ officially support it. We do not recommend you use any version of
+ BerkeleyDB before 3.3.11 as we've seen locking (and other)
+ problems with earlier releases.
To obtain the latest source release of BerkeleyDB, see the
- SleepyCat site:
+ Sleepycat site:
http://www.sleepycat.com
@@ -103,12 +118,25 @@
the BerkeleyDB patch and reinstall BerkeleyDB, and then re-run the
PyBSDDB tests.
+ Here's the build command I've used for the PyBSDDB distribution.
+ This assumes a default installation of BerkeleyDB 3.3.11.
+
+ % python setup.py build_ext --inplace --berkeley-db=/usr/local/BerkeleyDB.3.3/ --lflags="-Xlinker -rpath -Xlinker /usr/local/BerkeleyDB.3.3/lib"
+
When you can run the tests which ship with PyBSDDB, you'll know
you've been successful at both BerkeleyDB and PyBSDDB
installation.
-Installing bsddb3Storage
+Installing bsddb3Storage as part of StandaloneZODB
+
+ You should not need to do anything special to install bsddb3Storage
+ if you are installing it as part of the StandaloneZODB
+ distribution. If you've received bsddb3Storage as a separate
+ package, then read on for installation instructions.
+
+
+Installing bsddb3Storage as a separate package
The bsddb3Storage is distributed as a Python distutils package, so
the simplest thing to do is to use distutils to install it:
@@ -128,7 +156,7 @@
directions.
-Using bsddb3Storage
+Using bsddb3Storage with Zope
By default, Zope uses a FileStorage to hold ZODB data. To tell
Zope to use an alternate storage such as Packless, you need to set
@@ -161,7 +189,7 @@
remotely mounted filesystems such as NFS.
-Use with ZEO
+Using bsddb3Storage with ZEO
The Berkeley storages are compatible with ZEO. For general
information on how to use alternate storage implementations with
@@ -193,7 +221,7 @@
try/finally clause).
-BerkeleyDB Files
+BerkeleyDB files
After Zope is started with one of the Berkeley storages, you will
see a number of different types of files in your BerkeleyDB
@@ -227,19 +255,18 @@
get to it soon).
-BerkeleyDB Log Files
+BerkeleyDB log files
BerkeleyDB is a transactional database system. In order to
maintain transactional integrity, BerkeleyDB writes data to "log
files" before the data is committed. These log files live in the
- BerkeleyDB "environment" directory unless you take steps to
+ BerkeleyDB environment directory unless you take steps to
configure your BerkeleyDB environment differently. BerkeleyDB log
- files can become quite large, as well, so it may be necessary to
- place them on a separate partition with lots of free disk space.
- The log file directory can be changed by creating a file named
- 'DB_CONFIG' in the BerkeleyStorage "environment" directory you've
- chosen within 'custom_zodb.py', customizing the following
- content:
+ files can become quite large, so it may be necessary to place them
+ on a separate partition with lots of free disk space. The log
+ file directory can be changed by creating a file named 'DB_CONFIG'
+ in the BerkeleyStorage environment directory you've chosen
+ (e.g. within custom_zodb.py), customizing the following content:
set_lg_dir /the/path/to/the/log/file/directory
@@ -247,7 +274,7 @@
your actual database files will still be kept in the directory
specified by the "env" setting of your custom_zodb.py; only the
BerkeleyDB log files will be written to the directory you specify
- in 'DB_CONFIG'.
+ in DB_CONFIG.
Redirecting log files to a directory other than your environment
directory may improve recoverability in the case of BerkeleyDB
@@ -268,7 +295,7 @@
But be sure to read the db_archive manpages first!
-Setting BerkeleyDB Maximum Locks
+Setting BerkeleyDB maximum locks
ZODB transactions can be of almost arbitrary sizes (actually, they
"top out" at a total size of 2GB). BerkeleyDB is configured to
@@ -284,7 +311,7 @@
BerkeleyDB locking.
To manually size locking, create (or edit) the file DB_CONFIG
- within the BerkeleyDB "environment" directory you've chosen in
+ within the BerkeleyDB environment directory you've chosen in
custom_zodb.py, adding the following directives to the DB_CONFIG
file:
@@ -292,9 +319,9 @@
set_lk_max_objects 500
set_lk_max_lockers 3
- Change the integers as necessary. When one of the Berkeley
- storages starts up, the Berkeley directives supplied in DB_CONFIG
- will override the defaults.
+ Change the values as necessary. When one of the Berkeley storages
+ starts up, the Berkeley directives supplied in DB_CONFIG will
+ override the defaults.
Precision-sizing BerkeleyDB locking is a site-dependent task.
Sleepycat recommends that you run the "db_stat -c" command against
@@ -332,7 +359,7 @@
bsddb3Storage databases all use BTree access method.
-Archival and Maintenance
+Archival and maintenance
Log file rotation for Berkeley DB is closely related to database
archival.
@@ -368,17 +395,18 @@
the above-referenced link regarding archival for more information.
-Disaster Recovery
+Disaster recovery
To recover from an out-of-disk-space error on the log file
partition, or another recoverable failure which causes the storage
to raise a fatal exception, you may need to use the BerkeleyDB
"db_recover" executable. For more information, see the BerkeleyDB
documentation at
- http://www.sleepycat.com/docs/ref/transapp/recovery.html.
+
+ http://www.sleepycat.com/docs/ref/transapp/recovery.html
-BerkeleyDB Temp Files
+BerkeleyDB temporary files
BerkeleyDB creates temporary files in the directory referenced by
the $TMPDIR environment variable. If you do not have a $TMPDIR
@@ -393,13 +421,14 @@
Linux 2GB Limit
BerkeleyDB is effected by the 2GB single-file-size limit on 32-bit
- Linux ext2-based systems. The Berkeley storage "pickle" database
+ Linux ext2-based systems. The Berkeley storage pickle database
(by default named "zodb_pickle"), which holds the bulk of the data
for the Berkeley storages is particularly susceptible to large
growth. If you notice that this file's size (or any other
Berkeley storage-related file) is nearing 2GB, you'll need to move
your BerkeleyDB environment to a filesystem which supports > 2GB
- files.
+ files. If you enable largefile support for your filesystem, you
+ should be able to create much larger databases.
IMPORTANT NOTE: If any of your BerkeleyDB files reaches the 2GB
limit before you notice the failure situation, you will most
@@ -415,7 +444,7 @@
Information about ZODB in general is kept on the ZODB Wiki at
- http://www.zope.org/Wikis/ZODB/FrontPage
+ http://www.zope.org/Wikis/ZODB
Information about the Berkeley storages in particular is at