[ZODB-Dev] duplicate functionality in bsddb3Storage

[Barry A. Warsaw]

Clearing out some email backlog...

>>>>> "AD" == Andrew Dalke <adalke@mindspring.com> writes:

    AD> Oops, sorry about that.  bsddb3Storage-1.0beta4.tar.gz.  The
    AD> company I'm doing this for is behind some proxy firewall and
    AD> I don't know how to do anything but HTTP/FTP through it, so
    AD> doing the CVS options would be .. tricky.

I plan on releasing a 1.0b5 today, if I work out the last few kinks in
my new pack() implementation.

    AD> And the README links to 'http://www.zope.org/Wikis/ZODB.' but
    AD> the Wiki really should not link the final "." in a URL.

Yup, that's a Wikibug/stx problem which will be fixed in the next
release.  Note that the ZODB wiki has a section about "Community"
under which there is a link to the CVS information for all of Zope
(and the StandaloneZODB project is one module you can check out
separately).

    AD> I'm guessing I need to download StandaloneZODB to get
    AD> bsddb3Storage.

You could check out just the bsddb3Storage module, but it's probably
better to get the whole StandaloneZODB module.
    
    AD> In other words, it's really, really hard to figure out
    AD> how to get to the CVS version of bsddb3Storage.

I've added the exact commands you need to check out an anonymous
read-only version of the StandaloneZODB project to:

    http://www.zope.org/Wikis/ZODB/StandaloneZODB

    AD> In my case, I have a function called "create_database".  Is
    AD> your suggestion that the proper behaviour of the function be:

    | def create_database(dbname, various, other, args):
    |     os.mkdir(dbname)
    |     outfile = open(os.path.join(dbname, "DB_CONFIG"), "w")
    |     outfile.write(default_configuration)
    |     outfile.close()
    |     store = Full.Full(dbname)
    |      ...
    |  ?

    AD> I think I would rather pass my own DBEnv (with appropriate
    AD> lock info) to Full's constructor.

Agreed.  The next version will add back the ability to pass in a DBEnv
object for the `env' argument, but it'll be up to you to make sure the
environment is created and opened correctly, and with the appropriate
flags.  I.e. other than testing env for None or string-iness, no other
typechecking on the passed in value is done.

    AD> We don't need versioning.  We're mostly using ZODB as a
    AD> persistent object store / memory management system.  So I'm
    AD> considering using Minimal/Packless since it's tedious to hit
    AD> pack() to remove old.  And we all forget to that.

One of the the things we'll be working on after the StandaloneZODB 1.0
release, will be support for auto-pack policies.  It's not completely
fleshed out yet, but the idea will be that you can specify certain
coarse-grained packing policies, like "pack away all revisions older
than a day", and the storage will just do the right thing.

    AD> Sigh.  And what we're doing with ZODB is probably so outside
    AD> of what most others do that we'll have to discover our own
    AD> tuning tweaks.  (But then, I needed to do that with
    AD> FileStorage - we needed to commit every 1,000 compounds added
    AD> to the database else the machine would start thrashing.)

Heck, if you were running my migration script with the same data, and
same code base, but just on different hardware, you'd probably need to
come up with your own tuning tweaks. ;)  Things like amount of RAM,
speed of disk drives, number of disk controllers, etc. can all have an
effect on where you're going to want some of those numbers to be.

    BAW> Below are some useful urls to read about performance tuning issues.

    AD> Thanks!  Though to understand it means learning things like,
    AD> does bsddb3Storage use the "Btree access method."  Grep says
    AD> yes.  But I'm losing my blissful ignorance.  :)

Good point.  I'll add this tidbit to the README file.

Cheers,
-Barry