[ZODB-Dev] duplicate functionality in bsddb3Storage

[Andrew Dalke]

Barry:
>Are you using the CVS version of this stuff, the latest StandaloneZODB
>release, or the latest bsddb3Storage release?

Oops, sorry about that.  bsddb3Storage-1.0beta4.tar.gz.  The
company I'm doing this for is behind some proxy firewall and
I don't know how to do anything but HTTP/FTP through it, so
doing the CVS options would be .. tricky.

>This should be easy to trace though, if you're using Full (which I
>recommend).  Full derives from ...

Oh, it was easy enough to trace.  But I'm trying to work under
the ignorance-is-bliss principle - there weren't enough hints
for me to figure out I was looking at the wrong place, or that
the code I was looking at was deprecated.

> To paraphrase John Cleese, "base.py don't enter inna it."

To quote from Gary Larson: "may I be excused? My brain is full."

 :)

>The explanation below assumes the latest CVS snapshot of
>bsddb3Storage.

Okay, I'll get the code from CVS.  Umm, where?  I can't find
any CVS download info starting from
  http://www.zope.org/Products/bsddb3Storage
AMK's pages point to
  http://cvs.zope.org/Releases/StandaloneZODB/
but there's no info on how to get the code.  And the README
links to 'http://www.zope.org/Wikis/ZODB.' but the Wiki really
should not link the final "." in a URL.

  I'm guessing I need to download StandaloneZODB to get
bsddb3Storage.  Looking at
  http://www.zope.org/Wikis/ZODB/StandaloneZODB
I need to "Do a CVS checkout of the module StandaloneZODB".
To figure that out I eventually looked in the "Zope CVS"
link on the left, then the "http://dev.zope.org/CVS" link, then
the "ZopeCVSFAQ" then the ReadOnlyAccess to figure out that
I didn't need a cvs password.

In other words, it's really, really hard to figure out
how to get to the CVS version of bsddb3Storage.

>Now, to control the various BerkeleyDB knobs, create a file inside BDB
>called DB_CONFIG.

In my case, I have a function called "create_database".  Is your
suggestion that the proper behaviour of the function be:

def create_database(dbname, various, other, args):
    os.mkdir(dbname)
    outfile = open(os.path.join(dbname, "DB_CONFIG"), "w")
    outfile.write(default_configuration)
    outfile.close()
    store = Full.Full(dbname)
     ...
 ?
I think I would rather pass my own DBEnv (with appropriate
lock info) to Full's constructor.

def create_database(dbname, various, other, args):
    dbenv = DBEnv()
    dbenv.set_lk_max_locks(20000)
      .. other tweaks ..
    e.open(dbname, db.DB_CREATE | .... )
    store = Full.Full(dbname, dbenv)

(Doing this from memory - my pardons for any API approximations.)

Or maybe env_from_string could take an optional DBEnv, so I
can call it and won't have to dup the parameters needed for
'open'.  Hmmmmmm...

>Full is the best supported, and for us, most important storage.
>Minimal is what I plan on replacing Packless with.  Packless mostly
>works, but isn't supported.

We don't need versioning.  We're mostly using ZODB as a persistent
object store / memory management system.  So I'm considering
using Minimal/Packless since it's tedious to hit pack() to remove
old.  And we all forget to that.

>As a side note, BerkeleyBase.py doesn't call set_lk_max()
>directly.  We've decided that we won't twist these BerkeleyDB knobs in
>the bsddb3Storage code because then we'd have to expose ways of
>letting you tune them, and BerkeleyDB already has a good way of
>letting you do that.  No need to re-invent the wheel here.

And I've no problem passing in my own DBEnv, as you all support.

>I'm currently experimenting with a migration script that converts a
>~6000 transaction FileStorage to a Full storage.  The FileStorage is a
>snapshot of activity on zope.org.

>Fortunately, BerkeleyDB 3.3.11 seems to have cleared all that up for
>us, and I believe the Subversion folks too.  So I highly recommend
>upgrading to that version, and using PyBSDDB 3.3.x.

That's what we're using.  We nearing the end of upgrading pretty
much everything - Python, ZODB, BSDDB3, Numeric, so going from
1.5.2 to the latest stock.

>My migration
>script now requires just under 20,000 locks, 3 lockers, and just over
>15,000 lock objects.  That seems much more reasonable.

Sigh.  And what we're doing with ZODB is probably so outside of
what most others do that we'll have to discover our own tuning
tweaks.  (But then, I needed to do that with FileStorage - we
needed to commit every 1,000 compounds added to the database
else the machine would start thrashing.)


>Below are some useful urls to read about performance tuning issues.

Thanks!  Though to understand it means learning things like,
does bsddb3Storage use the "Btree access method."  Grep says
yes.  But I'm losing my blissful ignorance.  :)

                    Andrew
                    dalke@dalkescientific.com