[Zope-dev] DB.close() needs to be called

[Toby Dickenson]

(cc zodb-dev, who may also be interested)

On Wednesday 13 November 2002 4:18 pm, Barry A. Warsaw wrote:
> >>>>> "TD" =3D=3D Toby Dickenson <tdickenson@geminidataloggers.com> wri=
tes:
>     >> worse" part.  If you've enable autopacking and you don't
>     >> cleanly close the storage, you won't exit the process because
>     >> the autopack thread won't get stopped and joined.
>
>     TD> Yes, I had the same problem with DirectoryStorage, which uses
>     TD> a seperate thread to move writes from its journal directory
>     TD> into the database directory.
>
> Ah.  Is this transactional?  What would happen if the thread got
> killed in the middle of the move?

There are lots of ways to look at that question....

I am relying on the filesystem guarantee that the files are all either in=
=20
their original directory, or their new directory. DirectoryStorage has it=
s=20
equivalent of BerkeleyStorage's autorecovery which, at startup,=20
asynchronously resumes the file moving wherever it left off.

A more interesting question is what happens if the thread is killed when =
it=20
has moved some, but not all, of the files which relate to one transaction=
=2E In=20
DirectoryStorage terminology this leaves the data directory in a state wh=
ich=20
"is not a snapshot". The only disadvantage is that you can not use the to=
ols=20
which assume the data directory is a snapshot: the checking tool, the=20
incremental back tool, and the replication tool.

>     >> We could make the
>     >> autopack thread a daemon process
>
>     TD> Thats how DirectoryStorage now works.
>
> Hmm, maybe we make it a daemon thread and put a timeout on the join,
> so we'll try to exit cleanly and won't hang if we can't.=20

Because some of the operations performed in the daemon thread take a long=
 time=20
to complete? Would it be possible to break those operations into=20
transactionally-coherent chunks which complete in a reasonable time? clos=
e()=20
could wait for the current chunk to finish.

> Autopacking
> should be safe transactionally, but we'd have to do a recovery if it
> got killed uncleanly.

Doesnt having a good checkpointing strategy mean that this should never t=
ake=20
long?

>     TD> Last time I looked at your BerkeleyDB storages, the
>     TD> administrator needed to implement his own checkpointing
>     TD> policy. I always thought that was a disadvantage. Would it now
>     TD> be a good idea for the storages to trigger their own
>     TD> checkpointing inside the autopacker thread?
>
> Actually, now you can configure the storages to automatically
> checkpoint every nth ZODB transaction.  Checkpointing in a thread may
> or may not provide additional benefit.

Interesting. BerkeleyStorage's automatic checkpointing is currently trigg=
ered=20
inside a commit or abort. This means the checkpoint overhead is incurred =
at=20
the one time you dont want it - while a user is waiting for his transacti=
on=20
to commit. DirectoryStorage's main use for the thread is to perform its=20
equivalent asynchronously. Assuming your storage is not saturated with wr=
ites=20
(which only happens in benchmarks ;-) then checkpointing happens for free=
=2E