"TD" == Toby Dickenson <tdickenson@geminidataloggers.com> writes:
>> What would happen if the thread got killed in the middle of the >> move? TD> A more interesting question is what happens if the thread is TD> killed when it has moved some, but not all, of the files which TD> relate to one transaction. That's more what I was thinking about. TD> In DirectoryStorage terminology this leaves the data directory TD> in a state which "is not a snapshot". The only disadvantage is TD> that you can not use the tools which assume the data directory TD> is a snapshot: the checking tool, the incremental back tool, TD> and the replication tool. Interesting. Berkeley storage has something similar. Let's say the storage crashes in the middle of tpc_vote() or tpc_finish(). First, BerkeleyDB's recovery should clean up any of its corrupt transactions. Then the storage has its own recovery process for the "current" transaction, i.e. the one we were in the middle of at the time of the crash. Depending on when the crash occurred, the storages will either commit or abort the current transaction (if the crash happened in tpc_finish(), we'll complete the commit, otherwise we abort the transaction). So I believe -- although this isn't battle tested -- that the Berkeley storages should be pretty safe against inconsistency and corruption. >> We could make the autopack thread a daemon process >> TD> Thats how DirectoryStorage now works. Hmm, maybe we make >> it a daemon thread and put a timeout on the join, so we'll try >> to exit cleanly and won't hang if we can't. TD> Because some of the operations performed in the daemon thread TD> take a long time to complete? Potentially yes, although the steps in the pack process are BerkeleyDB transactionally protected. I think the mark-and-sweep phases are examples of things that could take a long time. It should be possible to craft some escape hatches to do a more controlled shutdown. TD> Would it be possible to break those operations into TD> transactionally-coherent chunks which complete in a reasonable TD> time? close() could wait for the current chunk to finish. Yep. >> Autopacking should be safe transactionally, but we'd have to do >> a recovery if it got killed uncleanly. TD> Doesnt having a good checkpointing strategy mean that this TD> should never take long? I think checkpointing itself is a trade-off between the length of time it takes to checkpoint and the length of time it could potentially take to recover. >> TD> Last time I looked at your BerkeleyDB storages, the TD> >> administrator needed to implement his own checkpointing TD> >> policy. I always thought that was a disadvantage. Would it now >> TD> be a good idea for the storages to trigger their own TD> >> checkpointing inside the autopacker thread? >> Actually, now you can configure the storages to automatically >> checkpoint every nth ZODB transaction. Checkpointing in a >> thread may or may not provide additional benefit. TD> Interesting. BerkeleyStorage's automatic checkpointing is TD> currently triggered inside a commit or abort. This means the TD> checkpoint overhead is incurred at the one time you dont want TD> it - while a user is waiting for his transaction to TD> commit. DirectoryStorage's main use for the thread is to TD> perform its equivalent asynchronously. Assuming your storage TD> is not saturated with writes (which only happens in benchmarks TD> ;-) then checkpointing happens for free. That's a very interesting idea! Hmm, checkpoint in a thread. Yeah, that sounds much better, although I'm mildly concerned with concurrancy issues, especially given that we don't use the BerkeleyDB locking subsystem. I'll have to read up on the docs, but it's a good idea. -Barry