[ZODB-Dev] zodb conversion questions

[Jim Fulton]

On Thu, Feb 7, 2013 at 10:48 AM, Jürgen Herrmann
<Juergen.Herrmann at xlhost.de> wrote:
> Am 06.02.2013 15:05, schrieb Jürgen Herrmann:
>
>> Hi there!
>>
>> I hav a relstorage with mysql backend that grew out of bounds
>> and we're looking into different backend solutions now. Possibly
>> also going back to FileStorage and using zeo...
>>
>> Anyway we'll have to convert the databases at some point. As these
>> are live DBs we cannot shut them down for longer than the
>> ususal maintenance interval during the night, so for maybe 2-3h.
>>
>> a full conversion process will never complete in this time so
>> we're looking for a process that can split the conversion into
>> two phases:
>>
>> 1. copy transactions from backup of the source db to the destination
>>    db. this can take a long time, we don't care. note the last
>>    timestamp/transaction_id converted.
>> 2. shut down the source db
>> 3. copy transactions from the source db to the destination db, starting
>>    at the last converted transaction_id. this should be fast, as only
>>    a few transactions need to be converted, say < 1% .
>>
>>
>> if i would reimplement copyTransactionsFrom() to accept a start
>> transaction_id/timestamp, would this result in dest being an exact
>> copy of source?
>>
>> source = open_my_source_storage()
>> dest = open_my_destination_storage()
>> dest.copyTransactionsFrom(source)
>> last_txn_id = source.lastTransaction()
>> source.close()
>> dest.close()
>>
>> source = open_my_source_storage()
>> # add some transactions
>> source.close()
>>
>> source = open_my_source_storage()
>> dest = open_my_destination_storage()
>> dest.copyTransactionsFrom(source, last_txn_id=last_txn_id)
>> source.close()
>> dest.close()
>
>
> I will reply to myself here :) This actually works, tested with a
> modified version of FileStorage for now. I modified the signature
> of copyTransactionsFrom to look like this:
>
> def copyTransactionsFrom(self, source, verbose=0, not_before_tid=None):

``start`` would be better to be consistent with the iterator API.

> not_before_tid is a packed tid or None, None meaning "copy all"
> (the default, so no existing API usage would break).
>
> Is there public interest in modifying this API permamently?

+.1

This "API" is a bit of an attractive nuisance.  I'd rather people
learn how to use iterators in their own scripts, as they are very
useful and powerful.  This API just hides that.

The second part, replaying old transactions is a bit more subtle,
but it's still worth it for people to be aware of it.

If I were doing this today, I'd make this documentation
rather than API. But then, documentation ... whimper.

> Anybody want to look at the actual code changes?

Sure, if they have tests.  Unfortunately, we can only accept
pull requests from zope contributors. Are you one?
Wanna be one? :)

Jim

-- 
Jim Fulton
http://www.linkedin.com/in/jimfulton
Jerky is better than bacon! http://zo.pe/Kqm