[ZODB-Dev] zodb conversion questions

[Jürgen Herrmann]

@jim, resent to the list, sorry.

Am 07.02.2013 17:11, schrieb Jim Fulton:
> On Thu, Feb 7, 2013 at 10:48 AM, Jürgen Herrmann
> <Juergen.Herrmann at xlhost.de> wrote:
>> Am 06.02.2013 15:05, schrieb Jürgen Herrmann:
>>
>>> Hi there!
>>>
>>> I hav a relstorage with mysql backend that grew out of bounds
>>> and we're looking into different backend solutions now. Possibly
>>> also going back to FileStorage and using zeo...
>>>
>>> Anyway we'll have to convert the databases at some point. As these
>>> are live DBs we cannot shut them down for longer than the
>>> ususal maintenance interval during the night, so for maybe 2-3h.
>>>
>>> a full conversion process will never complete in this time so
>>> we're looking for a process that can split the conversion into
>>> two phases:
>>>
>>> 1. copy transactions from backup of the source db to the 
>>> destination
>>>    db. this can take a long time, we don't care. note the last
>>>    timestamp/transaction_id converted.
>>> 2. shut down the source db
>>> 3. copy transactions from the source db to the destination db, 
>>> starting
>>>    at the last converted transaction_id. this should be fast, as 
>>> only
>>>    a few transactions need to be converted, say < 1% .
>>>
>>>
>>> if i would reimplement copyTransactionsFrom() to accept a start
>>> transaction_id/timestamp, would this result in dest being an exact
>>> copy of source?
>>>
>>> source = open_my_source_storage()
>>> dest = open_my_destination_storage()
>>> dest.copyTransactionsFrom(source)
>>> last_txn_id = source.lastTransaction()
>>> source.close()
>>> dest.close()
>>>
>>> source = open_my_source_storage()
>>> # add some transactions
>>> source.close()
>>>
>>> source = open_my_source_storage()
>>> dest = open_my_destination_storage()
>>> dest.copyTransactionsFrom(source, last_txn_id=last_txn_id)
>>> source.close()
>>> dest.close()
>>
>>
>> I will reply to myself here :) This actually works, tested with a
>> modified version of FileStorage for now. I modified the signature
>> of copyTransactionsFrom to look like this:
>>
>> def copyTransactionsFrom(self, source, verbose=0, 
>> not_before_tid=None):
>
> ``start`` would be better to be consistent with the iterator API.

this was my first approach, though for my usecase it would be 
misleading
as the code roughly looks like this:

   if tid < not_before_tid:
     continue

and it excludes the given tid from the transactions re-stored. maybe
we can come up with a better name but start doesn't nail it :)

>
>> not_before_tid is a packed tid or None, None meaning "copy all"
>> (the default, so no existing API usage would break).
>>
>> Is there public interest in modifying this API permamently?
>
> +.1
>
> This "API" is a bit of an attractive nuisance.  I'd rather people
> learn how to use iterators in their own scripts, as they are very
> useful and powerful.  This API just hides that.

not sure i understand this correctly, maybe you could elaborate a
bit more? for my usecase you'd suggest i just use the storage iterator
and walk/re-store the transactions in my own code?
there's a lot of checking and branching going on inside
copyTransactionsFrom(), that's why i asked if this would work in the
first place.

> The second part, replaying old transactions is a bit more subtle,
> but it's still worth it for people to be aware of it.
>
> If I were doing this today, I'd make this documentation
> rather than API. But then, documentation ... whimper.
>
>> Anybody want to look at the actual code changes?
>
> Sure, if they have tests.  Unfortunately, we can only accept
> pull requests from zope contributors. Are you one?
> Wanna be one? :)

i'll look at the supplied test and see if i can make my test script
a proper test case for the test suite. shouldn't be too hard.

we'll decide about the contributor stuff after that :)

btw i need this to be in the ZODB version current Zope2 uses,
is this one on github already? if so, where can i find it? even
if i don't become a contributor this would make generating patches
much easier.

>
> Jim

thanks for your help!

Jürgen

-- 
>> XLhost.de ® - Webhosting von supersmall bis eXtra Large <<

XLhost.de GmbH
Jürgen Herrmann, Geschäftsführer
Boelckestrasse 21, 93051 Regensburg, Germany

Geschäftsführer: Jürgen Herrmann
Registriert unter: HRB9918
Umsatzsteuer-Identifikationsnummer: DE245931218

Fon:  +49 (0)800 XLHOSTDE [0800 95467833]
Fax:  +49 (0)800 95467830
Web:  http://www.XLhost.de