[ZODB-Dev] Re: How expensive are savepoints?

[Tim Peters]

[Christian Heimes]
>>> How expensive and costly are savepoints?

[Tim Peters]
>> 6, maybe 6.2, depending on the units you're using <wink>.  Seriously,
>> how can such a question be answered?  How expensive is math.log()?

[Christian]
> My professor for numerical mathmatics would say it is very expensive
> because log() takes much more than 10 cpu cycles. *g*

Ha!  It was a trick question <wink>.  It almost certainly depends on the
specific argument you pass to math.log().  That it made it a fair question
too, though, since the cost of a savepoint also depends on lots of the "bits
going into it".

...

> I wasn't sure if savepoint() is either just marking a point in the middle
> of a transaction or storing the transaction somewhere. From my point of
> view it is costly compared to __add__(). *g*

In the absence of savepoint/subtxn, modified objects live in RAM until the
final commit.  With savepoint/subtxn, they get written to disk "early".  I
think the cause for the worst-case time differences was illustrated by the
test code I showed:  when the same object is modified multiple times within
a transaction, savepoint/subtxn will save an object's state to disk each
time it's been modified since the last savepoint/subtxn.

>> If I were you, I'd just _try_ it ...

> I'm migrating CMF objects to Archetypes objects including metadata,
> security and so on. The migiration of a typical object takes about 0.2 to
> 1 sec including catalog updates. A folderish object with hundres to
> thousands of children requires much more time but that's the fault of the
> catalog. Every object is uncatalog and catalog again ... ugly, time
> consuming but required in Zope2. I'm wishing we have events ...

I'd still just try it.

...
> I could write some code that migrates all objects in a folder before
> calling savepoint() but it's not worth the complexity and code.

I expect you could do a hybrid too, along the lines of

    for each chunk of, say, 100 objects:
        make a savepoint
        try to migrate them
        if anything failed:
            roll back to the savepoint, and do this
            chunk of 100 over again, but one at a time

>> Of course if this is a one-time migration, I wouldn't worry about
>> expense at all -- for all I know, it took me longer to write this reply
>> than it will take you to run the migration script <0.6 wink>.

> I wouldn't be sure in your place. I'm migrating all data of nearly all
> objects from one set of content types (CMF) to another set
> (ATContentTypes). For a very large site like plone.org the migration was
> running about 1 to 2h.

If "h" means hour, I'm already spent close to 2h writing little test cases
and writing up what I found for you.  Two hours "for a very large site"
doesn't strike me as worth fretting over at all -- but then I admit it's not
my site ;-).