[Zope] Running multiple transactions during single request

[Charl Coetzee]

Hmmn, I should have realized this.

This is one of those cases where functionality was needed and got added 
quickly, now needs to be "kept afloat" to buy time in order to implement 
it more sensibly.

What was originally going to be small imports, now has the potential to 
have each single import request run for several hours just creating the 
data (e.g. saw a 2h one this morning).  We are already using savepoints, 
but during the course of the import the objects do logically come in 
"units" (sort of akin to a human creating them individually by filling 
out forms, which would of course be separate transactions);  besides 
concerns for ConflictErrors, there is value to the objects incrementally 
appearing to other users even while the import is still running.  (So 
e.g. QueueCatalog helps some aspects of this, but not all -- there is 
value to this being dealt with as multiple transactions).

Candidates for tackling this in the longer run, include:
- Having an external process split the file in small batches, and 
sending them as individual web requests.  (But it seems there must be a 
better way).
- Having a separate app connecting to ZEO, using Zope2.app() and 
manipulating everything programmatically outside of the publisher.  
(Though this is new terrain for me, and I haven't had much time to check 
this out and all the various ramifications -- e.g the import needs to 
use things like Plone's _createObjectByType(), and potentially may also 
want to piggy-back on existing formlib validation).
- Some sort of queuing system, perhaps lovely.remotetask (again new terrain)

( BTW its only a handful of back-office administrative users who need 
this type of function, so their "user experience" of the import is not 
really important.)

For the short term, I have a feeling that breaking the CSV into smaller 
chunks externally and running separate requests, will be a faster 
interim solution, than trying to run multiple transactions inside the 
request properly (sounds like that would need monkeypatching).

Thanks,

Charl


Chris Withers wrote:
> Charl Coetzee wrote:
>>        if everyFifthPass:
>>            transaction.commit()
>>            transaction.begin()
>>        # end of while loop
>>
>> really does have the desired effect, where also ConflictError 
>> handling etc continue to work as expected, without nasty side effects 
>> etc.
>
> No, this does not handle conflict errors.
> The conflict error will be caught in the publisher and the whole 
> request will be retried, resulting in whatever commits you've manually 
> done up until that point being re-done...
>
> Why are you using commits rather than savepoints here?
>
> cheers,
>
> Chris
>