hi there! as our zope2 zodb grows we experience more and more conflicts with long running transactions. so i'm going to rewrite these methods to commit transactions in batches - here my attempt to tackle this problem. the idea is to get a list of objects to be modified in this transaction, break it down to batches and commit after each batch was processed. to avoid zope's retry-on-conflict machinery and still have a chance to retry per batch transaction commits, the iteration over one batch is guarded by a try:except ConflictError: and a retry logic. i wrote some more or less pseudo code to make this clear. def myLongRunningMethod(self): BATCH_SIZE = 10 MAX_CONFLICTS = 3 work_items = [some, work, to , do, ...] # list of persistent objects to be modified in this method idx = 0 while idx < len(work_items): conflicts = 0 try: my_batch = work_items[idx, idx+BATCH_SIZE] for work_item in my_batch: do_some_work(work_item) transaction.commit() except ConflicError: conflicts += 1 if conflicts > MAX_CONFLICTS: raise else: idx += BATCH_SIZE does this sound like a reasonable approach? jürgen --
XLhost.de - eXperts in Linux hosting ® <<
XLhost.de GmbH Jürgen Herrmann, Geschäftsführer Boelckestrasse 21, 93051 Regensburg, Germany Geschäftsführer: Volker Geith, Jürgen Herrmann Registriert unter: HRB9918 Umsatzsteuer-Identifikationsnummer: DE245931218 Fon: +49 (0)800 XLHOSTDE [0800 95467833] Fax: +49 (0)800 95467830 WEB: http://www.XLhost.de IRC: #XLhost@irc.quakenet.org
On Thu, May 20, 2010 at 14:18, Jürgen Herrmann <Juergen.Herrmann@xlhost.de>wrote:
hi there!
as our zope2 zodb grows we experience more and more conflicts with long running transactions. so i'm going to rewrite these methods to commit transactions in batches - here my attempt to tackle this problem. the idea is to get a list of objects to be modified in this transaction, break it down to batches and commit after each batch was processed. to avoid zope's retry-on-conflict machinery and still have a chance to retry per batch transaction commits, the iteration over one batch is guarded by a try:except ConflictError: and a retry logic.
Zope already has its own retry mechanism, maybe you might want to call your long running method multiple times, each time working on the next batch. Viele Grüße, Patrick
On Thu, 20 May 2010 15:59:58 +0200, Patrick Gerken <do3ccqrv@googlemail.com> wrote:
On Thu, May 20, 2010 at 14:18, Jürgen Herrmann <Juergen.Herrmann@xlhost.de>wrote:
hi there!
as our zope2 zodb grows we experience more and more conflicts with long running transactions. so i'm going to rewrite these methods to commit transactions in batches - here my attempt to tackle this problem. the idea is to get a list of objects to be modified in this transaction, break it down to batches and commit after each batch was processed. to avoid zope's retry-on-conflict machinery and still have a chance to retry per batch transaction commits, the iteration over one batch is guarded by a try:except ConflictError: and a retry logic.
Zope already has its own retry mechanism, maybe you might want to call your long running method multiple times, each time working on the next batch.
in fact zope's retry machinery is kicking in too often which is a bad thing if the retried transaction runs for say 30min or so. having to gather the list of objects outside of zope, doing the batching there and calling into zope multiple times - i wanted to avoid that. i'd like to have all the logic contained in one zope method for the sake of simplicity on the caller's side (most of these are kicked off by cron jobs). jürgen --
XLhost.de - eXperts in Linux hosting ® <<
XLhost.de GmbH Jürgen Herrmann, Geschäftsführer Boelckestrasse 21, 93051 Regensburg, Germany Geschäftsführer: Volker Geith, Jürgen Herrmann Registriert unter: HRB9918 Umsatzsteuer-Identifikationsnummer: DE245931218 Fon: +49 (0)800 XLHOSTDE [0800 95467833] Fax: +49 (0)800 95467830 WEB: http://www.XLhost.de IRC: #XLhost@irc.quakenet.org
Am 20.05.2010 16:05, schrieb Jürgen Herrmann:
On Thu, 20 May 2010 15:59:58 +0200, Patrick Gerken <do3ccqrv@googlemail.com> wrote:
On Thu, May 20, 2010 at 14:18, Jürgen Herrmann <Juergen.Herrmann@xlhost.de>wrote:
hi there!
as our zope2 zodb grows we experience more and more conflicts with long running transactions. so i'm going to rewrite these methods to commit transactions in batches - here my attempt to tackle this problem. the idea is to get a list of objects to be modified in this transaction, break it down to batches and commit after each batch was processed. to avoid zope's retry-on-conflict machinery and still have a chance to retry per batch transaction commits, the iteration over one batch is guarded by a try:except ConflictError: and a retry logic.
Zope already has its own retry mechanism, maybe you might want to call your long running method multiple times, each time working on the next batch.
in fact zope's retry machinery is kicking in too often which is a bad thing if the retried transaction runs for say 30min or so.
having to gather the list of objects outside of zope, doing the batching there and calling into zope multiple times - i wanted to avoid that.
i'd like to have all the logic contained in one zope method for the sake of simplicity on the caller's side (most of these are kicked off by cron jobs).
jürgen
take a look at http://pypi.python.org/pypi/affinitic.zamqp maybe it is a solution for you. You can deligate your jobs out of zope and get it back after some time. zamqp then fire an event and you subscribe this event to handle thie results. with kind regards Maik Derstappen -- ======================================================================== Derstappen I T Consulting Tel: +49 ( 341 ) 600 13 0 31 Zope/E-Mail/Backup/Monitoring Mobil: +49 ( 178 ) 861 2 833 M a i k D e r s t a p p e n Fax: +49 ( 180 ) 5 021 121 90 56 H e r l o ß s o h n s t r 12 Email: maik.derstappen@derstappen-it.de 0 4 1 5 5 L e i p z i g Internet: http://www.derstappen-it.de ========================================================================
Am 20.05.2010 19:34, schrieb Maik Derstappen, Derstappen IT:
Am 20.05.2010 16:05, schrieb Jürgen Herrmann:
On Thu, 20 May 2010 15:59:58 +0200, Patrick Gerken <do3ccqrv@googlemail.com> wrote:
On Thu, May 20, 2010 at 14:18, Jürgen Herrmann <Juergen.Herrmann@xlhost.de>wrote:
hi there!
as our zope2 zodb grows we experience more and more conflicts with long running transactions. so i'm going to rewrite these methods to commit transactions in batches - here my attempt to tackle this problem. the idea is to get a list of objects to be modified in this transaction, break it down to batches and commit after each batch was processed. to avoid zope's retry-on-conflict machinery and still have a chance to retry per batch transaction commits, the iteration over one batch is guarded by a try:except ConflictError: and a retry logic.
Zope already has its own retry mechanism, maybe you might want to call your long running method multiple times, each time working on the next batch.
in fact zope's retry machinery is kicking in too often which is a bad thing if the retried transaction runs for say 30min or so.
having to gather the list of objects outside of zope, doing the batching there and calling into zope multiple times - i wanted to avoid that.
i'd like to have all the logic contained in one zope method for the sake of simplicity on the caller's side (most of these are kicked off by cron jobs).
jürgen
take a look at http://pypi.python.org/pypi/affinitic.zamqp maybe it is a solution for you. You can deligate your jobs out of zope and get it back after some time. zamqp then fire an event and you subscribe this event to handle thie results.
with kind regards Maik Derstappen
maik, this looks interesting. just out of curiosity: how would one alert the user that a long running job (say after some 10 minutes) has finished. does the client have to poll? or is there some "magic" push functionality? robert
Am 20.05.2010 20:04, schrieb robert rottermann:
Am 20.05.2010 19:34, schrieb Maik Derstappen, Derstappen IT:
Am 20.05.2010 16:05, schrieb Jürgen Herrmann:
On Thu, 20 May 2010 15:59:58 +0200, Patrick Gerken <do3ccqrv@googlemail.com> wrote:
On Thu, May 20, 2010 at 14:18, Jürgen Herrmann <Juergen.Herrmann@xlhost.de>wrote:
hi there!
as our zope2 zodb grows we experience more and more conflicts with long running transactions. so i'm going to rewrite these methods to commit transactions in batches - here my attempt to tackle this problem. the idea is to get a list of objects to be modified in this transaction, break it down to batches and commit after each batch was processed. to avoid zope's retry-on-conflict machinery and still have a chance to retry per batch transaction commits, the iteration over one batch is guarded by a try:except ConflictError: and a retry logic.
Zope already has its own retry mechanism, maybe you might want to call your long running method multiple times, each time working on the next batch.
in fact zope's retry machinery is kicking in too often which is a bad thing if the retried transaction runs for say 30min or so.
having to gather the list of objects outside of zope, doing the batching there and calling into zope multiple times - i wanted to avoid that.
i'd like to have all the logic contained in one zope method for the sake of simplicity on the caller's side (most of these are kicked off by cron jobs).
jürgen
take a look at http://pypi.python.org/pypi/affinitic.zamqp maybe it is a solution for you. You can deligate your jobs out of zope and get it back after some time. zamqp then fire an event and you subscribe this event to handle thie results.
with kind regards Maik Derstappen
maik, this looks interesting. just out of curiosity: how would one alert the user that a long running job (say after some 10 minutes) has finished. does the client have to poll? or is there some "magic" push functionality?
robert
hi robert, zamqp has an process that continuously receive new message from his message queue and call a callback method which fire the event, that you can subscribe. see zamqp and carrot docs for more info ;) regards maik -- ======================================================================== Derstappen I T Consulting Tel: +49 ( 341 ) 600 13 0 31 Zope/E-Mail/Backup/Monitoring Mobil: +49 ( 178 ) 861 2 833 M a i k D e r s t a p p e n Fax: +49 ( 180 ) 5 021 121 90 56 H e r l o ß s o h n s t r 12 Email: maik.derstappen@derstappen-it.de 0 4 1 5 5 L e i p z i g Internet: http://www.derstappen-it.de ========================================================================
Am 20.05.2010 21:01, schrieb Maik Derstappen, Derstappen IT:
Am 20.05.2010 20:04, schrieb robert rottermann:
Am 20.05.2010 19:34, schrieb Maik Derstappen, Derstappen IT:
Am 20.05.2010 16:05, schrieb Jürgen Herrmann:
On Thu, 20 May 2010 15:59:58 +0200, Patrick Gerken <do3ccqrv@googlemail.com> wrote:
On Thu, May 20, 2010 at 14:18, Jürgen Herrmann <Juergen.Herrmann@xlhost.de>wrote:
hi there!
as our zope2 zodb grows we experience more and more conflicts with long running transactions. so i'm going to rewrite these methods to commit transactions in batches - here my attempt to tackle this problem. the idea is to get a list of objects to be modified in this transaction, break it down to batches and commit after each batch was processed. to avoid zope's retry-on-conflict machinery and still have a chance to retry per batch transaction commits, the iteration over one batch is guarded by a try:except ConflictError: and a retry logic.
Zope already has its own retry mechanism, maybe you might want to call your long running method multiple times, each time working on the next batch.
in fact zope's retry machinery is kicking in too often which is a bad thing if the retried transaction runs for say 30min or so.
having to gather the list of objects outside of zope, doing the batching there and calling into zope multiple times - i wanted to avoid that.
i'd like to have all the logic contained in one zope method for the sake of simplicity on the caller's side (most of these are kicked off by cron jobs).
jürgen
take a look at http://pypi.python.org/pypi/affinitic.zamqp maybe it is a solution for you. You can deligate your jobs out of zope and get it back after some time. zamqp then fire an event and you subscribe this event to handle thie results.
with kind regards Maik Derstappen
maik, this looks interesting. just out of curiosity: how would one alert the user that a long running job (say after some 10 minutes) has finished. does the client have to poll? or is there some "magic" push functionality?
robert
hi robert,
zamqp has an process that continuously receive new message from his message queue and call a callback method which fire the event, that you can subscribe.
see zamqp and carrot docs for more info ;)
regards maik
thanks maik, this I expected. however (I assume) these are all "things" that happen on the server. how do I push the "all done" event to the client? the user will probably have navigated to some different place. robert
-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1 Patrick Gerken wrote:
On Thu, May 20, 2010 at 14:18, Jürgen Herrmann <Juergen.Herrmann@xlhost.de <mailto:Juergen.Herrmann@xlhost.de>> wrote:
hi there!
as our zope2 zodb grows we experience more and more conflicts with long running transactions. so i'm going to rewrite these methods to commit transactions in batches - here my attempt to tackle this problem. the idea is to get a list of objects to be modified in this transaction, break it down to batches and commit after each batch was processed. to avoid zope's retry-on-conflict machinery and still have a chance to retry per batch transaction commits, the iteration over one batch is guarded by a try:except ConflictError: and a retry logic.
Zope already has its own retry mechanism, maybe you might want to call your long running method multiple times, each time working on the next batch.
There retry applies only for methods called through the web - but not when you use bin/instance run or so. Andreas -----BEGIN PGP SIGNATURE----- Version: GnuPG v1.4.10 (Darwin) Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org/ iEYEARECAAYFAkv1RdkACgkQCJIWIbr9KYzAPACgmuwVdknppy0Ilg0MzC6nzWJA 1UoAn2DW/IVPfIjqfC4x6pG0b1IMObVU =gHpt -----END PGP SIGNATURE-----
On 05/20/2010 06:18 AM, Jürgen Herrmann wrote:
def myLongRunningMethod(self):
BATCH_SIZE = 10 MAX_CONFLICTS = 3
work_items = [some, work, to , do, ...] # list of persistent objects to be modified in this method
idx = 0 while idx< len(work_items): conflicts = 0 try: my_batch = work_items[idx, idx+BATCH_SIZE] for work_item in my_batch: do_some_work(work_item) transaction.commit() except ConflicError: conflicts += 1 if conflicts> MAX_CONFLICTS: raise else: idx += BATCH_SIZE
does this sound like a reasonable approach?
More than that, it looks nearly finished. ;-) The transaction machinery does all the heavy lifting, so you don't need to do much. Issues to fix: - Your code resets the "conflicts" counter in every iteration. You obviously didn't intend that. - You should start each loop with "transaction.begin()" to avoid any side effects of transactions started before the long running method was called. - Writing thorough tests for this kind of code is very important. If you don't, it might be a while before you discover that ConflictError was misspelled. Shane
On Thu, 20 May 2010 18:00:43 -0600, Shane Hathaway <shane@hathawaymix.org> wrote:
On 05/20/2010 06:18 AM, Jürgen Herrmann wrote:
def myLongRunningMethod(self):
BATCH_SIZE = 10 MAX_CONFLICTS = 3
work_items = [some, work, to , do, ...] # list of persistent objects to be modified in this method
idx = 0 while idx< len(work_items): conflicts = 0 try: my_batch = work_items[idx, idx+BATCH_SIZE] for work_item in my_batch: do_some_work(work_item) transaction.commit() except ConflicError: conflicts += 1 if conflicts> MAX_CONFLICTS: raise else: idx += BATCH_SIZE
does this sound like a reasonable approach?
More than that, it looks nearly finished. ;-) The transaction machinery
does all the heavy lifting, so you don't need to do much. Issues to fix:
- Your code resets the "conflicts" counter in every iteration. You obviously didn't intend that. hmm, why not? every batch should be retried up to MAX_CONFLICTS times?
- You should start each loop with "transaction.begin()" to avoid any side effects of transactions started before the long running method was called. cool.
- Writing thorough tests for this kind of code is very important. If you don't, it might be a while before you discover that ConflictError was misspelled. actually python wouldn't compile that code above as ConflicError must resolve to a class at compile time but i get your point, agreed :)
Shane
thanks for your answers and have a nice day everbody. jürgen --
XLhost.de - eXperts in Linux hosting ® <<
XLhost.de GmbH Jürgen Herrmann, Geschäftsführer Boelckestrasse 21, 93051 Regensburg, Germany Geschäftsführer: Volker Geith, Jürgen Herrmann Registriert unter: HRB9918 Umsatzsteuer-Identifikationsnummer: DE245931218 Fon: +49 (0)800 XLHOSTDE [0800 95467833] Fax: +49 (0)800 95467830 WEB: http://www.XLhost.de IRC: #XLhost@irc.quakenet.org
participants (6)
-
Andreas Jung -
Jürgen Herrmann -
Maik Derstappen, Derstappen IT -
Patrick Gerken -
robert rottermann -
Shane Hathaway