[Zope] Double and Triple Posts

[Andy Yates]

Thanks for the info!  I wish I had posted this question months ago.  

When a user comes to the main page on my site 100+ values are loaded
from a MySQL db and are stored in the SESSION object.  These values are
used to customize everything from text to images on the site.  So, there
are tons of reads and writes to the SESSION object going on all the
time.

The only problem I've seen is when I am sending email (multiple emails),
processing credit cards (double,triple billing), and inserts into the db
(duplicate inserts, key violations).  

1.  Is using the session object like this just a bad approach in
general?  

2.  In the critical places where retries cause a problem can I just
avoid reads and writes to session object?  Will this prevent conflicts
and the retries they cause?

3.  Is there some other sessioning engine that can be used?

Andy 
 
> -----Original Message-----
> From: Chris McDonough [mailto:chrism at plope.com]
> Sent: Tuesday, March 15, 2005 11:39 PM
> To: Andy Yates
> Cc: zope at zope.org
> Subject: Re: [Zope] Double and Triple Posts
> 
> This is almost certainly due to ZODB conflict errors, which are an
> artifact of ZODB using an "optimistic concurrency" strategy for
> concurrent updates.  A lot of times  people will come to the list and
> say "My application is sending two or three mails when it should only
> send one!" and that's a giveaway symptom of the code that sends the
mail
> experiencing a conflict error and retrying.
> 
> Conflict errors are normal to a degree.  If a ZODB conflict error
> happens, Zope retries the request up to three times.  The retry is
> "invisible"; it happens within Zope itself.  Most Zope apps can deal
> with it because they use transactional data storages only, which get
> rolled back, and then the request is replayed.  It makes things
slower,
> but it eventually works.
> 
> Conflict errors are common during sessioning usage (because sessions
use
> ZODB "under the hood" and people who use sessions may not have a lot
of
> experience writing "conflict-resistant" code), but they can be
mitigated
> somewhat.
> 
> The best way to fix these is to change your code to maybe not do as
much
> work, to do its work faster, or to always use transactional data
> storages instead of nontransactional ones (like syslog or MailHost, or
> so forth).  In the case of sessioning, using multiple frames/windows
> that access the session at the same time is a common source of
conflict
> errors too.  Someone "leaning" on their browser's reload button could
> also cause conflict errors to occur.  Also, if you store your session
> data in a ZEO storage, conflict errors are likely to occur more
> frequently due to increased latency when committing transactions.
> 
> Unfortunately, there's no magic wand here and reviewing your code for
> "conflict-resistance" borders on consulting work rather than simple
> maillist help (it's particularly time-consuming and usually requires
> actually running the application), so unless you ditch Zope sessions,
> implement a more conflict resistant sessioning storage, or hire
somebody
> who is intimiately familiar with ZODB, you will likely need to learn
> what causes these errors and how to mitigate them (or at least work
> around them), maybe even by trial and error.
> 
> It's worthwhile to know that any access to a Zope session actually has
> the potential to *write* to the ZODB (even though it may appear you
are
> just reading), so heavy session usage is a prime candidate for
> generating conflict errors.  However, I've reduced them to an
acceptable
> level in my own projects, so I know it's possible.
> 
> See also "Conflict Errors" within
> http://www.plope.com/Books/2_7Edition/Sessions.stx .. this work by
> Michael Dunstan might also be interesting to you..
> http://www.plope.com/Members/dunny/conflicts/view
> 
> HTH,
> 
> - C
> 
> PS: please don't send HTML email to the list, it makes it hard for
> people searching the maillist archives to read threads; thanks!
> 
> On Tue, 2005-03-15 at 22:06, Andy Yates wrote:
> > I have a form that posts to a python script which can take several
> > seconds to run.  Sometimes the script gets called 2 or 3 times in
the
> > same request.  It is not easy to reproduce but it seems to happen at
> > least once every 20 requests.  This is happening on two different
> > versions of Zope and in two completely different projects.
> >
> >
> >
> > Zope only logs a single request.  However, I know it is getting
called
> > more than once because the script writes to syslog.
> >
> >
> >
> > I've packet sniffed the browser and it is only sending one post.  It
> > happens in IE and FireFox but seems to be easier to reproduce in IE.
> > IE will let you click more than once on the submit button.  When I
> > click the submit button twice sometimes I get 3 or 4 runs of the
> > script.  I would expect to get only two.  It seems to happen much
more
> > often when I do this.  (FireFox will only let you press the submit
> > button once)
> >
> >
> >
> > It seems to happen if the script reads and writes values to
> > context.REQUEST.SESSION.  When I take this out it seems to stop.
(It
> > can be hard to reproduce)
> >
> >
> >
> > To reproduce this I created a script that calls an external method
to
> > syslog a message.  Then it reads and writes to
> > context.REQUEST.SESSION.   Then it calls another external method to
> > sleep 15 seconds.   Finally it prints a message and returns.
> >
> >
> >
> >
> >
> > I see this behavior on Zope-2.7.3 and Zope-2.6.1.
> >
> >
> >
> >
> >
> > context.zsyslog('Snooze test' )
> >
> >
> >
> > foo = context.REQUEST.SESSION.get('foo', 1)
> >
> > if foo==1 :
> >
> >     context.REQUEST.SESSION.set('foo', 0)
> >
> >
> >
> > context.REQUEST.SESSION.set('bar',123)
> >
> >
> >
> > context.Snooze(15)
> >
> >
> >
> > print "I'm awake now!"
> >
> > return printed
> >
> >
> >
> > Please let me know if this is happening anywhere else.
> >
> >
> >
> > Andy
> >
> >
> >
> >
> >
> >
> >
______________________________________________________________________
> > _______________________________________________
> > Zope maillist  -  Zope at zope.org
> > http://mail.zope.org/mailman/listinfo/zope
> > **   No cross posts or HTML encoding!  **
> > (Related lists -
> >  http://mail.zope.org/mailman/listinfo/zope-announce
> >  http://mail.zope.org/mailman/listinfo/zope-dev )