[ZODB-Dev] Fwd: Zope/Plone/Zeo Concurrency Issue

[Malthe Borch]

It sounds like a race condition. It might be that you've got a very
low network latency between the web server and web browser such that
there's hardly any time for the invalidation messages to propagate.

But just to be certain it's not an issue with system time, check if
your entire cluster is time-synchronized.

\malthe

On 29 May 2013 08:42, David Glick (Plone) <david.glick at plone.org> wrote:
> On 5/28/13 7:53 PM, Dylan Jay wrote:
>
> Hi,
>
> My colleague is having the issue outlined below. I thought this list might
> be better to give an insight as to what is going on?
>
> Begin forwarded message:
>
> From: Adam Terrey <adam at pretaweb.com>
> Subject: Fwd: Zope/Plone/Zeo Concurrency Issue
> Date: 28 May 2013 5:23:33 PM AEST
> To: Dylan Jay <djay at pretaweb.com>
>
>
>
>
> -------- Original Message --------
> Subject: Zope/Plone/Zeo Concurrency Issue
> Date: Tue, 28 May 2013 16:04:21 +1000
> From: Adam Terrey <adam at pretaweb.com>
> To: Zope at zope.org
>
>
> Hi,
>
> Can someone offer some insight into what might be going on here and perhaps
> how I can debug the following issue?
>
> In Plone there is a request patten used to create content which looks
> like...
>
> (A) GET request to /MySite/createObject?type_name=Document
> Responds with a redirect to a tempory document at a location such as
> /MySite/portal_factory/Document/document.2013-05-28.1878040976/edit
> (B) GET /MySite/portal_factory/Document/document.2013-05-28.1878040976/edit
> Responds with from to set fields in the Document
> (C) POST request to
> /MySite/portal_factory/Document/document.2013-05-28.1878040976/atct_edit
> Responds with a redirect to the final location of the document such as
> /Mysite/my-page
> (D) GET /Mysite/my-page
> Responds with the newly created page
>
> In one of our production systems we are running multiple Zope/Plone
> instances connecting to a Zeo server. And the above patten works about 95%
> of the time. However, sometimes request (D) will respond with 404 Page Not
> Found. My assumptions is that request (C) and (D) are going to different
> instances and somehow the instance handling request (D) does not yet see the
> transaction with completed the page creation.
>
> I've checked that the transaction commit happens before response headers are
> emitted for request (C) - suspecting the case that the browser handles the
> redirect before the transaction is completed - but this is clearly not the
> case from what i can see. The transaction is well committed before response
> headers are returned.
>
> Versions are as follows:
>
> Plone 4.1.3 (4112)
> CMF 2.2.4
> Zope 2.13.10
> Python 2.7.2+ (default, Oct 4 2011, 20:06:09) [GCC 4.6.1]
>
> I have been using the following script to recreate the issue. I seem to be
> able to produce the error quicker if I put server2 under apache-bench load,
> and I have been able to recreate the issue on a local copy of our production
> system but it is a far more rare case.
>
> auth = ('admin', 'admin')
> server1 = 'http://localhost:46101'
> server2 = 'http://localhost:46102'
> site = '/Plone'
>
> import requests
>
> ses = requests.Session()
> ses.auth=auth
>
> for i in range(10):
>
>     # Create tempory Object
>     create_url = server1 + site + '/createObject?type_name=Document'
>     print "GET", create_url
>     res = ses.get (create_url)
>     tempory_obj = res.url.rsplit("/", 1)[0]
>     print "tempory object:", tempory_obj
>
>
>     # Submit data to create object (response will
>     # be the redirect ot the ojbect's location)
>
>     data = {
>                  'id': '',
>                  'title': 'page' + str(i)   ,
>                  'form.button.save': 'Save',
>                  'form.submitted': '1',
>                  "description": "some secriot",
>                  "text": "some text"
>                  }
>
>     post_url = tempory_obj + "/atct_edit"
>     print "POST", post_url
>     res = ses.post(post_url, data=data, allow_redirects=False)
>
>     redirect_location = "/" + res.headers['location'][7:].split('/',1)[1]
>     print 'redirect_location', redirect_location
>
>
>     # Request object from the redirect from server2
>     redirect_url = server2 + redirect_location
>     print "GET", redirect_url
>     res = ses.get(redirect_url)
>
>     if res.status_code != 200:
>         last404 = res
>         print res;
>         break;
>
>     print "---"
>
>
> Any help would be appreciated.
>
>
> I've encountered a problem like this once or twice and had some trouble
> reproducing it consistently, but assume it's due to ZEO invalidation
> messages not making it to the other instance before it serves request D.
>
> You can work around the issue using sticky sessions in your load balancer.
> If you don't want users to be stuck to a particular backend indefinitely you
> can set a cookie with a brief expiration in an ObjectAddedEvent handler and
> use it as the basis for the stickiness.
>
>
> _______________________________________________
> For more information about ZODB, see http://zodb.org/
>
> ZODB-Dev mailing list  -  ZODB-Dev at zope.org
> https://mail.zope.org/mailman/listinfo/zodb-dev
>



--
Au revoir, et tous mes voeux pour un avenir plein de succès et de bonheur ––

Malthe Borch
mborch at gmail.com