[ZODB-Dev] Fwd: Zope/Plone/Zeo Concurrency Issue

[David Glick (Plone)]

On 5/28/13 7:53 PM, Dylan Jay wrote:
> Hi,
>
> My colleague is having the issue outlined below. I thought this list 
> might be better to give an insight as to what is going on?
>
> Begin forwarded message:
>
>> *From: *Adam Terrey <adam at pretaweb.com <mailto:adam at pretaweb.com>>
>> *Subject: **Fwd: Zope/Plone/Zeo Concurrency Issue*
>> *Date: *28 May 2013 5:23:33 PM AEST
>> *To: *Dylan Jay <djay at pretaweb.com <mailto:djay at pretaweb.com>>
>>
>>
>>
>>
>> -------- Original Message --------
>> Subject: 	Zope/Plone/Zeo Concurrency Issue
>> Date: 	Tue, 28 May 2013 16:04:21 +1000
>> From: 	Adam Terrey <adam at pretaweb.com>
>> To: 	Zope at zope.org
>>
>>
>>
>> Hi,
>>
>> Can someone offer some insight into what might be going on here and 
>> perhaps how I can debug the following issue?
>>
>> In Plone there is a request patten used to create content which looks 
>> like...
>>
>>   * (A) GET request to /MySite/createObject?type_name=Document
>>     Responds with a redirect to a tempory document at a location such
>>     as
>>     /MySite/portal_factory/Document/document.2013-05-28.1878040976/edit
>>   * (B) GET
>>     /MySite/portal_factory/Document/document.2013-05-28.1878040976/edit
>>     Responds with from to set fields in the Document
>>   * (C) POST request to
>>     /MySite/portal_factory/Document/document.2013-05-28.1878040976/atct_edit
>>     Responds with a redirect to the final location of the document
>>     such as /Mysite/my-page
>>   * (D) GET /Mysite/my-page
>>     Responds with the newly created page
>>
>> In one of our production systems we are running multiple Zope/Plone 
>> instances connecting to a Zeo server. And the above patten works 
>> about 95% of the time. However, sometimes request (D) will respond 
>> with 404 Page Not Found. My assumptions is that request (C) and (D) 
>> are going to different instances and somehow the instance handling 
>> request (D) does not yet see the transaction with completed the page 
>> creation.
>>
>> I've checked that the transaction commit happens before response 
>> headers are emitted for request (C) - suspecting the case that the 
>> browser handles the redirect before the transaction is completed - 
>> but this is clearly not the case from what i can see. The transaction 
>> is well committed before response headers are returned.
>>
>> Versions are as follows:
>>
>>   * Plone 4.1.3 (4112)
>>   * CMF 2.2.4
>>   * Zope 2.13.10
>>   * Python 2.7.2+ (default, Oct 4 2011, 20:06:09) [GCC 4.6.1]
>>
>> I have been using the following script to recreate the issue. I seem 
>> to be able to produce the error quicker if I put server2 under 
>> apache-bench load, and I have been able to recreate the issue on a 
>> local copy of our production system but it is a far more rare case.
>>
>> auth = ('admin', 'admin')
>> server1 = 'http://localhost:46101'
>> server2 = 'http://localhost:46102'
>> site = '/Plone'
>>
>> import requests
>>
>> ses = requests.Session()
>> ses.auth=auth
>>
>> for i in range(10):
>>
>>     # Create tempory Object
>>     create_url = server1 + site + '/createObject?type_name=Document'
>>     print "GET", create_url
>>     res = ses.get (create_url)
>>     tempory_obj = res.url.rsplit("/", 1)[0]
>>     print "tempory object:", tempory_obj
>>
>>
>>     # Submit data to create object (response will
>>     # be the redirect ot the ojbect's location)
>>
>>     data = {
>>                  'id': '',
>>                  'title': 'page' + str(i)   ,
>>                  'form.button.save': 'Save',
>>                  'form.submitted': '1',
>>                  "description": "some secriot",
>>                  "text": "some text"
>>                  }
>>
>>     post_url = tempory_obj + "/atct_edit"
>>     print "POST", post_url
>>     res = ses.post(post_url, data=data, allow_redirects=False)
>>
>>     redirect_location = "/" + res.headers['location'][7:].split('/',1)[1]
>>     print 'redirect_location', redirect_location
>>
>>
>>     # Request object from the redirect from server2
>>     redirect_url = server2 + redirect_location
>>     print "GET", redirect_url
>>     res = ses.get(redirect_url)
>>
>>     if res.status_code != 200:
>>         last404 = res
>>         print res;
>>         break;
>>
>>     print "---"
>>
>>
>> Any help would be appreciated.
>>

I've encountered a problem like this once or twice and had some trouble 
reproducing it consistently, but assume it's due to ZEO invalidation 
messages not making it to the other instance before it serves request D.

You can work around the issue using sticky sessions in your load 
balancer. If you don't want users to be stuck to a particular backend 
indefinitely you can set a cookie with a brief expiration in an 
ObjectAddedEvent handler and use it as the basis for the stickiness.

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.zope.org/pipermail/zodb-dev/attachments/20130528/1476bf82/attachment-0001.html>