Randomness (RE: [Zope-dev] CoreSessionTracking 0.8)
Allright, let me try again. I wish I had a small piece of code to give you so you can reproduce it, but right now you'd have to get our entire CMF-based website. The bug basically manifests itself in that there are two versions of the variable we put in the session (a shopping cart dict). When I browse through the site (not even updating the shopping cart) it'll show one version for some links (1-40) before it switches to show the other, and so on. It looks like the website has two shopping carts that it switches back and forth between. You can see the shopping cart on every page in the website (it's embedded into the template). We were using frames, but I tried it several times without frames now and the bug remains. I even noticed that other variables disappeared randomly as well, e.g., USER_PREF_LANGUAGES which is set by the Localizer, resulting in a key error (I've probably seen 300 pages views, and then suddenly one going back to another page gives a key error?). I'm very curious what could possibly be causing such problems. I thought there might be something wrong in the shared memory between threads, as I can't see anything else changing but the threads (is there a way to display which thread is doing the publishing?). I've seen similar randomness displayed in other situations where I've been reloading pages that would sometimes (same interval, about every 1-40 times) show one character set, and other times another. I think nobody likes to see that kind of randomness. It gives me a very bad stomach feeling. I definately think it's something deeper than a CoreSessionTracking problem. Bye, -- Bjorn Stabell <bjorn@exoweb.net> -----Original Message----- From: Chris McDonough [mailto:chrism@digicool.com] Sent: Wednesday, May 23, 2001 20:34 To: Howard Zhang Cc: chrisme@digicool.com; zope-dev@zope.org; Exoweb Subject: Re: [Zope-dev] CoreSessionTracking 0.8 I remember this problem, but I haven't been able to reproduce it. But maybe it's because I'm not understanding the steps to reproduce it. The sentence "user adds coke to shopping cart and click link to add coke again before request finished" is hard to understand. Can you explain? Are you using frames? Howard Zhang wrote:
Hi The problem about CoreSessionTracking we describe before we can repeat again now. The step is: ( 1 ) User adds Burger to shopping cart ( 2 ) User adds coke to shopping cart and click link to add
coke
again before request finished ( 3 ) The Burger is disappear in shopping cart and just one coke ( not two ) ( 4 ) Repeat the step 2,Burger is back
Anything you could tell me would be helpful.
Regards,
howard
_______________________________________________ Zope-Dev maillist - Zope-Dev@zope.org http://lists.zope.org/mailman/listinfo/zope-dev ** No cross posts or HTML encoding! ** (Related lists - http://lists.zope.org/mailman/listinfo/zope-announce http://lists.zope.org/mailman/listinfo/zope )
It's obvious. This is just Zope's way of telling you not live on hamburgers and coke. -----Original Message----- From: zope-dev-admin@zope.org [mailto:zope-dev-admin@zope.org]On Behalf Of Bjorn Stabell Sent: Thursday, May 24, 2001 12:33 AM To: Chris McDonough; Howard Zhang Cc: zope-dev@zope.org Subject: Randomness (RE: [Zope-dev] CoreSessionTracking 0.8) Allright, let me try again. I wish I had a small piece of code to give you so you can reproduce it, but right now you'd have to get our entire CMF-based website. The bug basically manifests itself in that there are two versions of the variable we put in the session (a shopping cart dict). When I browse through the site (not even updating the shopping cart) it'll show one version for some links (1-40) before it switches to show the other, and so on. It looks like the website has two shopping carts that it switches back and forth between. You can see the shopping cart on every page in the website (it's embedded into the template). We were using frames, but I tried it several times without frames now and the bug remains. I even noticed that other variables disappeared randomly as well, e.g., USER_PREF_LANGUAGES which is set by the Localizer, resulting in a key error (I've probably seen 300 pages views, and then suddenly one going back to another page gives a key error?). I'm very curious what could possibly be causing such problems. I thought there might be something wrong in the shared memory between threads, as I can't see anything else changing but the threads (is there a way to display which thread is doing the publishing?). I've seen similar randomness displayed in other situations where I've been reloading pages that would sometimes (same interval, about every 1-40 times) show one character set, and other times another. I think nobody likes to see that kind of randomness. It gives me a very bad stomach feeling. I definately think it's something deeper than a CoreSessionTracking problem. Bye, -- Bjorn Stabell <bjorn@exoweb.net> -----Original Message----- From: Chris McDonough [mailto:chrism@digicool.com] Sent: Wednesday, May 23, 2001 20:34 To: Howard Zhang Cc: chrisme@digicool.com; zope-dev@zope.org; Exoweb Subject: Re: [Zope-dev] CoreSessionTracking 0.8 I remember this problem, but I haven't been able to reproduce it. But maybe it's because I'm not understanding the steps to reproduce it. The sentence "user adds coke to shopping cart and click link to add coke again before request finished" is hard to understand. Can you explain? Are you using frames? Howard Zhang wrote:
Hi The problem about CoreSessionTracking we describe before we can repeat again now. The step is: ( 1 ) User adds Burger to shopping cart ( 2 ) User adds coke to shopping cart and click link to add
coke
again before request finished ( 3 ) The Burger is disappear in shopping cart and just one coke ( not two ) ( 4 ) Repeat the step 2,Burger is back
Anything you could tell me would be helpful.
Regards,
howard
_______________________________________________ Zope-Dev maillist - Zope-Dev@zope.org http://lists.zope.org/mailman/listinfo/zope-dev ** No cross posts or HTML encoding! ** (Related lists - http://lists.zope.org/mailman/listinfo/zope-announce http://lists.zope.org/mailman/listinfo/zope )
_______________________________________________ Zope-Dev maillist - Zope-Dev@zope.org http://lists.zope.org/mailman/listinfo/zope-dev ** No cross posts or HTML encoding! ** (Related lists - http://lists.zope.org/mailman/listinfo/zope-announce http://lists.zope.org/mailman/listinfo/zope )
What we experience with CoreSessionTracking: We have a manage mode in the Kontentor CMS hacky thing (no, it is NOT a product yet ;-)). It works well for some time (i.e. if I click on it, the system switches to manage mode and stays in manage mode until I click again), but from time to time it is just reset (quite randomly, too) to the non-manage mode. The click just sets a session variable. Haven't got any deeper with it yet, but the session seems not to be really reliable ... Joachim
These "it appears CST is unstable" reports are helpful to an extent (from Bjorn, Joachim, and Howard), as it lets me know that something needs to be done to CST. However, a much more helpful report would be one which provides a repeatable test case which invariably reproduces the problem instead of one which states the symptoms and effects of the problem. At this point, I understand that there *is* at least one problem, and I understand what constitutes the problem(s), but I can't seem to reproduce it/them. Since I'm not able to reliably reproduce the error state, I can't find the bugs, and this means I can't fix them. It doesn't help much that I have almost no time to devote to this at the moment, either. If anyone can come up with a repeatable test case which uses CST "out of the box" in a fresh Zope 2.3.X install which demonstrates this failure, I would be most grateful, and it would cause the problem to be fixed much faster! Shane has come up with a way to fix the "None has no attribute 'load'" bug that has plagued CST since the beginning, so maybe I'll make a release next week with the fix in it to see if the problems that are being reported are related to that. I will try some more stuff in the meantime, although I'm at a loss currently as to what could be causing these symptoms. :-( - C Joachim Werner wrote:
What we experience with CoreSessionTracking:
We have a manage mode in the Kontentor CMS hacky thing (no, it is NOT a product yet ;-)). It works well for some time (i.e. if I click on it, the system switches to manage mode and stays in manage mode until I click again), but from time to time it is just reset (quite randomly, too) to the non-manage mode. The click just sets a session variable. Haven't got any deeper with it yet, but the session seems not to be really reliable ...
Joachim
_______________________________________________ Zope-Dev maillist - Zope-Dev@zope.org http://lists.zope.org/mailman/listinfo/zope-dev ** No cross posts or HTML encoding! ** (Related lists - http://lists.zope.org/mailman/listinfo/zope-announce http://lists.zope.org/mailman/listinfo/zope )
On Thu, 24 May 2001, Chris McDonough wrote:
These "it appears CST is unstable" reports are helpful to an extent (from Bjorn, Joachim, and Howard), as it lets me know that something needs to be done to CST. However, a much more helpful report would be one which provides a repeatable test case which invariably reproduces the problem instead of one which states the symptoms and effects of the problem.
Just to add myself to the list, I too am having problems with CoreSessionTracking :( I am trying to find a test case for the problem, but I really can't replicate it. It first I thought it was a cookie issue, but I am now noting down the session id generated and it stays the same even when my session data is lost, so I don't think it is that. what I experience: I add an item to my shopping cart system and it shows up fine, all state is maintained. If I however leave the page and not touch anything for a couple of minutes and then reload the page, the cart is now empty. I know this is not that helpful, but I'm trying to tie it down myself! One thing that I do need to check though -- I am right in assuming I can insert complex objects into the SessionData right? I have an 'Order' class that amongst other things has a dict containing instances of a 'Product' class. The Order instance is stored in the SessionData. As I said it appears to work fine and I can add and delete items from my Order fine. Just it randomly looses it now and then. -Matt -- Matt Hamilton matth@netsight.co.uk Netsight Internet Solutions, Ltd. Business Vision on the Internet http://www.netsight.co.uk +44 (0)117 9090901 Web Hosting | Web Design | Domain Names | Co-location | DB Integration
Matt Hamilton wrote:
Just to add myself to the list, I too am having problems with CoreSessionTracking :( I am trying to find a test case for the problem, but I really can't replicate it. It first I thought it was a cookie issue, but I am now noting down the session id generated and it stays the same even when my session data is lost, so I don't think it is that.
what I experience: I add an item to my shopping cart system and it shows up fine, all state is maintained. If I however leave the page and not touch anything for a couple of minutes and then reload the page, the cart is now empty. I know this is not that helpful, but I'm trying to tie it down myself!
Bummer. How long is the session data container timeout set for? Are you sure you're just not exceeding the timeout?
One thing that I do need to check though -- I am right in assuming I can insert complex objects into the SessionData right? I have an 'Order' class that amongst other things has a dict containing instances of a 'Product' class. The Order instance is stored in the SessionData. As I said it appears to work fine and I can add and delete items from my Order fine. Just it randomly looses it now and then.
Yes, any sort of object can go into session data... there are some caveats documented in the helpfile (for instance, acquisition-wrapped objects shouldn't be stored in session data), but otherwise anything's game. - C
On Fri, 25 May 2001, Chris McDonough wrote:
Bummer. How long is the session data container timeout set for? Are you sure you're just not exceeding the timeout?
The session timeout is 120 minutes. I am using an external SessionDataComainter stored in the normal undo-able ZODB (I'm not expecting that much traffic to it). Looking into it further I don't think it is a CST fault :) I think I have fixed the problem. It was me not setting _p_changed on an object with a dict after adding items to the dict. Hence the session wasn't being lost, just the conents of the cart itself were not very persistent :) I'll be able to confirm this later today.
Yes, any sort of object can go into session data... there are some caveats documented in the helpfile (for instance, acquisition-wrapped objects shouldn't be stored in session data), but otherwise anything's game.
I can't find anything about acquisition-wrapped objects in the helpfile, what is the problem with them? -Matt -- Matt Hamilton matth@netsight.co.uk Netsight Internet Solutions, Ltd. Business Vision on the Internet http://www.netsight.co.uk +44 (0)117 9090901 Web Hosting | Web Design | Domain Names | Co-location | DB Integration
Matt Hamilton wrote:
Looking into it further I don't think it is a CST fault :) I think I have fixed the problem. It was me not setting _p_changed on an object with a dict after adding items to the dict. Hence the session wasn't being lost, just the conents of the cart itself were not very persistent :)
Gotcha... (I can't tell you how many times I've done that ;-)
Yes, any sort of object can go into session data... there are some caveats documented in the helpfile (for instance, acquisition-wrapped objects shouldn't be stored in session data), but otherwise anything's game.
I can't find anything about acquisition-wrapped objects in the helpfile, what is the problem with them?
Eek, you're right. This is a funny situation. I had thought I had documented this. But first I need to figure out what to document. Maybe that's why I didn't. The ZODB in general does not like to store acquisition-wrapped objects. It's usually the case in CST that you're prevented from doing this by the fact that you're generally using a mounted storage, and acqusition-wrapped objects in the context of Zope usually contain object references from the "main" storage (raising the InvalidObjectReference error documented in the helpfile). However, in cases like yours, where you're using a session data container that's *in* the "main" storage, you'll get past that hurdle (because your objects can't/don't contain references to another database). But you *should* be prevented from storing acqusition-wrapped objects in any database because it doesn't make sense to preserve an acqusition wrapper anywhere, they're completely ephemeral. But of course I just tried it, and it didn't raise any error. ;-) Hee hee. This means one of two things: 1. ZODB was "fixed" at some point to unwrap acquisition-wrapped objects before putting them in storage. This is likely. I dimly remember something like this happening, although I can't find it documented in any HISTORY or CHANGES files (not altogether surprising). 2. ZODB is blithely storing acquistion wrappers. If the answer is 1, everything's fine, and ignore my claim about not being able to pass acquisition-wrapped objects into session storage. Know that if you move to a mounted database in the future, and your code passes in objects pulled out of "main" Zope storage to session storage, that code will likely break. If the answer is 2, everything's not so fine. I'll likely need to change the CST code to unwrap acquisition-wrapped objects before storing them, just to head potential problems off at the pass. I just did a preliminary test, and it appears that it *is* storing acquisition-wrapped objects. This is bad if it's true. :-( In the meantime, to be safe, before you store an object in session storage, unwrap it by getting its aq_base, e.g.: ob = getattr(ob, 'aq_base', on) or use the utility function aq_base from Acquistion to safely do the same: from Acquisition import aq_base ob = aq_base(ob) Ah the bugs keep on piling on over here, - C
-Matt
-- Matt Hamilton matth@netsight.co.uk Netsight Internet Solutions, Ltd. Business Vision on the Internet http://www.netsight.co.uk +44 (0)117 9090901 Web Hosting | Web Design | Domain Names | Co-location | DB Integration
Chris McDonough wrote:
If the answer is 2, everything's not so fine. I'll likely need to change the CST code to unwrap acquisition-wrapped objects before storing them, just to head potential problems off at the pass. I just did a preliminary test, and it appears that it *is* storing acquisition-wrapped objects. This is bad if it's true. :-(
Note that I just changed CST to unwrap aq-wrapped objects just in case, because what little testing I've been able to do says that aquisition wrappers are being stored in ZODB. I'm sure someone's going to come along and tell me that's the dumbest thing they've ever heard, but the changes don't hurt anything if I'm wrong. The requisite changes to the SessionData class (in the SessionData.py module) are: from: def __setitem__(self, k, v): # if the key or value is a persistent instance, # set up its _p_jar immediately if hasattr(v, '_p_jar') and v._p_jar is None: v._p_jar = self._p_jar v._p_changed = 1 if hasattr(k, '_p_jar') and k._p_jar is None: k._p_jar = self._p_jar k._p_changed = 1 self._container[k] = v self._p_changed = 1 to: def __setitem__(self, k, v): # if the key or value is a persistent instance, # set up its _p_jar immediately if hasattr(v, '_p_jar') and v._p_jar is None: v._p_jar = self._p_jar v._p_changed = 1 if hasattr(k, '_p_jar') and k._p_jar is None: k._p_jar = self._p_jar k._p_changed = 1 # unwrap this thing if it's wrapped from Acquisition import aq_base k = aq_base(k) v = aq_base(v) self._container[k] = v self._p_changed = 1 weeping (although grateful for the exchange), - C
However, a much more helpful report would be one which provides a repeatable test case which invariably reproduces the problem instead of one which states the symptoms and effects of the problem.
I know, but that's why the errors are called "random": They are not easy to replicate ...
Joachim Werner wrote:
However, a much more helpful report would be one which provides a repeatable test case which invariably reproduces the problem instead of one which states the symptoms and effects of the problem.
I know, but that's why the errors are called "random": They are not easy to replicate ...
I understand. I just wrote a test case that emulates many visitors firing off sessions and making changes to shared data objects simultaneously. It even emulates aborted connections. Unfortunately, I see nothing out of the ordinary. :-( This is very frustrating. - C
Chris McDonough wrote:
I just wrote a test case that emulates many visitors firing off sessions and making changes to shared data objects simultaneously. It even emulates aborted connections. Unfortunately, I see nothing out of the ordinary. :-( This is very frustrating.
Maybe the problem folks are experiencing with CST are related to the "None has no attribute 'load'" bug that pops up every so often using a mounted storage (RAM-based session data containers use a mounted storage internally). If there's any chance that you can turn the STUPID_LOG_FILE on, if the problem happens while you're using the site, it would be helpful to be able to correlate (or not correlate) the problems folks are having with any messages in the error log. -C
To further try to track this problem down, I've developed a "burger, fries, coke" application in honor of this thread. So far, I've eaten 84 burgers, 31 cokes, and 42 fries. I can't seem to break this thing, and I'm not hungry anymore. :-( I just wanted to throw this out there. I'm certain this isn't the problem because it's too simple, but just in case... How many session data managers do you (Howard, Michael, Joachim, and.. I forget who else is having problems) have? Just one, I assume. It's not possible that different DTML pages in your application call out to *different* session data managers, is it? Maybe via unanticipated acquisition? I know the answer is probably no, but I need to ask, because the symptoms are consistent with this kind of misconfiguration. Questions (assuming the previous question was a throwaway): Is there something in the standard_html_header that sets request variables out of session data? Or is this done via an access rule? Or is the session data logic just embedded within every page? Is everybody using cookies? Is everybody using the RAM-based session data container? How long is the data container timeout? Is this too many questions already? ;-) stuffed-ly y'rs, - C Chris McDonough wrote:
Chris McDonough wrote:
I just wrote a test case that emulates many visitors firing off sessions and making changes to shared data objects simultaneously. It even emulates aborted connections. Unfortunately, I see nothing out of the ordinary. :-( This is very frustrating.
Maybe the problem folks are experiencing with CST are related to the "None has no attribute 'load'" bug that pops up every so often using a mounted storage (RAM-based session data containers use a mounted storage internally). If there's any chance that you can turn the STUPID_LOG_FILE on, if the problem happens while you're using the site, it would be helpful to be able to correlate (or not correlate) the problems folks are having with any messages in the error log.
-C
_______________________________________________ Zope-Dev maillist - Zope-Dev@zope.org http://lists.zope.org/mailman/listinfo/zope-dev ** No cross posts or HTML encoding! ** (Related lists - http://lists.zope.org/mailman/listinfo/zope-announce http://lists.zope.org/mailman/listinfo/zope )
Chris McDonough wrote:
To further try to track this problem down, I've developed a "burger, fries, coke" application in honor of this thread. So far, I've eaten 84 burgers, 31 cokes, and 42 fries. I can't seem to break this thing, and I'm not hungry anymore. :-(
Sorry, but I had to :-) http://www.zopezen.org/Zope/Quotes *grinz* Chris
I know, but that's why the errors are called "random": They are not easy to replicate ...
I understand.
I don't know if this helps a bit: It's the code used in KONTENTOR for putting the session into the REQUEST namespace. I think it is not the most elegant way of doing this, but it seems to work, except for the times it doesn't ;-) The actual problem I can trace (we don't have a shopping cart) is that the manage_content variable falls back to "NO" without a reason. Please tell me that the problem is in our code, not in the CST: <dtml-let data="SESSION.getSessionData()"> <dtml-unless "data.has_key('manage_content')"> <dtml-call "data.set('manage_content','NO')"> </dtml-unless> <dtml-in "REQUEST.form.keys()"> <dtml-let si=sequence-item> <dtml-unless "si=='file'"> <dtml-call "data.set(si, REQUEST[si])"> </dtml-unless> </dtml-let> </dtml-in> <dtml-in "data.keys()"> <dtml-let si=sequence-item> <dtml-call "REQUEST.set(si, data[si])"> </dtml-let> </dtml-in> </dtml-let> Cheers Joachim
Hi Joachim, I'm confused as to the utility of this bit of code. You loop over the items in the REQUEST.form dict, putting them all into session storage except for 'file'. Then you loop over all the items in the session data object, putting them into the REQUEST.other dict. Why the first step? What if I did: http://yoursite/amethod?bag=YES&farb=FOO I'd have both bag and farb in my session data object? Would this be sufficient instead: <dtml-let data="SESSION.getSessionData()"> manage="REQUEST.get('manage_content') or data.get('manage_content')"> <dtml-unless manage> <dtml-call "data.set('manage_content','NO')"> </dtml-unless> <dtml-in "data.keys()"> <dtml-let si=sequence-item> <dtml-call "REQUEST.set(si, data[si])"> </dtml-let> </dtml-in> </dtml-let> I'm also confused why you're using YES and NO to represent a boolean instead of just checking that a key exists or is true. Aside from that, the code looks fine to me. Your problem is consistent with Bjorn's iasmuch as it seems that your session data object gets cleared inappropriately... but I can't make this happen on my copy of Zope and CST. :-( Maybe after I look at Bjorn's site, I'll have a few more clues. Bjorn also provided some additional data that is helpful as well. I'll try to synthesize this stuff and come up with some sort of answer next week (I'll be away for the weekend). - C Joachim Werner wrote:
I know, but that's why the errors are called "random": They are not easy to replicate ...
I understand.
I don't know if this helps a bit: It's the code used in KONTENTOR for putting the session into the REQUEST namespace. I think it is not the most elegant way of doing this, but it seems to work, except for the times it doesn't ;-)
The actual problem I can trace (we don't have a shopping cart) is that the manage_content variable falls back to "NO" without a reason. Please tell me that the problem is in our code, not in the CST:
<dtml-let data="SESSION.getSessionData()"> <dtml-unless "data.has_key('manage_content')"> <dtml-call "data.set('manage_content','NO')"> </dtml-unless> <dtml-in "REQUEST.form.keys()"> <dtml-let si=sequence-item> <dtml-unless "si=='file'"> <dtml-call "data.set(si, REQUEST[si])"> </dtml-unless> </dtml-let> </dtml-in> <dtml-in "data.keys()"> <dtml-let si=sequence-item> <dtml-call "REQUEST.set(si, data[si])"> </dtml-let> </dtml-in> </dtml-let>
Cheers Joachim
_______________________________________________ Zope-Dev maillist - Zope-Dev@zope.org http://lists.zope.org/mailman/listinfo/zope-dev ** No cross posts or HTML encoding! ** (Related lists - http://lists.zope.org/mailman/listinfo/zope-announce http://lists.zope.org/mailman/listinfo/zope )
<dtml-let data="SESSION.getSessionData()"> manage="REQUEST.get('manage_content') or data.get('manage_content')"> <dtml-unless manage> <dtml-call "data.set('manage_content','NO')"> </dtml-unless> <dtml-in "data.keys()"> <dtml-let si=sequence-item> <dtml-call "REQUEST.set(si, data[si])"> </dtml-let> </dtml-in> </dtml-let>
This will work for the "manage_content", but I want to make the whole request persistent. E.g., box states, search filters, and sequence-start variables for batching are all made persistent ...
I'm also confused why you're using YES and NO to represent a boolean instead of just checking that a key exists or is true.
Just consider it a bad habit ;-) -- I had some trouble working with booleans some time ago, so I switched to this more explicit version.
participants (6)
-
Albert Langer -
Bjorn Stabell -
Chris McDonough -
Chris Withers -
Joachim Werner -
Matt Hamilton