RE: [Zope] Recursive aquisition: good or bad?
-----Original Message----- From: Tom Schwaller [mailto:Tom.Schwaller@linux-magazin.de] Sent: Monday, June 07, 1999 5:19 PM To: zope@zope.org Subject: [Zope] Recursive aquisition: good or bad?
Hi all,
during the last 2 weeks I faced a massive performance problem with Zope because of a mistake I made setting relative links instead of absolute ones. I got things like:
/change/change/all/update/search/change
(different menu items clicked one after the other). Zope caches all this paths (is that correct?), so after a while Zope was very slow
Hmm.. this was a performance problem? I would guess it wouldn't be that much of a problem unless your paths got REALLY big.
(many people working with the pages quite a lot, during a hot production phase, so I got many different paths. Clearing the cache or restarting Zope "fixed" the problem)
After that I detected machines calling objects like
/issue1/article1/issue2/article3/news/index_html?newsid=500
although this kind of links do not appear when just surfing around (looks like search engines doing bad things)
So my question is: Is a link like
http://www.zope.org/Documentation/Information/Community/Information/QA
(which works) a feature or a bug of Zope?
It's 'featurish'. This kind of thing falls under the feature/bug anti-duality law. It's not a bug, because it's doing exactly the right thing, it's not a feature, because it's generally not very useful. I can imagine a situation somewhere in which it is useful (don't call my bluff) but I can also easily immagine situations in which it's not very useful at all (below).
The problem is, that nobody can forbid people to call such links, which is the problem I am facing right now (They just do it!). At least things like
/Documentation/Documentation/Documentation/Documentation
You can't forbid people, no, but you can prevent such paths from being constructed from *your* code. I don't think there is any kind of Denial of Server attack possible from this, a mild annoyance at best.
should not be allowed or did I miss some Zope Zen?
I think it should be allowed, and that the Zen be elaborated on a bit. You shouldn't design your site so that these things occur unless you really want it. -Michel
In article <613145F79272D211914B0020AFF640191BF133@gandalf.digicool.com>, Michel Pelletier <michel@digicool.com> wrote:
It's 'featurish'. This kind of thing falls under the feature/bug anti-duality law. It's not a bug, because it's doing exactly the right thing, it's not a feature, because it's generally not very useful. I can imagine a situation somewhere in which it is useful (don't call my bluff) but I can also easily immagine situations in which it's not very useful at all (below).
Definately a feature. Both of the production Zope/Principia installations I'm responsible for now (and one of them is actually several sub-sites that could be separate) use this feature quite a bit. It's effectively a way to do "multiple acquisition" (in the sense of multiple inheritance.) One use is something like this: app/ mode1/ mode2/ mode3/ objects (that implement app) You can have the user access app/... for "regular" access, or app/modeN/... to get a different view on things, by having the mode folders have objects that override the ones in app with different behavior. The real advantage comes when different aspects of the modes are not mutually exclusive. then you can have app/mode1/mode2/... to get both, if things are designed properly. You can also have app/mode2/mode1/... do something different, by having mode1 override mode2 (whichever is last takes precedence... it's like changing the order of base classes in a class definition). In the sites I'm managing, the "modes" are some cases different modes (like "show me this as text", "show me this as html", "provide the ability to edit the data", which can possibly be combined, like html view with editing). In other cases the modes are actually different access levels. app may reference a DTML method "menu" in several places. app can contain a basic definition of menu with the basic choices and Anonymous view permission. You might have a "mode" folder called "author" which contains a method of the same name but with higher security restructions and that provide more choices. by accessing app/author/... instead of app/..., the user is forced to authenticate and will be given more menu choices.
Michel Pelletier wrote:
/change/change/all/update/search/change
(different menu items clicked one after the other). Zope caches all this paths (is that correct?), so after a while Zope was very slow
Hmm.. this was a performance problem? I would guess it wouldn't be that much of a problem unless your paths got REALLY big.
I just looked at a few lines in the logfile, but this recursive links where accumulated during a whole day, so 20-30 steps until maybe 100 are the region we talk about here. Since I corrected the bug, the server runs normally (the few other unusual requests do not disturb it anymore..)
At least things like
/Documentation/Documentation/Documentation/Documentation
You can't forbid people, no, but you can prevent such paths from being constructed from *your* code. I don't think there is any kind of Denial
that's what I tried, :-) but nevertheless I still detected that kind of weird requests
of Server attack possible from this, a mild annoyance at best.
Anobody here to try that with a small script? Death by recursion :-)
should not be allowed or did I miss some Zope Zen?
I think it should be allowed, and that the Zen be elaborated on a bit. You shouldn't design your site so that these things occur unless you really want it.
As far as I can see I did exactly that (exept for some internal stuff, where I did a mistake which generated my problems..) Ty Sarna wrote:
In the sites I'm managing, the "modes" are some cases different modes (like "show me this as text", "show me this as html", "provide the ability to edit the data", which can possibly be combined, like html view with editing). In other cases the modes are actually different access levels. app may reference a DTML method "menu" in several places. app can contain a basic definition of menu with the basic choices and Anonymous view permission. You might have a "mode" folder called "author" which contains a method of the same name but with higher security restructions and that provide more choices. by accessing app/author/... instead of app/..., the user is forced to authenticate and will be given more menu choices.
yes, that's really cool and I had this kind o things in my mind too (frame-version, noframe-version as a simple example. You detailed much more exciting stuff which would certainly be interesting to look at ;-) I just got a strange feeling when the server was knocked down last week with this "side-effect", but you both persuaded me: it's a feature :-) cu and thanks for the insight -- Tom http://www.linux-magazin.de/
Michel Pelletier wrote It's 'featurish'. This kind of thing falls under the feature/bug anti-duality law. It's not a bug, because it's doing exactly the right thing, it's not a feature, because it's generally not very useful. I can imagine a situation somewhere in which it is useful (don't call my bluff) but I can also easily immagine situations in which it's not very useful at all (below).
It's _incredibly_ useful. For my current toy, an SQL data mining tool, I have a folder with a bunch of filters, ('cons_date', 'group_by_refcode', 'add_revenue', &c.) I can then build SQL queries with a URL like datamine/cons_date/group_by_refcode/add_revenue/show and it walks back through the objects and builds the SQL. More on this when it's released (hopefully shortly, but I keep finding new stuff to add :) Anthony
It's _incredibly_ useful. For my current toy, an SQL data mining tool, I have a folder with a bunch of filters, ('cons_date', 'group_by_refcode', 'add_revenue', &c.) I can then build SQL queries with a URL like datamine/cons_date/group_by_refcode/add_revenue/show and it walks back through the objects and builds the SQL. More on this when it's released (hopefully shortly, but I keep finding new stuff to add :)
Anthony
Neeat! (waits with baited breath....) tone.
Anthony Baxter wrote:
Michel Pelletier wrote It's 'featurish'. This kind of thing falls under the feature/bug anti-duality law. It's not a bug, because it's doing exactly the right thing, it's not a feature, because it's generally not very useful. I can imagine a situation somewhere in which it is useful (don't call my bluff) but I can also easily immagine situations in which it's not very useful at all (below).
It's _incredibly_ useful. For my current toy, an SQL data mining tool, I have a folder with a bunch of filters, ('cons_date', 'group_by_refcode', 'add_revenue', &c.) I can then build SQL queries with a URL like datamine/cons_date/group_by_refcode/add_revenue/show and it walks back through the objects and builds the SQL. More on this when it's released (hopefully shortly, but I keep finding new stuff to add :)
wow, this sounds like a killer app for Zope (I hear ESR say "publish early" ;-)) I think I changed my view on the topic by 180 degree now :-) -- Tom http://www.linux-magazin.de/
Tom Schwaller wrote It's _incredibly_ useful. For my current toy, an SQL data mining tool, I have a folder with a bunch of filters, ('cons_date', 'group_by_refcode', 'add_revenue', &c.) I can then build SQL queries with a URL like datamine/cons_date/group_by_refcode/add_revenue/show and it walks back through the objects and builds the SQL. More on this when it's released (hopefully shortly, but I keep finding new stuff to add :)
wow, this sounds like a killer app for Zope (I hear ESR say "publish early" ;-)) I think I changed my view on the topic by 180 degree now :-)
Yeah - everyone I've shown it to here has been happy. I'm very happy, as it means I can avoid writing ad-hoc queries for people who want to produce correlations between hair-color and call duration. There's a couple of really unpleasant bugs in it that I'd like to fix, but one (dealing with Acquisition, and Persistence) in particular I'm a bit stumped by. The system builds up a URL of the datamine contraints (DMCs) to apply. This might look like this: http://burswood:9080/ekorp/datamine_tnt/cons_day/opge1999.05.25/cons_day/con... or indeed might just look like this: http://blah/datamine/cons_day/data_minutes/showcsv The latter is the easier one, and works fine - so I'll talk about it first. cons_day and data_minutes are both DMCs. When applied in a chain like this, they will produce SQL of: SELECT billing.date date, sum(billing.quantity) minutes FROM billing GROUP BY billing.date and then show the output as a CSV file. All well and good, and pretty easy. Each DMC has configured into it what to insert in a select clause, in the from clause, and in the group by clause. So for cons_day, it has a select clause of 'billing.date date', select_from of 'billing', and group_by of 'billing.date'. data_minutes is select: sum(billing.quantity) minutes, select_from: billing, and no group_by. (yes it will insert join clauses if the DMCs span multiple tables) Now, back to the first, more complex example (the one that's causing me pain) http://burswood:9080/ekorp/datamine_tnt/cons_day/opge1999.05.25/cons_day/con... Say we don't just want to get all the dates in the system, but want to only get the ones greater than a certain number? A DMC has a __bobo_traverse__ function, so that a URL fragment of: cons_day/opge1999.05.25/ will cause the cons_day fragment to apply "op"eration of "ge" (greater or equal) to 1999.05.25. Unfortunately in doing this, it modifies 'self', so it writes something to the ZODB each time. What I need is a way to say 'hey, don't pay any attention to me for a minute, I'm just fiddling some stuff, but I don't want it to be persistent'. So that example again, broken down a bit: http://blah/datamine/ -- folder containing all this. cons_day/opge1999.05.25/ -- constrain where date > 1999.05.25 cons_day/ -- but also group by, and select date cons_item/opeqTIME%20USED/ -- pick entries where the 'item' is 'TIME USED' cons_refcode/ -- group by refcode data_minutes/ -- select sum(quantity) showgraph -- display it as a graph this produces SQL of: SELECT billing.date date, accounts.account_group refcode, sum(billing.quantity) minutes FROM billing, accounts WHERE billing.account = accounts.account AND billing.date >= '1999.05.25' and billing.item = 'TIME USED' GROUP BY billing.date, accounts.account_group and then gives me back an image of a graph with the x-axis of dates, y-axis of 'minutes' and multiple lines, (one for each refcode) and a little legend in a box on the left-hand-side. This is about a moderate-complexity query - there's some here that span 5 tables and do all sorts of interesting joins. In more detail, the problem I have is that in the bobo_traverse method, if I get a constraint, I want to store this. The way I do this is to instantiate a new DMC - but one that's non-persistant, populate it, then return it. Unfortunately to get it to acquire everything else it needs, I need to attach it to the 'self' object with something like self.new = DMC_nonP(self.id, self.name, ....) ... set the magic in the new object ... return self.new Unfortunately this self.new assignment makes an entry in the ZODB. Blah. Anyway, I plan to fling a release of some sort out tonight, bug or no bug. Anthony
Any particular reason not to stow the intermediate thing outside of the ZODB - an external method can write a string to disk or a cookie to the client ? Cookie maybe - so you don't need to futz with unique filenames and cleaning up... <199906080956.taa0719-@mbuna.arbhome.com.au> wrote: Original Article: http://www.egroups.com/group/zope/?start=5906
In more detail, the problem I have is that in the bobo_traverse method, if I get a constraint, I want to store this. The way I do this is to instantiate a new DMC - but one that's non-persistant, populate it, then return it. Unfortunately to get it to acquire everything else it needs, I need to attach it to the 'self' object with something like
self.new = DMC_nonP(self.id, self.name, ....) ... set the magic in the new object ... return self.new
Unfortunately this self.new assignment makes an entry in the ZODB. Blah.
Anyway, I plan to fling a release of some sort out tonight, bug or no bug.
Anthony
_______________________________________________ Zope maillist - Zope@zope.org http://www.zope.org/mailman/listinfo/zope
(For developer-specific issues, use the companion list, zope-dev@zope.org - http://www.zope.org/mailman/listinfo/zope-dev )
Anthony Baxter wrote:
so it writes something to the ZODB each time. What I need is a way to say 'hey, don't pay any attention to me for a minute, I'm just fiddling some stuff, but I don't want it to be persistent'.
I greped my Email for a note from Paul which came to my mind, when reading about your problem:
Paul Everitt wrote:
Zope2 will supply a facility to have multiple "Storage Managers" in the same Zope, where some of the Storage Managers are *not* multiple revision types.
I have not more information than this, but it could solve your problem (in the future? :-)) -- Tom http://www.linux-magazin.de/
participants (6)
-
Anthony Baxter -
Michel Pelletier -
rossl＠med.usyd.edu.au -
Tom Schwaller -
Tony.McDonald＠newcastle.ac.uk -
tsarna＠endicor.com