Zope does dynamic, Linux kernel does static pages?
Hey Zopesters - Just thought I'd point out something thart came up on the linux kernel mailing list this week. Apparently, as part of the aftermath of that whole Mindcraft 'benchmark' fiasco, it came to the kernel developers' attention that under NT, IIS has special status - it has some sneaky system calls that end up with the NT kernel effectively doing all the serving of static HTML pages. This had been discussed some time ago as an interesting idea under Linux, but this time around, someone started implementing it. It's still alpha, but it's at: http://www.fenrus.demon.nl This being Linux, it's not specific to any one HTTP server, it'll cooperate with anything that listens on a port. Basically, it'll handle serving the simple static pages directly from the filesystem as fast as possible, and pass anything else on to the userland application. Linus Torvalds himself seems to like the idea, and mentioned: So you could even use a pretty much unmodified apache (or other) web server to do all the non-static cases - although it would probably imply that the user-level deamon would be better off being optimized for the "uncommon" case because it never sees the common case. Hmm, a user-level deamon optimized for the "uncommon" non-static case: do I hear Zserver calling? Admittedly, it's still alpha, but anyone playing with Zope2 on linux might want to look at this. To use it effectively, you'd need to use absolute URLS to static pages (and images) outside the zope ORB, so the module can find them. In the long run, some sort of static 'shadow' directory structure, perhaps managed via one of the new ORB3 alternative storage managers, could allow the Zope developer to mark objects as 'static pages' that get pushed out to files, so the kernel daemon could see them. Hmm, this might even allow Zope's cache to become accessible to the kernel deamon? Does Zope do any caching of rendered of pages? If it did, That'd be a speed win for 'quasi-static' things that change slowly compared to typical user sessions. In fact, the caching would be a win no matter how you serve pages. My random thoughts for today, Ross -- Ross J. Reedstrom, Ph.D., <reedstrm@rice.edu> NSBRI Research Scientist/Programmer Computer and Information Technology Institute Rice University, 6100 S. Main St., Houston, TX 77005
"Ross J. Reedstrom" wrote:
Hey Zopesters - Just thought I'd point out something thart came up on the linux kernel mailing list this week. Apparently, as part of the aftermath of that whole Mindcraft 'benchmark' fiasco, it came to the kernel developers' attention that under NT, IIS has special status - it has some sneaky system calls that end up with the NT kernel effectively doing all the serving of static HTML pages. This had been discussed some time ago as an interesting idea under Linux, but this time around, someone started implementing it. It's still alpha, but it's at:
This being Linux, it's not specific to any one HTTP server, it'll cooperate with anything that listens on a port. Basically, it'll handle serving the simple static pages directly from the filesystem as fast as possible, and pass anything else on to the userland application.
Linus Torvalds himself seems to like the idea, and mentioned:
So you could even use a pretty much unmodified apache (or other) web server to do all the non-static cases - although it would probably imply that the user-level deamon would be better off being optimized for the "uncommon" case because it never sees the common case.
Hmm, a user-level deamon optimized for the "uncommon" non-static case: do I hear Zserver calling?
Admittedly, it's still alpha, but anyone playing with Zope2 on linux might want to look at this. To use it effectively, you'd need to use absolute URLS to static pages (and images) outside the zope ORB, so the module can find them.
In the long run, some sort of static 'shadow' directory structure, perhaps managed via one of the new ORB3 alternative storage managers, could allow the Zope developer to mark objects as 'static pages' that get pushed out to files, so the kernel daemon could see them. Hmm, this might even allow Zope's cache to become accessible to the kernel deamon? Does Zope do any caching of rendered of pages? If it did, That'd be a speed win for 'quasi-static' things that change slowly compared to typical user sessions. In fact, the caching would be a win no matter how you serve pages.
This sounds interesting. One idea I had that would fit with this was to introduce some caching in ZServer. Certain objects, especially images and files might get cached by ZServer, so that it can avoid calls to the object system when they are requested. Presumably, the ZServer cache could then play with this Linux kernal thing. ZServer may manage it's cache using standard HTTP headers, or it might do something more Zope specific, that perhaps uses knowledge of the way the object system works. If someone wanted a really easy place to start, one might add a cache to ZServer that always cached URL paths's beginning with "p_" or "misc_", since these point to static images (and possibly other resources) defined at the Python level. Jim -- Jim Fulton mailto:jim@digicool.com Technical Director (888) 344-4332 Python Powered! Digital Creations http://www.digicool.com http://www.python.org Under US Code Title 47, Sec.227(b)(1)(C), Sec.227(a)(2)(B) This email address may not be added to any commercial mail list with out my permission. Violation of my privacy with advertising or SPAM will result in a suit for a MINIMUM of $500 damages/incident, $1500 for repeats.
To take these musings a step further. I can imagine a mechanism by which: - Certain objects could inform ZServer of object IDs of the objects that constitute thier "source" data, - Zserver registers with the database to be notified when objects change (by object ID), This could allow ZServer to do some clever caching even for some relatively dynamic objects. (Note that figuring out what the "source" objects for an object are could be pretty tricky. While some objects, like Images, Files, and most DTMLDocuments are their own source, other objects, like DTMLMethods might depend on thousands of objects.) Jim -- Jim Fulton mailto:jim@digicool.com Technical Director (888) 344-4332 Python Powered! Digital Creations http://www.digicool.com http://www.python.org Under US Code Title 47, Sec.227(b)(1)(C), Sec.227(a)(2)(B) This email address may not be added to any commercial mail list with out my permission. Violation of my privacy with advertising or SPAM will result in a suit for a MINIMUM of $500 damages/incident, $1500 for repeats.
There may be a simpler approach, less clever but perhaps faster to implement. Have objects give ZServer a "time to live" value, which is set as a property on the object, but whose value is associated in the cache with the request URL. So if objects foo and bar have dtml method baz, and baz has a time-to-live of say 5 minutes, then the results of calling /foo/baz and /bar/baz should each be cached for a maximum of 5 minutes from the call. This value could be acquired, allowing the source object (or its class) to set it. While not perfect, this could do quite well for a lot of applications - especially of the portal/squishdot variety. At 05:39 PM 6/10/99 +0000, Jim Fulton wrote:
To take these musings a step further. I can imagine a mechanism by which:
- Certain objects could inform ZServer of object IDs of the objects that constitute thier "source" data,
- Zserver registers with the database to be notified when objects change (by object ID),
This could allow ZServer to do some clever caching even for some relatively dynamic objects.
(Note that figuring out what the "source" objects for an object are could be pretty tricky. While some objects, like Images, Files, and most DTMLDocuments are their own source, other objects, like DTMLMethods might depend on thousands of objects.)
Do you find nesting <!--#with "_.namespace()"--> tags too verbose? Don't like cluttering your REQUEST object with temporary variables? Want to get at weirdly named DTML variables from Python expressions without having to play _[] and _.render() tricks? Then check out: http://telecommunity.com/DT_Let.py It's a new tag that lets you create blocks like: <!--#in "1,2,3,4"--> <!--#let num=sequence-item index=sequence-index result="num*index"--> <!--#var num--> * <!--#var index--> = <!--#var result--> <!--#/let--> <!--#/in--> Which yields: 1 * 0 = 0 2 * 1 = 2 3 * 2 = 6 4 * 3 = 12 Notice that quoted arguments are treated as expressions, and unquoted arguments are treated as DTML names. Evaluation is in sequence with the result of earlier assignments available to later ones. Later assignments can also override earlier ones, which can be helpful for longer step-by-step calculations. The variables thus set are in effect for the life of the <!--#let--> block. To use this wonderful, lifesaving tag, just stick the module somewhere that Python can find it, and put an "import DT_Let" somewhere in your code where it will get executed before you need it. Permission is hereby granted to the wonderful folks at DC to incorporate this into Zope. The point of making this was to reduce verbosity for heavy DTML scripting, though, so if you do put it in, *please*, *please* don't rename it to #with-namespace or #with-variable-values or anything like that, okay? :) Thanks.
"Phillip J. Eby" wrote:
Do you find nesting <!--#with "_.namespace()"--> tags too verbose? Don't like cluttering your REQUEST object with temporary variables? Want to get at weirdly named DTML variables from Python expressions without having to play _[] and _.render() tricks? Then check out:
http://telecommunity.com/DT_Let.py
It's a new tag that lets you create blocks like:
<!--#in "1,2,3,4"--> <!--#let num=sequence-item index=sequence-index result="num*index"--> <!--#var num--> * <!--#var index--> = <!--#var result--> <!--#/let--> <!--#/in-->
Which yields:
1 * 0 = 0 2 * 1 = 2 3 * 2 = 6 4 * 3 = 12
Notice that quoted arguments are treated as expressions, and unquoted arguments are treated as DTML names. Evaluation is in sequence with the result of earlier assignments available to later ones. Later assignments can also override earlier ones, which can be helpful for longer step-by-step calculations. The variables thus set are in effect for the life of the <!--#let--> block.
To use this wonderful, lifesaving tag, just stick the module somewhere that Python can find it, and put an "import DT_Let" somewhere in your code where it will get executed before you need it. Permission is hereby granted to the wonderful folks at DC to incorporate this into Zope. The point of making this was to reduce verbosity for heavy DTML scripting, though, so if you do put it in, *please*, *please* don't rename it to #with-namespace or #with-variable-values or anything like that, okay? :) Thanks.
I like it. May we include it in the Zope distribution? Jim -- Jim Fulton mailto:jim@digicool.com Technical Director (888) 344-4332 Python Powered! Digital Creations http://www.digicool.com http://www.python.org Under US Code Title 47, Sec.227(b)(1)(C), Sec.227(a)(2)(B) This email address may not be added to any commercial mail list with out my permission. Violation of my privacy with advertising or SPAM will result in a suit for a MINIMUM of $500 damages/incident, $1500 for repeats.
At 07:13 PM 6/10/99 +0000, Jim Fulton wrote:
"Phillip J. Eby" wrote:
To use this wonderful, lifesaving tag, just stick the module somewhere that Python can find it, and put an "import DT_Let" somewhere in your code where it will get executed before you need it. Permission is hereby granted to the wonderful folks at DC to incorporate this into Zope. The point of making this was to reduce verbosity for heavy DTML scripting, though, so if you do put it in, *please*, *please* don't rename it to #with-namespace or #with-variable-values or anything like that, okay? :) Thanks.
I like it. May we include it in the Zope distribution?
See the middle of the quoted paragraph above. In other words, yes. :)
Jim Fulton wrote:
It's a new tag that lets you create blocks like:
<!--#in "1,2,3,4"--> <!--#let num=sequence-item index=sequence-index result="num*index"--> <!--#var num--> * <!--#var index--> = <!--#var result--> <!--#/let--> <!--#/in-->
I like it. May we include it in the Zope distribution?
What is the current status for the <!--#try --> tag ? Will it also be included in 2.0.0 ? ------------ Hannu
Hannu Krosing wrote:
Jim Fulton wrote:
It's a new tag that lets you create blocks like:
<!--#in "1,2,3,4"--> <!--#let num=sequence-item index=sequence-index result="num*index"--> <!--#var num--> * <!--#var index--> = <!--#var result--> <!--#/let--> <!--#/in-->
I like it. May we include it in the Zope distribution?
What is the current status for the <!--#try --> tag ?
I think it is in 2.0a1.
Will it also be included in 2.0.0 ?
Yup. Jim -- Jim Fulton mailto:jim@digicool.com Python Powered! Technical Director (888) 344-4332 http://www.python.org Digital Creations http://www.digicool.com http://www.zope.org Under US Code Title 47, Sec.227(b)(1)(C), Sec.227(a)(2)(B) This email address may not be added to any commercial mail list with out my permission. Violation of my privacy with advertising or SPAM will result in a suit for a MINIMUM of $500 damages/incident, $1500 for repeats.
At 05:39 PM 6/10/99 +0000, you wrote:
To take these musings a step further. I can imagine a mechanism by which:
- Certain objects could inform ZServer of object IDs of the objects that constitute thier "source" data,
- Zserver registers with the database to be notified when objects change (by object ID),
This could allow ZServer to do some clever caching even for some relatively dynamic objects.
(Note that figuring out what the "source" objects for an object are could be pretty tricky. While some objects, like Images, Files, and most DTMLDocuments are their own source, other objects, like DTMLMethods might depend on thousands of objects.)
Caching in Zope is an interesting, though complex issue. Clearly some object always return the same or similar content, but many more return the same content most of the time, but not all the time. The rest are totally unpredictable ;-) In any event, I'm happy to see Jim yield a little on the idea of caching. I think caching could considerably improve Zope performance for heavily trafficked, non-dynamic sites. I would like to add that the response generated by some objects depend on the user. So for example we wouldn't want to cache DTML Methods which display different data depending on the roles of the user who accessed the page. OK, maybe this can be abstracted to say that caching could be tied to request environment variables, such as "AUTHENTICATED_USER", and "PATH_INFO". Presumably caching info would be communicated to ZServer via HTTP headers... Another issue is being able to write a cache to the filesystem so that other things besides ZServer will be able to serve up cached items. Hmm. Interesting stuff, but who's got time to implement it ;-) -Amos
Amos Latteier wrote:
(snip)
I would like to add that the response generated by some objects depend on the user. So for example we wouldn't want to cache DTML Methods which display different data depending on the roles of the user who accessed the page.
These are probably pages that don't want to be cached.
OK, maybe this can be abstracted to say that caching could be tied to request environment variables, such as "AUTHENTICATED_USER", and "PATH_INFO".
And cookies....Ugh. I think it would be best to start with things that are known not to depend on the request or on other objects like images files, etc. (Note that this implied public objects.) I imagine that there might be a knob that a content manager or programmer could use to give Zope some hints in this regard.
Presumably caching info would be communicated to ZServer via HTTP headers...
I could imagine object change messages being sent "out of band" from the database to the cache.
Another issue is being able to write a cache to the filesystem so that other things besides ZServer will be able to serve up cached items. Hmm.
I agree that results should be written to the file system, however, I don't see a need for other things to get at the files. Server could use Medusa's standard file serving logic when results are returned from the cache. Jim -- Jim Fulton mailto:jim@digicool.com Technical Director (888) 344-4332 Python Powered! Digital Creations http://www.digicool.com http://www.python.org Under US Code Title 47, Sec.227(b)(1)(C), Sec.227(a)(2)(B) This email address may not be added to any commercial mail list with out my permission. Violation of my privacy with advertising or SPAM will result in a suit for a MINIMUM of $500 damages/incident, $1500 for repeats.
Jim Fulton wrote:
Amos Latteier wrote:
(snip)
I would like to add that the response generated by some objects depend on the user. So for example we wouldn't want to cache DTML Methods which display different data depending on the roles of the user who accessed the page.
These are probably pages that don't want to be cached.
or that want to be cached with an url (key) that includes the role and/or AUTHENTICATED_USER.
OK, maybe this can be abstracted to say that caching could be tied to request environment variables, such as "AUTHENTICATED_USER", and "PATH_INFO".
And cookies....Ugh.
Maybe we can just attach some callback mechanisms to variable lookups which, among other things, also invalidate cache or change/lookup_new cache key ?
I imagine that there might be a knob that a content manager or programmer could use to give Zope some hints in this regard.
Like the cacheing time for SQL Methods ? -------------------- Hannu
participants (5)
-
Amos Latteier -
Hannu Krosing -
Jim Fulton -
Phillip J. Eby -
Ross J. Reedstrom