Storing part of an object on the file system
Ok whack idea #34... Well sort of, the idea was sparked by ExtFile (yes its all his fault honest). We want to have files on the file system for many reasons which I won't go into now, just take it for granted. But we also want some of the objects information to be in the standard data.fs, so we can catalog it use acquisition provide a simple interface to it and so on. So far you are saying thats ExtFile and you are right. But we want to extend it to any object anywhere, instead of storing the data attribute in the object and pickling it in the ZODB I want to be able to store the data attribute on the file system. This gives us loads of advantages we also thought this would be extremely useful to other people. We've bounced around ideas on how to do this and here the only two so far: - we could overload the data attribute with a class that on Pickling into the ZODB instead writes it on to the filesystem... - we could in the ZODB put a hack to say if pickling something with so and so attribute do this instead... I feel like this something I should just not be doing, but it would be great if I could get it work... -- Andy McKay.
I would like to throw in something I've been thinking about for the last week or so. I would like to use Tux as the front-end of my website. For info on Tux. ftp://ftp.redhat.com/pub/redhat/tux/tux-2.0/ http://www.redhat.com/products/software/ecommerce/tux/
From the website: """TUX is a kernel-based, threaded, extremely high performance HTTP server. It is able to efficiently and safely serve both static and dynamic data. TUX moves the HTTP protocol stack to the kernel, and can handle requests for data with both kernel-space and user-space modules."""
It has been reported to do over 10,000 requests per second on a static html file of 1666 bytes. It did over 1,000 rps doing a cgi program, opening and closing the cgi app each request. I read this from Linux Weekly News which got it off of the Linux Kernel Mailing List.
From what I understand Tux 1.0 holds the SpecWeb record. Tux 2.0 is faster. :)
I think it would be interesting to see Zope fronted by Tux. If there were a Tux cache manager for Zope that would be awesome. I've been trying to figure out how to maximize what Tux can serve and what Zope can manage. An optimal mix between the two could create a very manageable and fast website. Tux primarily caches what is available on the file system. It can serve multiple files (objects) sequentially as instructed by the "user module" as part of a single client request, thus assembling the request to send to the client. I'm butchering the explanation, but any ways, it's fast. Unfortunately, I am still working on understanding Tux and how to best use it. I don't program C and so reading the sample code hasn't sunk in yet. I don't know how to integrate it with Zope. It would be great it one of the brilliant people on this list could figure this out. :) :) :) Just thinking out loud. Jimmie Houchin Andy McKay wrote:
Ok whack idea #34... Well sort of, the idea was sparked by ExtFile (yes its all his fault honest). We want to have files on the file system for many reasons which I won't go into now, just take it for granted. But we also want some of the objects information to be in the standard data.fs, so we can catalog it use acquisition provide a simple interface to it and so on. So far you are saying thats ExtFile and you are right.
But we want to extend it to any object anywhere, instead of storing the data attribute in the object and pickling it in the ZODB I want to be able to store the data attribute on the file system. This gives us loads of advantages we also thought this would be extremely useful to other people. We've bounced around ideas on how to do this and here the only two so far:
- we could overload the data attribute with a class that on Pickling into the ZODB instead writes it on to the filesystem... - we could in the ZODB put a hack to say if pickling something with so and so attribute do this instead...
I feel like this something I should just not be doing, but it would be great if I could get it work...
-- Andy McKay.
Jimmie Houchin wrote:
I think it would be interesting to see Zope fronted by Tux. If there were a Tux cache manager for Zope that would be awesome.
Actually, if Tux works the way I think, an accelerated HTTP cache manager should be just what you're looking for. If you can just get Tux to forward requests to Zope, you should be home free! I've been considering upgrading to 2.4.1, maybe this is a good reason to do it. If I read the README correctly, it may also be possible to make use of Tux's server-side include directives to cache pieces of pages. If this worked, you'd almost be able to run cnn.com on a 486. ;-) Shane
Hello Shane, This would be great. I just compiled my first kernel to upgrade to 2.4.1. Yeah!!! Zope/ZServer fronted by Tux could be a tremendous tool. Allowing Tux to do as much of what it does best and allowing Zope to do as much as it does best would be unbeatable. If Zope could feed Tux cacheable objects without them being required to be on the filesystem, that would be great. One could develop their site pretty much the same way with the exception of using the Tux HTTP cache manager and selecting the objects to be cached. Tux will forward requests to Zope via either cgi or a user-module. Using cgi Tux doesn't cache the objects. It wouldn't be as big a gain. A user-module can take advantage of Tux's user-module API for handling requests and caching. But as I said, I don't read or program C so I don't know how to do this. I have started looking at some C tutorial material so I can at least understand some of the sample code. There is a sample program written for SpecWeb which gives a better example of it's user api. http://www.spec.org/osg/web99/results/api-src/Dell-20001128.tar.gz The CAD_u.c module. Tux is multiprocessor capable, so theoretically you could put Zope/ZEO on a multiprocessor machine with Tux and have an interesting time. :) For those who didn't read the initial post about Tux in the other thread here's info. ftp://ftp.redhat.com/pub/redhat/tux/tux-2.0/ http://www.redhat.com/products/software/ecommerce/tux/ Jimmie Houchin Shane Hathaway wrote:
Jimmie Houchin wrote:
I think it would be interesting to see Zope fronted by Tux. If there were a Tux cache manager for Zope that would be awesome.
Actually, if Tux works the way I think, an accelerated HTTP cache manager should be just what you're looking for. If you can just get Tux to forward requests to Zope, you should be home free!
I've been considering upgrading to 2.4.1, maybe this is a good reason to do it.
If I read the README correctly, it may also be possible to make use of Tux's server-side include directives to cache pieces of pages. If this worked, you'd almost be able to run cnn.com on a 486. ;-)
Shane
I had a similar problem. I'm using ExtImage, but I need some extensions for it. Because I using a RDMBS (currently MySQL) I wanted to have all the meta data in my RDMBS. For all my objects there is normally a 'InsertMethod'. This can be a ZSQL method (if just MySQL is involved) or it can be a DTML method. For inserting an Picture object I have two steps in my DMTL InsertMethod: <dtml-let ret="manage_addProduct['ExtFile'].manage_addExtImage(title=_['PICTURE.TITLE'], descr=_['PICTURE.DESCR'], file=_['PICTURE.FILE'], content_type='', create_prev=create_prev, maxx=maxx, maxy=maxy, ratio=ratio, permission_check=permission_check, REQUEST=REQUEST)"> <dtml-if expr="_.string.find(ret,'success') != -1"> <dtml-call "REQUEST.set('PICTURE.PIC_NAME_ID',_.string.strip(getImageName(message=ret)))"> <dtml-call "REQUEST.set('PICTURE.TITLE',_['PICTURE.TITLE'])"> <dtml-call PictureInsertMethod> // this is a ZSQL Method <dtml-call "RESPONSE.redirect('SearchResult?GALERIE.GALERIE_ID='+ _.str(_['GALERIE.GALERIE_ID']))"> <dtml-else> <dtml-var ret> </dtml-if> </dtml-let> Okay, it's not transaction safe for now, but that's is acceptable in my environment. Of course there can be some improvements. Hint: I extended ExtImage for dynamic resizing. A preview is computed on the fly. Maybe there is a need for this. I will send the extension to Gregor. Best regards Arno Gross, arno.gross@consotec.de On Thu, 08 Feb 2001, you wrote:
Ok whack idea #34... Well sort of, the idea was sparked by ExtFile (yes its all his fault honest). We want to have files on the file system for many reasons which I won't go into now, just take it for granted. But we also want some of the objects information to be in the standard data.fs, so we can catalog it use acquisition provide a simple interface to it and so on. So far you are saying thats ExtFile and you are right.
But we want to extend it to any object anywhere, instead of storing the data attribute in the object and pickling it in the ZODB I want to be able to store the data attribute on the file system. This gives us loads of advantages we also thought this would be extremely useful to other people. We've bounced around ideas on how to do this and here the only two so far:
- we could overload the data attribute with a class that on Pickling into the ZODB instead writes it on to the filesystem... - we could in the ZODB put a hack to say if pickling something with so and so attribute do this instead...
I feel like this something I should just not be doing, but it would be great if I could get it work...
-- Andy McKay.
_______________________________________________ Zope-Dev maillist - Zope-Dev@zope.org http://lists.zope.org/mailman/listinfo/zope-dev ** No cross posts or HTML encoding! ** (Related lists - http://lists.zope.org/mailman/listinfo/zope-announce http://lists.zope.org/mailman/listinfo/zope )
Andy McKay wrote:
Ok whack idea #34... Well sort of, the idea was sparked by ExtFile (yes its all his fault honest). We want to have files on the file system for many reasons which I won't go into now, just take it for granted. But we also want some of the objects information to be in the standard data.fs, so we can catalog it use acquisition provide a simple interface to it and so on. So far you are saying thats ExtFile and you are right.
But we want to extend it to any object anywhere, instead of storing the data attribute in the object and pickling it in the ZODB I want to be able to store the data attribute on the file system. This gives us loads of advantages we also thought this would be extremely useful to other people. We've bounced around ideas on how to do this and here the only two so far:
Use ZPatterns. This is exactly what ZPatterns is designed for; choosing different places to store and retrieve different attributes.
- we could overload the data attribute with a class that on Pickling into the ZODB instead writes it on to the filesystem...
You can easily do this with ZPatterns.
- we could in the ZODB put a hack to say if pickling something with so and so attribute do this instead...
You can easily do this with ZPatterns.
I feel like this something I should just not be doing, but it would be great if I could get it work...
Take a look at the SkinScript reference. http://www.zope.org//Members/pje/Wikis/ZPatterns/SkinScriptSyntax -- Steve Alexander Software Engineer Cat-Box limited http://www.cat-box.net
Hmm sounds like ZPatterns might help ;-)... will take a look thanks. -- Andy McKay. ----- Original Message ----- From: "Steve Alexander" <steve@cat-box.net> To: "Andy McKay" <andym@activestate.com> Cc: <zope-dev@zope.org> Sent: Friday, February 09, 2001 1:27 AM Subject: Re: [Zope-dev] Storing part of an object on the file system
Andy McKay wrote:
Ok whack idea #34... Well sort of, the idea was sparked by ExtFile (yes its all his fault honest). We want to have files on the file system for many reasons which I won't go into now, just take it for granted. But we also want some of the objects information to be in the standard data.fs, so we can catalog it use acquisition provide a simple interface to it and so on. So far you are saying thats ExtFile and you are right.
But we want to extend it to any object anywhere, instead of storing the data attribute in the object and pickling it in the ZODB I want to be able to store the data attribute on the file system. This gives us loads of advantages we also thought this would be extremely useful to other people. We've bounced around ideas on how to do this and here the only two so far:
Use ZPatterns. This is exactly what ZPatterns is designed for; choosing different places to store and retrieve different attributes.
- we could overload the data attribute with a class that on Pickling into the ZODB instead writes it on to the filesystem...
You can easily do this with ZPatterns.
- we could in the ZODB put a hack to say if pickling something with so and so attribute do this instead...
You can easily do this with ZPatterns.
I feel like this something I should just not be doing, but it would be great if I could get it work...
Take a look at the SkinScript reference.
http://www.zope.org//Members/pje/Wikis/ZPatterns/SkinScriptSyntax
-- Steve Alexander Software Engineer Cat-Box limited http://www.cat-box.net
Hi Andy, I think that ZPatterns may be helpful for something like this. You could have certain attributes loaded from an exteral method that looks on the filesystem for stuff.. and others in ZODB. It could be configurable via SkinScript. At least this might be a quick way to get something up for testing.... before mucking around in the pickling code.... -steve
"AM" == Andy McKay <andym@ActiveState.com> writes:
AM> Ok whack idea #34... Well sort of, the idea was sparked by AM> ExtFile (yes its all his fault honest). We want to have files AM> on the file system for many reasons which I won't go into now, AM> just take it for granted. But we also want some of the objects AM> information to be in the standard data.fs, so we can catalog AM> it use acquisition provide a simple interface to it and so on. AM> So far you are saying thats ExtFile and you are right. AM> But we want to extend it to any object anywhere, instead of AM> storing the data attribute in the object and pickling it in AM> the ZODB I want to be able to store the data attribute on the AM> file system. This gives us loads of advantages we also thought AM> this would be extremely useful to other people. We've bounced AM> around ideas on how to do this and here the only two so far: AM> - we could overload the data attribute with a class that on AM> Pickling into the ZODB instead writes it on to the AM> filesystem... - we could in the ZODB put a hack to say if AM> pickling something with so and so attribute do this instead... AM> I feel like this something I should just not be doing, but it AM> would be great if I could get it work... AM> -- Andy McKay. AM> _______________________________________________ Zope-Dev AM> maillist - Zope-Dev@zope.org AM> http://lists.zope.org/mailman/listinfo/zope-dev ** No cross AM> posts or HTML encoding! ** (Related lists - AM> http://lists.zope.org/mailman/listinfo/zope-announce AM> http://lists.zope.org/mailman/listinfo/zope )
Hi, Can use Plugins as Dataskins? If so, do I need to put them inside a Customizer? And how does customizes play with Specialists, or should they even do that in the first place? Regards, Johan Carlsson
Johan Carlsson wrote:
Hi, Can use Plugins as Dataskins?
The PlugIns product is separate from the ZPatterns/DataSkins product. ZPatterns requires that PlugIns be installed, though. There's some experimental support for having Dataskins as plug-ins. However, I can't think of why I'd need to do that. PlugIns and DataSkins are there for different purposes. They solve different problems.
If so, do I need to put them inside a Customizer?
I can't really answer that.
And how does customizes play with Specialists,
Generally, you create "glue" using SkinScript in both the Customizers and Specialists.
or should they even do that in the first place?
That's up to you. The best way to develop with ZPatterns is first to think clearly about the problem domain you want to work in. The classes from ZPatterns, such as Specialists, DataSkins and Folders with Customizer Support, can help you design and implement your application, leaving many of the important decisions about methods of collaboration, and about storage of attributes, until later. -- Steve Alexander Software Engineer Cat-Box limited http://www.cat-box.net
Johan Carlsson wrote:
Hi, Can use Plugins as Dataskins?
The PlugIns product is separate from the ZPatterns/DataSkins product. ZPatterns requires that PlugIns be installed, though.
There's some experimental support for having Dataskins as plug-ins. However, I can't think of why I'd need to do that. PlugIns and DataSkins are there for different purposes. They solve different problems.
Yes, what much I know. I find Plugins really useful for dynamically assigning content and behavior to objects at run-time. What I want to do is to make plugins created inside regular ZODB objects, delegate there storage to ZPatterns. As I see it might be possible to have both ZODB-based plugins (subclassed from PlugIn) or virtual plugin objects (subclassed from PlugInBase) getting attributes and/or properties from a Customizer. So is it worth a shot? Johan Carlsson
Actually just follow up I realised I never told you how I solved this problem. I compacted all data types into File, Image or DTML Document. Since I had ExtFile and ExtImage I chopped those into one class called it ExtThing, and the wrote classes off it. ExtFile and ExtImage I cut and pasted (not complete) for the most part. Then I wrote a class that called ExtHTML that is a DTML Document that overrides the munge() and read() read functions. Instead I passed those onto ExtThing. And there you go a DTML Document that has the main body stored on the local file system. Thanks McGregor. -- Andy McKay. ----- Original Message ----- From: "Andy McKay" <andym@ActiveState.com> To: <zope-dev@zope.org> Sent: Thursday, February 08, 2001 4:15 PM Subject: [Zope-dev] Storing part of an object on the file system
Ok whack idea #34... Well sort of, the idea was sparked by ExtFile (yes its all his fault honest). We want to have files on the file system for many reasons which I won't go into now, just take it for granted. But we also want some of the objects information to be in the standard data.fs, so we can catalog it use acquisition provide a simple interface to it and so on. So far you are saying thats ExtFile and you are right.
But we want to extend it to any object anywhere, instead of storing the data attribute in the object and pickling it in the ZODB I want to be able to store the data attribute on the file system. This gives us loads of advantages we also thought this would be extremely useful to other people. We've bounced around ideas on how to do this and here the only two so far:
- we could overload the data attribute with a class that on Pickling into the ZODB instead writes it on to the filesystem... - we could in the ZODB put a hack to say if pickling something with so and so attribute do this instead...
I feel like this something I should just not be doing, but it would be great if I could get it work...
-- Andy McKay.
_______________________________________________ Zope-Dev maillist - Zope-Dev@zope.org http://lists.zope.org/mailman/listinfo/zope-dev ** No cross posts or HTML encoding! ** (Related lists - http://lists.zope.org/mailman/listinfo/zope-announce http://lists.zope.org/mailman/listinfo/zope )
Andy, I'm just checking an assumption that I'm making about the Ext* Products: They do no work out of the box with ZEO, right? That is, they would require some sort of shared network drive for all ZEO clients to functions correctly. Thanks, John Andy McKay wrote:
Actually just follow up I realised I never told you how I solved this problem. I compacted all data types into File, Image or DTML Document. Since I had ExtFile and ExtImage I chopped those into one class called it ExtThing, and the wrote classes off it. ExtFile and ExtImage I cut and pasted (not complete) for the most part.
Then I wrote a class that called ExtHTML that is a DTML Document that overrides the munge() and read() read functions. Instead I passed those onto ExtThing. And there you go a DTML Document that has the main body stored on the local file system.
Thanks McGregor. -- Andy McKay.
----- Original Message ----- From: "Andy McKay" <andym@ActiveState.com> To: <zope-dev@zope.org> Sent: Thursday, February 08, 2001 4:15 PM Subject: [Zope-dev] Storing part of an object on the file system
Ok whack idea #34... Well sort of, the idea was sparked by ExtFile (yes
its
all his fault honest). We want to have files on the file system for many reasons which I won't go into now, just take it for granted. But we also want some of the objects information to be in the standard data.fs, so we can catalog it use acquisition provide a simple interface to it and so on. So far you are saying thats ExtFile and you are right.
But we want to extend it to any object anywhere, instead of storing the
data
attribute in the object and pickling it in the ZODB I want to be able to store the data attribute on the file system. This gives us loads of advantages we also thought this would be extremely useful to other people. We've bounced around ideas on how to do this and here the only two so far:
- we could overload the data attribute with a class that on Pickling
into
the ZODB instead writes it on to the filesystem... - we could in the ZODB put a hack to say if pickling something with so and so attribute do this instead...
I feel like this something I should just not be doing, but it would be
great
if I could get it work...
-- Andy McKay.
_______________________________________________ Zope-Dev maillist - Zope-Dev@zope.org http://lists.zope.org/mailman/listinfo/zope-dev ** No cross posts or HTML encoding! ** (Related lists - http://lists.zope.org/mailman/listinfo/zope-announce http://lists.zope.org/mailman/listinfo/zope )
_______________________________________________ Zope-Dev maillist - Zope-Dev@zope.org http://lists.zope.org/mailman/listinfo/zope-dev ** No cross posts or HTML encoding! ** (Related lists - http://lists.zope.org/mailman/listinfo/zope-announce http://lists.zope.org/mailman/listinfo/zope )
-- . . . . . . . . . . . . . . . . . . . . . . . . John D. Heintz | Senior Engineer 1016 La Posada Dr. | Suite 240 | Austin TX 78752 T 512.633.1198 | jheintz@isogen.com w w w . d a t a c h a n n e l . c o m
participants (8)
-
Andy McKay -
Arno Gross -
Jimmie Houchin -
Johan Carlsson -
John D. Heintz -
Shane Hathaway -
Steve Alexander -
Steve Spicklemire