Question about application design
Hi, I'm not sure how to do my current project in a Zope environment: It's something like a trip planner that computes an itinerary. Pretty simple use case actually - the user submits origin and destination, and potential trip plans are returned. BUT: What I want to do is keep the data structure in memory. Right now, I have this written as a standalone Python app that creates all the objects it needs, and then uses an algorithm to find the shortest path. This is the typical application server scenario: I want the Python classes to only do the startup work once, and then just answer requests which run an algorithm over the data. But how to do this with Zope? As far as I can tell, the Python Scripts are stateless. The only idea I've had so far is to make my Python app into a standalone server which answers requests via XML-RPC. Then Zope stands between it and the user. But is there any way to use just Zope, and not maintain my own server too? Thanks! Robb
From: "Robb Shecter" <robb@acm.org>
It's something like a trip planner that computes an itinerary. Pretty simple use case actually - the user submits origin and destination, and potential trip plans are returned.
BUT: What I want to do is keep the data structure in memory. Right now, I have this written as a standalone Python app that creates all the objects it needs, and then uses an algorithm to find the shortest path.
Zope objects stored in a 'temp_folder' are held in RAM for faster access. You could create an object or property field in the temp_folder and then access it as required. HTH Jonathan
Small Business Services said:
Zope objects stored in a 'temp_folder' are held in RAM for faster access. You could create an object or property field in the temp_folder and then access it as required.
Interesting. So I could modify my Python code to be run as an external method. It would then somehow get a reference to the temp_folder, and it'd create a reference to the data structure from the temp_folder? And then, on future invocations, it'd check to see if the data structure is in the folder or not. Is this a scenario that's used for building web services? Thanks, Robb
Robb Shecter wrote:
Small Business Services said:
Zope objects stored in a 'temp_folder' are held in RAM for faster access.
This isn't correct. They're there because they don't get stored to disk, the name's a bit of a give away ;-)
Interesting. So I could modify my Python code to be run as an external method. It would then somehow get a reference to the temp_folder, and it'd create a reference to the data structure from the temp_folder?
Why the need for an external method? cheers, Chris -- Simplistix - Content Management, Zope & Python Consulting - http://www.simplistix.co.uk
You can keep data in memory by applying a RAM cache to the object that contains or generates it. This is one way to keep resource lists and dictionaries 'on air'. Ausum ----- Original Message ----- From: "Robb Shecter" <robb@acm.org> To: <zope@zope.org> Sent: Tuesday, April 20, 2004 8:12 AM Subject: [Zope] Question about application design
Hi,
I'm not sure how to do my current project in a Zope environment:
It's something like a trip planner that computes an itinerary. Pretty simple use case actually - the user submits origin and destination, and potential trip plans are returned.
BUT: What I want to do is keep the data structure in memory. Right now, I have this written as a standalone Python app that creates all the objects it needs, and then uses an algorithm to find the shortest path.
This is the typical application server scenario: I want the Python classes to only do the startup work once, and then just answer requests which run an algorithm over the data. But how to do this with Zope?
As far as I can tell, the Python Scripts are stateless.
The only idea I've had so far is to make my Python app into a standalone server which answers requests via XML-RPC. Then Zope stands between it and the user.
But is there any way to use just Zope, and not maintain my own server too?
Thanks! Robb
_______________________________________________ Zope maillist - Zope@zope.org http://mail.zope.org/mailman/listinfo/zope ** No cross posts or HTML encoding! ** (Related lists - http://mail.zope.org/mailman/listinfo/zope-announce http://mail.zope.org/mailman/listinfo/zope-dev )
Ausum Studio said:
You can keep data in memory by applying a RAM cache to the object that contains or generates it. This is one way to keep resource lists and dictionaries 'on air'.
Is there a way, from a Python script to just add python objects to a container? For example, a list of strings to a folder? I looked in the API for Folder, (and then Content Manager), and saw that one can only add content by adding products. So, does that mean that arbitrary data can't be stored and retrieved from containers (like temp_folder)? Robb
On Tuesday 20 April 2004 10:35 am, Robb Shecter wrote:
Ausum Studio said: Is there a way, from a Python script to just add python objects to a container?
For example, a list of strings to a folder?
I looked in the API for Folder, (and then Content Manager), and saw that one can only add content by adding products. So, does that mean that arbitrary data can't be stored and retrieved from containers (like temp_folder)?
You can store a list of strings as a property of the Folder (look at the PropertyManager API). It might not be terribly efficient, and will likely not work for more complex structures. The better option may be a simple python product implementing some sort of persistent object which stores the data structures and access methods you need (OOBTrees and OOSets might be more convenient than standard lists and mappings because they are naturally persistent), that product could also encapsulate your existing python logic in any number of ways. Alec Mitchell
Alec Mitchell said:
...The better option may be a simple python product implementing some sort of persistent object which stores the data structures and access methods you need...
Yes, from my diving into the docs, it looks like that's the route I might have to go: I'll have about 150,000 objects - all in one or two "Sparse Matrix" containers I use. And then I do a Dijkstra-like shortest path finding algorithm over the data. So apparently, I can't just take this data structure and stick it in a temp_folder. It sounds to me like Zope isn't so much an application server as a CMS server. Too bad! For example, with Tomcat, a Java servlet engine, my scenario would be pretty easy to implement.
On Tuesday 20 April 2004 11:20 am, Robb Shecter wrote:
Yes, from my diving into the docs, it looks like that's the route I might have to go:
I'll have about 150,000 objects - all in one or two "Sparse Matrix" containers I use. And then I do a Dijkstra-like shortest path finding algorithm over the data.
So apparently, I can't just take this data structure and stick it in a temp_folder.
It sounds to me like Zope isn't so much an application server as a CMS server. Too bad! For example, with Tomcat, a Java servlet engine, my scenario would be pretty easy to implement.
I think you'll be surprised how easy it is to create a persistent container for that data (just inheriting Item.SimpleItem should get you most of the way there, then you just create instances of your container data structure as an instance variable in the constructor). Zope is definitely more of an application server than a CMS (though with the right products it makes a hell of a CMS), it is just a very different way of thinking about the problem than a java servlet engine (or really anything else). Though the ZMI looks like a filesystem, it is a very powerful and general persistent object storage with a very useful acquisition model. Good Luck, Alec Mitchell
Hi Robb, Perhaps I can paraphrase Alec Mitchell's advice, if it will help to hear it in a slightly different way: What it sounds like you want to do is quite simple in Zope. Create a new Product, subclassed from a persistent class (such as SimpleItem), which will contain all of the code and data for your application. In this Product's constructor, create and initialize an instance of your "sparse matrix" data. Because your product is persistent, this data will be stored in the Zope database, and available to you as a regular instance variable when your other methods are called, just the same as if it had always been in RAM. If your sparse matrix is a custom class (i.e., it's not just a basic Python list or dictionary), it may also need to inherit from a persistent class (such as Persistent), to make sure that it's stored in the database when the rest of your product data is stored. Cheers, Finlay Cannon On 20 Apr 2004, at 11:36 AM, Alec Mitchell wrote:
On Tuesday 20 April 2004 11:20 am, Robb Shecter wrote:
Yes, from my diving into the docs, it looks like that's the route I might have to go:
I'll have about 150,000 objects - all in one or two "Sparse Matrix" containers I use. And then I do a Dijkstra-like shortest path finding algorithm over the data.
So apparently, I can't just take this data structure and stick it in a temp_folder.
It sounds to me like Zope isn't so much an application server as a CMS server. Too bad! For example, with Tomcat, a Java servlet engine, my scenario would be pretty easy to implement.
I think you'll be surprised how easy it is to create a persistent container for that data (just inheriting Item.SimpleItem should get you most of the way there, then you just create instances of your container data structure as an instance variable in the constructor). Zope is definitely more of an application server than a CMS (though with the right products it makes a hell of a CMS), it is just a very different way of thinking about the problem than a java servlet engine (or really anything else). Though the ZMI looks like a filesystem, it is a very powerful and general persistent object storage with a very useful acquisition model.
Good Luck, Alec Mitchell
...The better option may be a simple python product implementing some sort of persistent object
I just discovered the TransientObject. How about I add a Transient Object Container to the temp_folder, which will store my application data? That looks like it'll work... My data set is read-only, by the way. I also just discovered how easy it is to get access to the SESSION transient object from a python script, and how you can easily use that. Very nice.
On Tuesday 20 April 2004 11:39 am, Robb Shecter wrote:
I just discovered the TransientObject. How about I add a Transient Object Container to the temp_folder, which will store my application data? That looks like it'll work...
My data set is read-only, by the way.
I also just discovered how easy it is to get access to the SESSION transient object from a python script, and how you can easily use that. Very nice. I've never used any transient objects myself. If your data is read-only and it is associated with a single user session, then it seems almost ideal to store it in the SESSION object. Even if it were mutable you could probably store it there as long as you set _p_changed explicitly when necessary (not possible from a python script though). Sounds like you are getting pretty deep into Zope rather quickly, enjoy.
Alec Mitchell
On Tue, Apr 20, 2004 at 02:28:04PM -0700, Alec Mitchell wrote:
On Tuesday 20 April 2004 11:39 am, Robb Shecter wrote:
I just discovered the TransientObject. How about I add a Transient Object Container to the temp_folder, which will store my application data? That looks like it'll work...
My data set is read-only, by the way.
I also just discovered how easy it is to get access to the SESSION transient object from a python script, and how you can easily use that. Very nice. I've never used any transient objects myself. If your data is read-only and it is associated with a single user session, then it seems almost ideal to store it in the SESSION object. Even if it were mutable you could probably store it there as long as you set _p_changed explicitly when necessary (not possible from a python script though). Sounds like you are getting pretty deep into Zope rather quickly, enjoy.
Be careful with very heavy use of sessions... check recent threads from Chris McDonough on the subject, either here or on the zope-dev list. There are apparently some problems that crop up with sessions under heavy load. -- Paul Winkler http://www.slinkp.com
Let's say you want to have access to the whole list of cities of the world and its corresponding coordinates. If I would want that information in RAM, I just would put in inside a Python script, inside a RAM cache manager, as a dictionary: # script name: 'cities_of_the_world' cities = {'city1':'coordinates1','city2':'coordinates2','city3':'coordinates3', ...} return cities If you need to call the coordinates of city1, whithin another script in the same container: cities = context.cities_of_the_world() city1_coordinates = cities['city1'] Ausum ----- Original Message ----- From: "Robb Shecter" <robb@acm.org> To: "Robb Shecter" <robb@acm.org> Cc: <zope@zope.org> Sent: Tuesday, April 20, 2004 1:39 PM Subject: Re: [Zope] Question about application design
...The better option may be a simple python product implementing some sort of persistent object
I just discovered the TransientObject. How about I add a Transient Object Container to the temp_folder, which will store my application data? That looks like it'll work...
My data set is read-only, by the way.
I also just discovered how easy it is to get access to the SESSION transient object from a python script, and how you can easily use that. Very nice.
_______________________________________________ Zope maillist - Zope@zope.org http://mail.zope.org/mailman/listinfo/zope ** No cross posts or HTML encoding! ** (Related lists - http://mail.zope.org/mailman/listinfo/zope-announce http://mail.zope.org/mailman/listinfo/zope-dev )
Robb Shecter wrote:
I also just discovered how easy it is to get access to the SESSION transient object from a python script, and how you can easily use that. Very nice.
Bear in mind that each user has their own SESSION object... Chris -- Simplistix - Content Management, Zope & Python Consulting - http://www.simplistix.co.uk
Robb Shecter wrote:
It sounds to me like Zope isn't so much an application server as a CMS server. Too bad! For example, with Tomcat, a Java servlet engine, my scenario would be pretty easy to implement.
mod_python may better suite your needs... Chris -- Simplistix - Content Management, Zope & Python Consulting - http://www.simplistix.co.uk
Alec Mitchell wrote:
You can store a list of strings as a property of the Folder (look at the PropertyManager API). It might not be terribly efficient,
It's pretty efficient ;-)
and will likely not work for more complex structures.
It will, but it shouldn't ;-) You can store dictionaries as properties, eve nthough there is no PropertyManager type for them. The only downside is that the Properties tab doesn't display the value of those properties... Chris -- Simplistix - Content Management, Zope & Python Consulting - http://www.simplistix.co.uk
Robb Shecter wrote:
Ausum Studio said:
You can keep data in memory by applying a RAM cache to the object that contains or generates it. This is one way to keep resource lists and dictionaries 'on air'.
Is there a way, from a Python script to just add python objects to a container?
For example, a list of strings to a folder?
folder.manage_addProperty('my_stuff',['one','two','three'],'lines')
I looked in the API for Folder,
Look at the PropertyManager API... Chris -- Simplistix - Content Management, Zope & Python Consulting - http://www.simplistix.co.uk
participants (7)
-
Alec Mitchell -
Ausum Studio -
Chris Withers -
Finlay Cannon -
Paul Winkler -
Robb Shecter -
Small Business Services