Question on: How-To: When Cookies won't do it
I browsed trough the How-To: When Cookies won't do it and I have some questions. How much speed do you really gain from using file for data marshaling? Would it be a good idea to use Object Pooling (pre created objects) to speed up a session object solution? The reason I don't like the file based solution is that I won't scale with ZEO but a Object solution would, right? There would also be possible for cool stuff like session failover and long time session (much list version but individual and perhaps with a limited scoop) Another reason is that I like the concept of the ZODB, why think in a non ZODB way? Regards, Johan Carlsson torped johan carlsson birkagatan 9 113 36 stockholm telefon 08-32 31 23 mobil 070-558 25 24 fax 08-32 89 47 johanc@torped.se
At 11:10 18/10/99 , Johan Carlsson wrote:
I browsed trough the How-To: When Cookies won't do it and I have some questions.
How much speed do you really gain from using file for data marshaling?
Would it be a good idea to use Object Pooling (pre created objects) to speed up a session object solution?
The reason I don't like the file based solution is that I won't scale with ZEO but a Object solution would, right? There would also be possible for cool stuff like session failover and long time session (much list version but individual and perhaps with a limited scoop)
Another reason is that I like the concept of the ZODB, why think in a non ZODB way?
One word: Transactions. Every change to a ZODB persistent object causes another transaction to be written to storage. A busy site with lots of sessions will cause the Data.fs file to grow at quite a rate. Pavlos choose this solution to avoid this growth. I suggested trying to use a seperate instance of the ZODB, using MemoryStorage, so that all session objects would be in memory. This could then maybe be extended to support ZEO as well, but I haven't really dived into that yet. -- Martijn Pieters, Web Developer | Antraciet http://www.antraciet.nl | Tel: +31-35-7502100 Fax: +31-35-7502111 | mailto:mj@antraciet.nl http://www.antraciet.nl/~mj | PGP: http://wwwkeys.nl.pgp.net:11371/pks/lookup?op=get&search=0xA8A32149 ------------------------------------------
I browsed trough the How-To: When Cookies won't do it and I have some questions.
How much speed do you really gain from using file for data marshaling?
Would it be a good idea to use Object Pooling (pre created objects) to speed up a session object solution?
The reason I don't like the file based solution is that I won't scale with ZEO but a Object solution would, right? There would also be possible for cool stuff like session failover and long time session (much list version but individual and perhaps with a limited scoop)
Another reason is that I like the concept of the ZODB, why think in a non ZODB way?
One word: Transactions.
Every change to a ZODB persistent object causes another transaction to be written to storage. A busy site with lots of sessions will cause the Data.fs file to grow at quite a rate. Pavlos choose this solution to avoid this growth.
I suggested trying to use a seperate instance of the ZODB, using MemoryStorage, so that all session objects would be in memory. This could then maybe be extended to support ZEO as well, but I haven't really dived into that yet.
Ok. One (or more:) question(s). - When does filestorage or rdbm-storage write to disk? - Why can't the session object rely on the ZODB memory cache. They are (for the most) fast deployed objects. Maybe even use _v_ objects (which would be automatically destroyed). - How does transactions slow down the process, and how are transactions connected to storage (in this situation)? Transaction support for sessions are one of the things you really want. I even been thinking about using private versions for sessions, for instance in a e-shop you would be able to customize your order directly in the OLAP-system but within a version. Not until you commit your transaction the changes gets propagated to the OLAP-system. Regards, //johan
Hi Johan -
I suggested trying to use a seperate instance of the ZODB, using MemoryStorage, so that all session objects would be in memory. This could then maybe be extended to support ZEO as well, but I haven't really dived into that yet.
Martijn's suggestion is probably the best (as usual), but it involves more effort (and thinking!) than the 10 lines script I wrote.
Ok. One (or more:) question(s).
- Why can't the session object rely on the ZODB memory cache. They are (for the most) fast deployed objects. Maybe even use _v_ objects (which would be automatically destroyed).
That is one way but your temporary data will be lost as soon as the object is deactivated (as you mentioned). With tmpStorage I can retrieve the info at any point, even days later.
- How does transactions slow down the process, and how are transactions connected to storage (in this situation)?
I am not sure what is the overhead of transactions but what I am trying to avoid is not transaction control (in principle I could make tmpStorage transaction aware), but as Martijn mentioned, to avoid new object creation.
Transaction support for sessions are one of the things you really want. I even been thinking about using private versions for sessions, for instance in a e-shop you would be able to customize your order directly in the OLAP-system but within a version. Not until you commit your transaction the changes gets propagated to the OLAP-system.
Agreed - Again, tmpStorage is a simple solution that is thread safe and fast. Those were my requirements at the time. A proper more general solution is the one mentioned by Martijn. Pavlos
Again, tmpStorage is a simple solution that is thread safe and fast. Those were my requirements at the time. A proper more general solution is the one mentioned by Martijn.
Hi Pavlos, I'm not criticizing you solution, I think it's a great How-To for a simple and clean solution. I just want to explore what other options there are. I think the lack of session support is one big disadvantage for Zope at the moment. Just so that you know where I am standing :-) Keep up the good work. Cheers, Johan Carlsson
At 13:16 18/10/99 , Johan Carlsson wrote:
Every change to a ZODB persistent object causes another transaction to be written to storage. A busy site with lots of sessions will cause the Data.fs file to grow at quite a rate. Pavlos choose this solution to avoid this growth.
I suggested trying to use a seperate instance of the ZODB, using MemoryStorage, so that all session objects would be in memory. This could then maybe be extended to support ZEO as well, but I haven't really dived into that yet.
Ok. One (or more:) question(s).
- When does filestorage or rdbm-storage write to disk?
I don't really know, and don't have the time right now to figure this out properly. Jim Fulton will be a better person to answer this. Also, you could look through the ZODB UML Model: http://www.zope.org/Documentation/Developer/Models/ZODB
- Why can't the session object rely on the ZODB memory cache. They are (for the most) fast deployed objects. Maybe even use _v_ objects (which would be automatically destroyed).
_v_ properties are not shared between threads. My understanding is that every thread will get its own copy of the object from the ZODB, which propagates changes to those objects to other threads. As _v_ properties are not stored, no other thread will ever see them, defeating their use as session variables. They are only useful for caching. There is at the moment no API for using the ZODB memory cache outside the ZODB code itself, afaik.
- How does transactions slow down the process, and how are transactions connected to storage (in this situation)?
Not so much slow down as cause undesirable side effects, like storing a complete change history of the changed object. See the ZODB UML for more information.
Transaction support for sessions are one of the things you really want.
Oh yes, definitely. With the current implementation of the ZODB however, any session specific data that has a reasonably high change rate will have to be stored outside the ZODB. The ZODB was designed with long-term persistence in mind, not short term. There are plans to allow use of multiple Storage types within one Zope object hierarchy. Also, there are Storage available now, that do not support Undo (discard old versions). When multiple storages are supported, you could use a in-memory storage without undo for session data.
I even been thinking about using private versions for sessions, for instance in a e-shop you would be able to customize your order directly in the OLAP-system but within a version. Not until you commit your transaction the changes gets propagated to the OLAP-system.
I am not sure wether or not this will work. I think that a Version only supports a long-term transaction on ZODB stored data, not on any external data. But I may be mistaken. -- Martijn Pieters, Web Developer | Antraciet http://www.antraciet.nl | Tel: +31-35-7502100 Fax: +31-35-7502111 | mailto:mj@antraciet.nl http://www.antraciet.nl/~mj | PGP: http://wwwkeys.nl.pgp.net:11371/pks/lookup?op=get&search=0xA8A32149 ------------------------------------------
I don't really know, and don't have the time right now to figure this out properly. Jim Fulton will be a better person to answer this. Also, you could look through the ZODB UML Model:
http://www.zope.org/Documentation/Developer/Models/ZODB
- Why can't the session object rely on the ZODB memory cache. They are (for the most) fast deployed objects. Maybe even use _v_ objects (which would be automatically destroyed).
There is at the moment no API for using the ZODB memory cache outside the ZODB code itself, afaik.
- How does transactions slow down the process, and how are transactions connected to storage (in this situation)? Not so much slow down as cause undesirable side effects, like storing a complete change history of the changed object. See the ZODB UML for more information.
Aha, I see! Great point there.
There are plans to allow use of multiple Storage types within one Zope object hierarchy. Also, there are Storage available now, that do not support Undo (discard old versions). When multiple storages are supported, you could use a in-memory storage without undo for session data.
And the in-memory storage would have it's own ZEO-server for distributing and load-balancing session data, I suppose. That explains the need for the in-memory storage option. (Is this what DC have in mind, DC folks?) It would be great to have a session API pretty soon, that could be used to create session aware prototypes that could migrate to ZEO and Multiple Storage/In-Memory Session solutions in the future. The session implementation could be something less complicated for a prototype. Or is that to much work and to much unknown facts about future implementations? //johan
At 14:29 18/10/99 , Johan Carlsson wrote:
There are plans to allow use of multiple Storage types within one Zope object hierarchy. Also, there are Storage available now, that do not support Undo (discard old versions). When multiple storages are supported, you could use a in-memory storage without undo for session data.
And the in-memory storage would have it's own ZEO-server for distributing and load-balancing session data, I suppose. That explains the need for the in-memory storage option.
(Is this what DC have in mind, DC folks?)
The way understand ZEO and ZODB in general works, and the way multiple storages work, the underlying Starge for different objects is otherwise transparent. An object just acts like a Persistent object (so, practically transparant to the developer), wether it is stored in the part of the object hierarchy that is FileStorage based, or MemoryStorage. This would automatically work on ZEO as well, no separate ZEO server required.
It would be great to have a session API pretty soon, that could be used to create session aware prototypes that could migrate to ZEO and Multiple Storage/In-Memory Session solutions in the future. The session implementation could be something less complicated for a prototype. Or is that to much work and to much unknown facts about future implementations?
It would be. I once wrote a proposal for Zope sessions, but back then my knowledge of Zope was too limited. I personally would like to see both cookie and URL based sessions, with a fixed set of variables to be stored (like a property sheet on a ZClass has right now, not like the property sheets on Folders), so that you can give access to the session object to a third party, without fear of denial of service attacks. -- Martijn Pieters, Web Developer | Antraciet http://www.antraciet.nl | Tel: +31-35-7502100 Fax: +31-35-7502111 | mailto:mj@antraciet.nl http://www.antraciet.nl/~mj | PGP: http://wwwkeys.nl.pgp.net:11371/pks/lookup?op=get&search=0xA8A32149 ------------------------------------------
participants (3)
-
Johan Carlsson -
Martijn Pieters -
Pavlos Christoforou