Proposed solution for ZODB woes
Right now we have a provision for non-persistent variables (with _v_, which I assume is for ``volatile''). The obvious way to fix the incontrolable growth of ZODBs is creating a new prefix for ``non-undoable'' variables, perhaps _t_. Implementation-wise, these variables would be stored in a different area. For filesystem storage, the best implementation is probably using a second file for all _t_ stuff. (Actually, they can be kept in RAM and only synced to this file periodically). The other part is giving Zope objects a way to have _t_ and perhaps even _v_ properties. []s, |alo +---- -- I am Lalo of deB-org. You will be freed. Resistance is futile. http://www.webcom.com/lalo mailto:lalo@webcom.com pgp key in the web page Debian GNU/Linux --- http://www.debian.org Brazil of Darkness -- http://zope.gf.com.br/BroDar
Though I admit that I'm not familiar with the inner workings of this part of Zope, I have a feeling that what you're suggesting is a lot of work. There is an existing alternative right now in the form of SQL databases. Just store any of your frequently changing things (hit counters and the like) in there. Yes, I know that's not very Zopish though. Two more possibilities: 1) Longer term, Jim has talked about being able to use multiple storages simultaneously. This would allow you to set up one storage for your frequently changing stuff and one for your less frequently changing things. 2) Or, implement a new storage (maybe an SQL storage). There's already Berkeley Storage, which doesn't have the growth issue of FileSystem, but also doesn't support versions and undo. If you make a storage that attaches to Postgres, you can probably get everything. Gotta love a system as extensible as Zope :) Kevin ----- Original Message ----- From: "Lalo Martins" <lalo@hackandroll.org> To: "Zope-Dev" <zope-dev@zope.org> Sent: Friday, February 18, 2000 8:21 PM Subject: [Zope-dev] Proposed solution for ZODB woes
Right now we have a provision for non-persistent variables (with _v_, which I assume is for ``volatile''). The obvious way to fix the incontrolable growth of ZODBs is creating a new prefix for ``non-undoable'' variables, perhaps _t_.
Implementation-wise, these variables would be stored in a different area. For filesystem storage, the best implementation is probably using a second file for all _t_ stuff. (Actually, they can be kept in RAM and only synced to this file periodically).
The other part is giving Zope objects a way to have _t_ and perhaps even _v_ properties.
On Fri, Feb 18, 2000 at 08:37:38PM -0500, Kevin Dangoor wrote:
Though I admit that I'm not familiar with the inner workings of this part of Zope, I have a feeling that what you're suggesting is a lot of work. There is an existing alternative right now in the form of SQL databases. Just store any of your frequently changing things (hit counters and the like) in there.
I realize that. This is how it is done now. The workaround already exists, I'm suggesting a possible fix.
Yes, I know that's not very Zopish though. Two more possibilities:
1) Longer term, Jim has talked about being able to use multiple storages simultaneously. This would allow you to set up one storage for your frequently changing stuff and one for your less frequently changing things.
I'm talking about fixing ZODB here, not Zope. This ``infinite growth'' is definitely a problem, so it must, sooner or later, be fixed. As for storing variant values in Zope, well, I hate SQL with all my forces, so I'll use anything else that works ;-) []s, |alo +---- -- I am Lalo of deB-org. You will be freed. Resistance is futile. http://www.webcom.com/lalo mailto:lalo@webcom.com pgp key in the web page Debian GNU/Linux --- http://www.debian.org Brazil of Darkness -- http://zope.gf.com.br/BroDar
On Fri, Feb 18, 2000 at 08:37:38PM -0500, Kevin Dangoor wrote:
Though I admit that I'm not familiar with the inner workings of this part of Zope, I have a feeling that what you're suggesting is a lot of work. There is an existing alternative right now in the form of SQL databases. Just store any of your frequently changing things (hit counters and the like) in there.
the hassle of maintaining and administrating an SQL database in addition to Zope doesn't seem to me to be worth it. in a small environment that is fine, but when it comes to hosting it would be much nicer if everything was centralized (and in Zope :).
1) Longer term, Jim has talked about being able to use multiple storages simultaneously. This would allow you to set up one storage for your frequently changing stuff and one for your less frequently changing things.
yep. multiple storages are a solution to a number of problems at the moment. what is the current status of multiple storages?
2) Or, implement a new storage (maybe an SQL storage). There's already Berkeley Storage, which doesn't have the growth issue of FileSystem, but also doesn't support versions and undo. If you make a storage that attaches to Postgres, you can probably get everything.
Gotta love a system as extensible as Zope :)
-d -- Dyon Balding, email: dyon@adroitnet.com.au work: +613 98888522, mobile: +61 0414992604 Adroit Internet Solutions
Dyon Balding wrote:
1) Longer term, Jim has talked about being able to use multiple storages simultaneously. This would allow you to set up one storage for your frequently changing stuff and one for your less frequently changing things.
yep. multiple storages are a solution to a number of problems at the moment. what is the current status of multiple storages?
I'm curious about it as well... obviously, something like it exists in the ZEO, but that's a little heavy for my needs Ethan Fremen imeme
ethan mindlace fremen wrote:
Dyon Balding wrote:
1) Longer term, Jim has talked about being able to use multiple storages simultaneously. This would allow you to set up one storage for your frequently changing stuff and one for your less frequently changing things.
yep. multiple storages are a solution to a number of problems at the moment. what is the current status of multiple storages?
I'm curious about it as well... obviously, something like it exists in the ZEO, but that's a little heavy for my needs
Actually, ZEO does not use multiple storages, it uses a single, client/server storage with a single storage server serving multiple storage clients. Of course, when (and I do mean when) multiple storages are supported then the storage server can also use multiple storages (including multiple client storages, I suppose, whew!). The status of multiple storages is that we're thinking really hard about it. There are a number of things to consider, many of them a ways over my head. Here is recent message from Jim that describes this: Jim Fulton wrote:
Michel Pelletier wrote:
-----Original Message----- From: Anthony Baxter [mailto:anthony@interlink.com.au] Sent: Thursday, January 06, 2000 2:27 AM To: zope-dev@zope.org Subject: [Zope-dev] multiple ZODB files?
Is it possible to set up a site such that as well as the single Data.fs file, a seperate folder (say, /foo) is loaded from a different ZODB file? In particular, changes to stuff inside /foo should be written to the foo ZODB file.
Not yet, we have not fully thought about this problem of multiple databases,
I'll add that we have thought quite a bit about it, just not enough. It's harder than one might at first expect.
but ZODB was architected to be multi-DB aware.
Unfortunately, I suspect more architectural work will be needed too. :(
Some issues:
- How to handle references accross databases. At a basic level, this is not too hard, but...
- How to handle garbage collection in a multi-db environment. Oops. Storages are currently responsible for GC. This won't account for references from other databases.
- How do databases get connected, from a UI point of view. Do you get to specify a database when you add an object? Any object? Where do databases get defined?
One idea I just thought of would be to have a databases area of the control panel where one could add database objects that modeled root database objects. Then, perhaps one would do the moral equivalent of a link from an object in one db to another.....
- .... There are probably other issues I haven't thought of.
If anyone wants to work on this, I'd be willing to set up a mailing list and provide advice.
Jim
-- Jim Fulton mailto:jim@digicool.com Technical Director (888) 344-4332 Python Powered! Digital Creations http://www.digicool.com http://www.python.org
Under US Code Title 47, Sec.227(b)(1)(C), Sec.227(a)(2)(B) This email address may not be added to any commercial mail list with out my permission. Violation of my privacy with advertising or SPAM will result in a suit for a MINIMUM of $500 damages/incident, $1500 for repeats.
_______________________________________________ Zope-Dev maillist - Zope-Dev@zope.org http://lists.zope.org/mailman/listinfo/zope-dev ** No cross posts or HTML encoding! ** (Related lists - http://lists.zope.org/mailman/listinfo/zope-announce http://lists.zope.org/mailman/listinfo/zope )
participants (5)
-
Dyon Balding -
ethan mindlace fremen -
Kevin Dangoor -
Lalo Martins -
Michel Pelletier