Jimmie Houchin wrote:
As far as usage is concerned I generally like the ZODB best because it is reasonably transparent to the building of a web app with Zope. However there are areas in which it does not currently excel that which if your site requires these skills then alternatives must be used. A couple of areas are data size and heavy writes.
This is often thought to be a deficiency in ZODB, but the root of this particular problem is really in FileStorage. Other Storages could implement much more write intensive abilities.
Some people use an RDBMS to solve these issues.
Right.
While this will work it does expect more from the developer. Some do not have the skills or the tools. Even if one does it still requires leaving the transparency of developing with the ZODB.
To get an orthogonal benefit, for write intensity you use a realtional database, instead of using a releational database to solve relational tasks.
Multiple file storage for the ZODB has been proposed as a solution
to a different problem. The question is not using multiple file storages in Zope, but just using multiple storages. This way you could use a FileStorage when you want its properties, BerkeleyStorage when you want something different, or some sort of storage based on a relational database at the same time in the same Zope system. An analogy of allowing Zope to 'mount' various storages into the object tree has been proposed, but it's a very tricky problem and not the one that creating a relational based storage will solve.
and there are 2 proposals currently on the ZODB ZWiki. I will add another.
Class/Object based db files.
Each class gets it's own db file. This could be similar to the current ZODB file except specific to a class. As objects are created they are appended to the db file for their class. This could be somewhat analogous to tables in an RDBMS.
Advantages would be spreading out the data space over multiple files which would help with some oses. Also I think that each class has different characteristics which would be able to be managed better if separate.
I'm not any thing of an expert on RDBMS, but we have thought this pretty though in-house, and this is not really the model we came up with. The interface of a storage object makes no assumptions about how objects are stored. A Storage is, in simple terms, a mapping from object id to a pickle. This can easily be a relational database table that is keyed on object id and contains CLOBs or BLOBs or whatever that represent the pickle. When ZODB needs to resolve a reference to an object id into an object is selects out of the object table the pickle (or pickles, who knows) it is looking for. When 'writes' are done, ZODB inserts new pickles with certain object ids. Some extra columns containing backlink references could allow undoing (and thus, sharing the same quick growing behavior file storage). Also, the fact that FileStorage *can't* be not undo-able (and not grow so rapidly) is because the FileStorage just appends to an end of a file. A relational database does not have any such restriction, and an relational Storage could be either undoable or not undoable. I'm not sure how your suggestion would be better than this, I don't know much about RDBMS, do they assume one file per table or database? That soulds like an implementation issue. Also, objects in classes would not be very fairly distributed. There would be gobs of ImplicitAcquirerWrappers and very few OFS.Application.Application. On Zope.org there are hundreds of classes, and over one hundred thousand objects in the database, not including previous versions. There could very well be over a million objects and their previous revisions.
Example: AutoParts You have an AutoParts class. The objects will change very little once created. However there are a lot of objects and news added periodically. This file will need packed seldom. It will also be simple to backup and not need backed up often as changes are periodic and regular.
RetailStore In a retail store the product objects are very volatile. Vendors can change. Prices do change. A productObject file would have different usage characteristics than the AutoParts object.
Some classes are perfect for few writes and many reads. Others less so.
Earlier Andrew Kuchling was wanting to walk the object tree. This would provide a relatively easy way to walk the object tree.
This could be implemented with some support classes which have to be inherited from to create a class.db file. Any class not so doing would go into the standard ZODB. This could help provide desired management features for the characteristics of each. It would be nice if in the management you could set the path to the file. This would allow for multiple disks or partitions for data storage. This too would help with backups and such.
Just a few ideas. They may not stand up to examination, but that's okay. I just thought I would put them on the table.
It does absolutly make sense to analyze your problems like this. The solutions to your needs here could be met if Zope supported multiple storages. Currently, BerkeleyStorage provides has proven high-write intensity and FileStorage is, of course, wildly useful for the often-read seldom written objects. A relational storage is also definatly needed. I'm suprised that more people don't use BerkeleyStorage today. Is this because it doesn't undo? I don't immagine it would be too dificult to extent it to support undo. Ty, what do you think? -Michel