I would like to request comments on our idea how to use different storages for our new model-driven approch with the name Devilstick. You dont need to know devilstick or its ideas in depth to give valuable input. More it would helps us to get input from people knowing zopes persistency layer in depth. Devilstick is model driven framework to describe and manage data inside and outside ZODB. some more information at http://devilstickproject.net At Blackforest sprint in august we researched how the goal to support different storages than ZODB can be achieved. After first thinking about an own layer we got there the idea of using the usal persistence and transaction API of zope. IIRC it was a result of a conversation between Florian Friesdorf and Roger Ineichen and probably others. Today is the last day of Bolzano sprint. We researched a lot how it currently works with ZODB and discussed about how to use all this framework for devilstick. Our outcome is a document describing what we found and how we want to use all this. It follows here. for the devilstick-team Jens Klein ========================================================================= ----------------------------------- Introduction to Devilstick Storages ----------------------------------- This document describes the future. One of Devilsticks power is to support different storages than ZODB easily. The storage layer uses 100% zopes persistency implementation. At some entry point we enter the model driven world of devilstick: We hit 'Cage'. The Cage itself is not a data-access-object (DAO). But its the bridge to the otherstorage layer. Inside Devilstick DAOs are still persistent objects. They may still live in ZODB. But they can live complete outside if it is needed. They may live in SQL-databases, in LDAP, filesystem or fetched over a webservice. For more about DAOs and its API please read API.txt. Excursus: Zopes Persistence Framework ------------------------------------- Classic zope objects are derived from 'persistence.Persistent'. Those objects are tracking themselfes for modifications. Once a modification is detected it joins it's data-manager to the current transaction-manager. All this happens in zope fully transparent. The data-manager is the key to the storage layer. Zope is designed to use different data-managers. Datamanagers are described well in 'transaction.interfaces.IDataManager'. They care about storing all data in a 2-phase commit. There is usally one data-manager for all modified object of one database. Transaction-manager collects all datamanagers (which are called resources inside the transaction-manager) with modifications. Once the transaction is committed the 2-phase commit is started: 1st 'tpc_begin' is called on each data-manager, 2nd the 'commit' is called for each, then 'tpc_vote' and finally 'tpc_finish'. After creation of a persistent object it has an attribute called '_p_jar' set to 'None'. _p_jar gets a datamanager set - almost magically - after it was added to a container. The datamanager taken there is copied over from the containers _p_jar attribute. Container and new object are marked as modifed and the datamanager joins the transaction. On commit both are written to the database. Devilstick persistency ---------------------- To provide other storages we alreay have a powerful framework: the persistent api and transaction api. Devilstick uses both. To use a different storage simply a new data-manager is needed. Anyway, for several uses-cases its fine to stay in the ZODB. Such a alternative datamanager might work different inside than the current ZODB one. Since we deal with SQL or LDAP we want to update a database with one query for several objects involved. So on commit we may need to look at the modified objects and build one sql-query from a bunch of modifications. Frameworks like SQLAlchemy may help us here for SQL and others are probably available for different use-cases. Entry-Points: Cages ------------------- We need one point where the datamanager is switched to a different storage.A model is assigned and there the world of generic DAOs is entered. This entry point is called 'Cage'. A cage is still persistent in the ZODB and uses the zopes default data-manager. A cage has the root container DAO (which is a generic molecule DAO) set as an attribute. Here some example code how it looks like: >>> cage = Cage() >>> cage._p_jar None >>> somezodbcontainer._p_jar <Connection at ... > >>> somezodbcontainer['data'] = cage >>> cage._p_jar <Connection at ... > >>> cage._root None >>> cage.model = 'examplemodel' >>> cage._root <Molecule at ...> >>> cage._root._p_jar <MyStoragesDataManager at ...> The cage also bridges the container API of the root molecule. It simplifies the usage of the API and avoids to introduce a extra access step on the cage. This way its more intuitive. >>> cage._root.keys() ['m1', 'm2', 'm3'] >>> cage.keys() ['m1', 'm2', 'm3'] >>> cage['m1] is cage._root['m1'] True