[Joachim Werner] As most of you might have recognized, there is a very active thread (actually two interwoven ones) about RO-mapping etc. going on on zope-dev. I'd be very happy if someone could wrap up the stuff from there on the SmartObjects list. ;-) [Albert] Thanks for the pointer. As I read them all through at once, after seeing your pointer, I made notes before joining the discussion, which may help, though far from a "wrap-up". * SUMMARY * There seem to be 6 separate topics: 1) Nature of SmartObjects, ZPatterns and Transwarp frameworks and relation to Zope/ZODB framework. 2) Improvements to ZCatalog search efficiency 3) Query syntax - see Zwiki: http://dev.zope.org/Wikis/DevSite/Proposals/UnionAndIntersectionOperations 4) Schema transparency (an oxymoron?) 5) Storage of ZODB objects in separate DBMS tables per class - "Object Mapping". 6) Requirements for "Object Relational Mapping". * DETAILED NOTES * Here's a list of all the items I found in the May archives with the term "ORMapping" in the subject line, from thread "Experiments in ORMapping" and interwoven thread "oodb philosophics was [above]" until I subscribed. Will comment later on subsequent messages rather than making notes. If there are any other subject headings missed, or other postings (including earlier months), please let me know. Highlights before links attempt to be objective but are limited by space, by my specific interests in the topic and by my lack of thorough understanding of Zope internals and consequent previous non-participation in [zope-dev] with consequent lack of appreciation of where various posters are "coming from". Quotes following links are verbatim extracts (not necessarily in same order or in context). "Must read" means important message with no or inadequate highlights or extracts given here. I have omitted numerous affirmations that ZODB meets most current Zope requirements (perhaps with improvements to ZCatalog) in view of the fact that nobody appeared to be arguing otherwise. BTW the date order used in list archives could be better in numeric order as that might more closely reflect which messages are likely to have been seen before replying. I have re-arranged. Starting from 10 May 2001: Shane Hathaway (DC) - start of discussion - must read http://lists.zope.org/pipermail/zope-dev/2001-May/011101.html includes link http://www.zope.org/Members/hathawsh/ormapping.tar.gz overview in file: sketch Tino Wildenhain - asks about ZPatterns, ZClasses, doesn't want "purely relational application server". http://lists.zope.org/pipermail/zope-dev/2001-May/011102.html SH - answers TW, says not hard to implement by mapping classes to a table. http://lists.zope.org/pipermail/zope-dev/2001-May/011103.html TW - answers SH, prefers improved OODB because doesn't like mapping classes to schema. http://lists.zope.org/pipermail/zope-dev/2001-May/011105.html SH - answers TW, enhanced ZODB storage (RDBMS and BerkelyDBM) is in some ways the best OODB. http://lists.zope.org/pipermail/zope-dev/2001-May/011107.html "The other motivations for an RDBMS are (1) people have existing schemas and want Zope to access the same data as their existing apps, and they want it to be transparent, and (2) tables with millions of entries are easily stored in Zope but the perception is that the catalog isn't as fast as a database index. No one has done any tests AFAIK." [...] "That's one reason ZODB is so nice. You can write an application without writing a formal schema." Casey Duncan (Kaivo) - suggests Matisse or Objectivity but limited by not supporting ZODB versions. BerkelyDBM best soon. Mentions slow XML storage startup. http://lists.zope.org/pipermail/zope-dev/2001-May/011108.html JW - already 2 OR mapping projects, SmartObjects and TransWarp, no duplication because Phillip from Transwarp participating in SmartObjects list. http://lists.zope.org/pipermail/zope-dev/2001-May/011109.html TW - answers CD, XML just another pickle format. http://lists.zope.org/pipermail/zope-dev/2001-May/011110.html SH - answers JW. SmartObjects, ZPatterns and Transwarp require new database API instead of maintaining transparency like ZODB. Projects should look at replacing parts of ZODB instead of adding complexity. http://lists.zope.org/pipermail/zope-dev/2001-May/011111.html "ZODB has pieces that can be split apart and replaced as needed, such as caching, persistence, transactions, the "pickle jar", the multi-threaded connection factory, and the storage layer. I'm hoping we can achieve OR mapping by only replacing the "pickle jar", i.e. Connection.py." Cees de Groot - answers SH. Confirms advantage of not having to write formal schema, migrating from PostgreSQL to ZODB for that. File Storage faster than Oracle and is basic transaction log so nothing could be more reliable. http://lists.zope.org/pipermail/zope-dev/2001-May/011124.html "Are people using ZODB for non-Zope data? I'd be very interested to discuss things like emulating extents, patterns for indexing, etcetera..." Chris Withers http://lists.zope.org/pipermail/zope-dev/2001-May/011125.html "doing this transparently would be unbelievably cool :-) So, if I had a.n.other SQL table containing some attributes, how would you see that mappign to Zope objects? What meta_type would they be? Where would they show up?" JW - answers SH, SmartObjects (SO) summary, must read. SO is framework, not just adding OR. Design assumes cannot change Zope core but would prefer if Zope extended to handle query and storage. Wants more efficient "cross-tree" queries than ZCatalog (or improved ZCatalog). SO is also "better ZClasses". Transparent for users but needs ZMI and DA plug-in as products. http://lists.zope.org/pipermail/zope-dev/2001-May/011129.html JW - answers CW, SO needs non-transparent API to define mapping. http://lists.zope.org/pipermail/zope-dev/2001-May/011130.html JW - answers SH, SO has something similar to ZClasses that defines the schemas to create both the classes and the tables on the fly. http://lists.zope.org/pipermail/zope-dev/2001-May/011131.html JW - speed tests should be done on ZODB. If could store full text of 30GB of Word and .pdf with ZCatalog indexes, and complete zope.org list archives efficiently would use for all document management needs. Also java ODBMS requires recompile for changes but Zope [ie python, not just ZODB - AL] can just refresh. ZODB has longer success record than other ODBMS. http://lists.zope.org/pipermail/zope-dev/2001-May/011134.html SH - answers JW. Adapting ZODB for OR can just replace Connection class in custom_zodb.py - no change to Zope core. ZCatalog not as flexible as an SQL query but can efficiently do filters on one table at a time. *Alternate* ZCatalog would be useful. CMF can restrict results based on user permissions. http://lists.zope.org/pipermail/zope-dev/2001-May/011146.html "Here's the goal I envision for OR mapping: to be able to move between OODBMS and RDBMS seamlessly. It makes sense to develop on top of ZODB then move to an RDBMS for testing and deployment." "It sounds like [SO] are tackling a lot of things at once. You might consider dividing into separate projects." "You see, I think it is not necessary to create a programming framework if the goal is OR mapping. The framework is already defined, and it's Python / ZODB. But SmartObjects seems to have many loosely related goals, making it difficult to assist." Ken Manheimer (DC) - answers JW. Speed tests done for mailing lists. Searches fast, loading slow. ZODB as flexible and powerful as he could want for python Zope programming. May have been canned searches, load testing may have resulted in improved tuning since. Eventually would like to incrementally stuff messages as they arrive. No experience with other ODBMS, very little with RDBMS. http://lists.zope.org/pipermail/zope-dev/2001-May/011147.html CD - answers CdG. Need better general query language than ZCatalog like those for most OODBMS and all RDBMS. http://lists.zope.org/pipermail/zope-dev/2001-May/011151.html David Brown - answers CD. Transparent access to and manipulation of data in python without having to think in SQL is more important than generic query language that can access different data sources, especially in view of SQL implementation differences. http://lists.zope.org/pipermail/zope-dev/2001-May/011152.html JW - answers KM. Fast search results promising. What exactly is problem for loading? Anything likely to be slow on large load. http://lists.zope.org/pipermail/zope-dev/2001-May/011153.html JW - answers DB. SQL can query objects in more than one container with an association table. No good way to implement or efficiently query "many to many" relations in object hierarchies. http://lists.zope.org/pipermail/zope-dev/2001-May/011154.html JW - answers SH. SO still collecting "feature requests". May need separate projects. SO not good name for most features of SO. Filtering on one table at a time inadequate for ad hoc data mining across many object relations. http://lists.zope.org/pipermail/zope-dev/2001-May/011155.html "But "Python/ZODB" is not a high-level framework. Like Zope is not the CMF. Nobody would implement GUIs in pure C now. They are using tools for that. And we don't want to have to implement things like HTML forms, event handling, etc. over and over again. We want to just configure and "compile". Of course this will be within the Python/ZODB framework, i.e. all the additional functionality will be "support classes" or mix-in classes like the ones Zope already offers (e.g. CatalogAware)." CD - answers DB. Python great for OO manipulation of data but not for generic queries. Python query language - needed eg criteria expression that returns objects meeting the criteria. ZCatalog argument list too limited. http://lists.zope.org/pipermail/zope-dev/2001-May/011157.html Reference to following link - must read. http://dev.zope.org/Wikis/DevSite/Proposals/UnionAndIntersectionOperations CdG - answers DB. Has Python objects above homebrew O-R layer, very useful. Should be possible to express queries as python code with mapping layer generating equivalent SQL from parse tree (like SmallTalk lambdas). http://lists.zope.org/pipermail/zope-dev/2001-May/011158.html DB - answers JW. Python can map specific indexes and mappings more easily and efficiently without re-writing database code or forcing developers to think in SQL. http://lists.zope.org/pipermail/zope-dev/2001-May/011159.html CD - answers CdG. Must be safe for "through the web" scripting. http://lists.zope.org/pipermail/zope-dev/2001-May/011160.html CdG - answers CD. http://lists.zope.org/pipermail/zope-dev/2001-May/011162.html "I think that TTW scripting and heavy duty application development are very incompatible with each other, so that's not a problem :-)" Phillip Eby - answers SH, JW. Explanation of Transwarp, must read. http://lists.zope.org/pipermail/zope-dev/2001-May/011163.html "I'm not quite clear on how exactly you suggest mapping from RDMBS -> ZODB. There's a *significant* (IMHO) impedance mismatch between ZODB's arbitrarily identified variably structured single records and SQL's content-identified fixed-structure record sets. This is what application frameworks/toolkits (such as your own DBAPI product) are needed for." "[Develop on ZODB then move to RDBMS requirement] can *only* be met through a common API or access pattern/framework/what-have-you, be it DBAPI, ZPatterns, SmartObjects, or TransWarp. The ZODB is both "too powerful" (in its flexibility) and "too weak" (in lack of any ZODB-level notions of record sets, schemas, and indexing) to be useful as a cross-database-platform API.[...]" "Anyway, all I'm really trying to say is, if you want implementation independence, you have to have implementation hiding. ZODB makes it easy to store Python objects with hardly a second thought. However, if you plan to use something other than ZODB for *any* part of your object at any time in future, you need an API that hides the implementation." SH - answers PE. Suggestion was to implement ZODB DB interface with different Connection.py. http://lists.zope.org/pipermail/zope-dev/2001-May/011164.html "If you implement this at the Storage level, yes, there is a major mismatch. But at the Connection level it makes a lot of sense. Connection basically exposes a pile of pickles as objects; an OR mapping exposes a complex schema as objects. I think that understanding will change the rest of your response. :-)" CdG - answers JW. http://lists.zope.org/pipermail/zope-dev/2001-May/011165.html "[...]AFAIK ZODB and most OODB's store an object only once, keyed by its object id. The rest is just references through that oid, so objects that belong to more than one container can be added to all these containers and n:m relations are implemented by having a list of objects on both sides." CdG answers PE. Can trigger external changes from application code without major risk or mess - gives example. http://lists.zope.org/pipermail/zope-dev/2001-May/011166.html CD - answers CdG. Acquisition may cause problems for n:m relations. http://lists.zope.org/pipermail/zope-dev/2001-May/011167.html CdG - answers CD. Doesn't want to use acquisition for business objects anyway. http://lists.zope.org/pipermail/zope-dev/2001-May/011168.html PE - answers SH. http://lists.zope.org/pipermail/zope-dev/2001-May/011171.html "Nope. As far as I can see, all that does is leverage ZODB Connections as a crude sort of Rack that only provides a mapping-like interface for retrieving objects, without helping any of the higher-order needs of an application. I guess if you define O-R mapping as "can you store and retrieve an object's properties from a database", then I guess you've succeeded at that point. But if that was all my O-R apps needed, there would've been no reason to use the RDBMS; I could've just used ZODB and a BTree with less bother." JW - Acquisition cool but sometimes sucks. http://lists.zope.org/pipermail/zope-dev/2001-May/011180.html JW - answers CdG. Use Symlinks (ref below) - should be in Zope. http://lists.zope.org/pipermail/zope-dev/2001-May/011181.html http://www.zope.org/Members/hathawsh/Symlink/index_html (13 May no further notes - thread still picking up steam, far too early to "wrap-up") * ELSEWHERE * Another relevant posting I noticed recently was from Nils Kassube in [zope], so I am CCing this to Nils as he seems not to be subscribed to the relevant lists: http://imail.iuveno-net.de/mailman/listinfo/smartobjects http://lists.zope.org/mailman/listinfo/zope-dev "According to the Zope Developer's Guide our favourite web application server prevents dirty reads with his transaction support. Is it possible to prevent non-repeatable reads and phantom reads, too? Or should I use a more appropriate tool like PostgreSQL with support for MVCC? Is there more documentation about Zope's transaction support? (Or is it another case for "Read the source, Luke"?)"