Re: [Zope-dev] Experiments with ORMapping
"Phillip J. Eby" wrote:
At 05:42 PM 5/11/01 -0400, Shane Hathaway wrote:
"Phillip J. Eby" wrote:
I'm not quite clear on how exactly you suggest mapping from RDMBS -> ZODB. There's a *significant* (IMHO) impedance mismatch between ZODB's arbitrarily identified variably structured single records and SQL's content-identified fixed-structure record sets. This is what application frameworks/toolkits (such as your own DBAPI product) are needed for.
If you implement this at the Storage level, yes, there is a major mismatch. But at the Connection level it makes a lot of sense. Connection basically exposes a pile of pickles as objects; an OR mapping exposes a complex schema as objects.
I think that understanding will change the rest of your response. :-)
Nope. As far as I can see, all that does is leverage ZODB Connections as a crude sort of Rack that only provides a mapping-like interface for retrieving objects, without helping any of the higher-order needs of an application. I guess if you define O-R mapping as "can you store and retrieve an object's properties from a database", then I guess you've succeeded at that point. But if that was all my O-R apps needed, there would've been no reason to use the RDBMS; I could've just used ZODB and a BTree with less bother.
I'm telling you there's a lot more you can do with the code that makes up ZODB. The mapping interface is not the key to the way Connection does its work. OIDs don't have to be strings. If we just use cPersistent, cPickleCache (which is misnamed), and Transaction to implement an OR mapping, here's what we get for free: - A transparent, transactional database. - Potential interoperability with everything written for ZODB. - Robust, tested code. - Optimizations in C. Since you challenged me :-) I decided to put together a prototype. What I came up with was a database connection that sets a different _p_jar for each persistent object. The _p_jar is an object that manages the storage for only one persistent object, making it possible to customize the storage. There was only one hurdle in this approach but I hacked around it: the tpc_* messages get sent to each _p_jar. Then I wrote a very minimal test that doesn't connect to an RDBMS but stores and retrieves simple objects in a dictionary. OIDs are still necessary to support multithreaded invalidation messages. But the OIDs in the prototype are a tuple consisting of a schema ID and a record ID. That way the record IDs only need to be unique within an RDBMS table (or combination of tables.) I didn't think anything like this was possible before I got to know Jim. I still didn't understand when he presented the idea months ago. But now I see the idea really is workable. The advantage is that it lets us completely isolate RDBMS storage details from the application. The next thing to do is to write a fishbowl proposal. Shane
Shane Hathaway wrote:
I'm telling you there's a lot more you can do with the code that makes
<snip>
The next thing to do is to write a fishbowl proposal.
This sounds cool but made my head hurt :-S Can you try and bring this back down to the level of us mere mortals by explaining how your OR stuff would let me take a table of data in an RDBMS table and have it appear as objects in the Management Inteferace? I know that's horribly oversimplified, but some of us only have small brains ;-) cheers, Chris
Chris Withers wrote:
Shane Hathaway wrote:
I'm telling you there's a lot more you can do with the code that makes
<snip>
The next thing to do is to write a fishbowl proposal.
This sounds cool but made my head hurt :-S
Can you try and bring this back down to the level of us mere mortals by explaining how your OR stuff would let me take a table of data in an RDBMS table and have it appear as objects in the Management Inteferace?
Sorry, this is at a pretty low level and I do need to explain it better. One would define an "ObjectMappingSchema" whose job it is to store and retrieve objects of a specific type and in a specific location. It would usually grab a database connection object to do its work. When loading, it would perform a query then manually put attributes into a persistent object. When storing, it would grab specific attributes from the persistent object and execute a statement to store those attributes. So let's say you want a ZODB to store and retrieve users in a specific table while putting everything else in pickles. You would create an instance of PickleSchema, which implements the ObjectMappingSchema interface, and tell it to manage everything *except* the users mapping in BasicUserFolder objects. You would tell it to store and retrieve this object using your UserFolderSchema instead. Your UserFolderSchema would store and retrieve the users from the USERS and USER_PREFS tables. The user table wouldn't require the use of OIDs but would require unique user IDs. So in the management interface nothing would change. Nor would the application-level Python code. You would only define a layer that maps objects to a relational database. You would still see the user folder as you do now. Now, it may be useful to provide a management interface for defining the schema mapping. I haven't approached that yet; AFAICT this is where the work done on SmartObjects and DBObjects would be very useful. Initially I was planning for people to code the mapping purely in Python so we could gain experience and find common patterns before inventing a UI. Shane
Shane Hathaway wrote:
One would define an "ObjectMappingSchema" whose job it is to store and retrieve objects of a specific type and in a specific location. It would usually grab a database connection object to do its work. When loading, it would perform a query then manually put attributes into a persistent object. When storing, it would grab specific attributes from the persistent object and execute a statement to store those attributes.
How does this differ from ZPatterns? Phil?
Now, it may be useful to provide a management interface for defining the schema mapping. I haven't approached that yet; AFAICT this is where the work done on SmartObjects and DBObjects would be very useful. Initially I was planning for people to code the mapping purely in Python so we could gain experience and find common patterns before inventing a UI.
Sounds good to me :-) Thanks for the explanation... cheers, Chris
Chris Withers wrote:
Shane Hathaway wrote:
One would define an "ObjectMappingSchema" whose job it is to store and retrieve objects of a specific type and in a specific location. It would usually grab a database connection object to do its work. When loading, it would perform a query then manually put attributes into a persistent object. When storing, it would grab specific attributes from the persistent object and execute a statement to store those attributes.
How does this differ from ZPatterns? Phil?
ZPatterns implements storage logic on the application level. Applications have to be aware of (in fact they have to be centered around) ZPatterns. This alternate approach keeps storage logic independent of application logic. It lets you take any code written for the ZODB and move it to your RDBMS schema without changing the code (most of the time :-) ). Shane
Shane Hathaway wrote:
ZPatterns implements storage logic on the application level. Applications have to be aware of (in fact they have to be centered around) ZPatterns. This alternate approach keeps storage logic independent of application logic. It lets you take any code written for the ZODB and move it to your RDBMS schema without changing the code (most of the time :-) ).
Now that sounds really f^%$^ing cool :-) How can I help? Chris
Chris Withers wrote:
Shane Hathaway wrote:
ZPatterns implements storage logic on the application level. Applications have to be aware of (in fact they have to be centered around) ZPatterns. This alternate approach keeps storage logic independent of application logic. It lets you take any code written for the ZODB and move it to your RDBMS schema without changing the code (most of the time :-) ).
Now that sounds really f^%$^ing cool :-) How can I help?
For now, I can make public the code I wrote for experimentation: http://www.zope.org/Members/hathawsh/orconn.tar.gz The archive contains two Python files and a patch. Put them in the ZODB directory. Perform the patch to DB.py. (It's very minimal.) Then run testOR.py. It just changes a mapping, but it does it transparently and transactionally. :-) I need to fishbowl this before going any further. Shane
At 12:26 PM 5/14/01 -0400, Shane Hathaway wrote:
Chris Withers wrote:
Shane Hathaway wrote:
One would define an "ObjectMappingSchema" whose job it is to store and retrieve objects of a specific type and in a specific location. It would usually grab a database connection object to do its work. When loading, it would perform a query then manually put attributes into a persistent object. When storing, it would grab specific attributes from the persistent object and execute a statement to store those attributes.
How does this differ from ZPatterns? Phil?
ZPatterns implements storage logic on the application level. Applications have to be aware of (in fact they have to be centered around) ZPatterns. This alternate approach keeps storage logic independent of application logic. It lets you take any code written for the ZODB and move it to your RDBMS schema without changing the code (most of the time :-) ).
I am reminded of a passage from "Dirk Gently's Holistic Detective Agency", wherein Dirk decides to cut to the heart of the problem, and simply writes down the answer. Now, he declares, he is faced with only a relatively much easier problem: What strange language did he write the answer down in? While I'm not saying that what you're doing isn't possible, I *will* say that I believe you are replacing a small amount of API-specificity in the code with a similar amount of specificity, coupled with a much larger complexity in the mapping layers. It is, IMHO, much harder to write generic mappings at the schema layer where you propose, than to do so at the domain logic layer where ZPatterns operates. Ty and I spent many weeks chewing over designs in the space you're talking about, back before ZPatterns existed, after consulting at length with Jim Fulton and Michel Pelletier. Our conclusion was that yes, it was *possible* to do the mapping you're talking about, but that it was very high impedance for the kinds of applications we had in mind. I'll give a simple example to show what I'm talking about. Consider a task framework where TaskPerformers are assigned Tasks, in a simple one-to-many relationship. TaskPerformers, in this framework, perform up to hundreds of tasks per day, thousands per month. Assuming an RDBMS backend, how do you propose to map this in your storage model? My thought on this, is that you will be forced to explicitly consider the nature of this relationship and its storage at the application level. If you write explicitly for ZODB, you might use a BTree, for example. Or perhaps you'd have some kind of global container for the Tasks, with a Catalog to index them, and a method on TaskPerformer to query the Catalog. How would you propose to map this to an RDMS? Well, substituting catalog queries might be relatively straightforward, but perhaps rather time consuming to develop in a generic way. The BTree might give you a bit more trouble. Probably you'd have to end up using some sort of generic queryable containers that translated to BTrees or Catalogs or RDBMS tables, depending... And lo! You now have an application framework. Oops. We haven't even addressed performance issues yet, which are the real killer. The only way (again IMHO) to get reasonably high performance that is "portable" across different databases is to be able to write code that is optimized based on the back-end. The only way you can push this code "under the hood" is to create an all-purpose query mechanism. Otherwise, you must expose enough of the inner workings to the application developer that they can include "platform-specific" code for each circumstance. All this is of course just my opinion, and quite possibly wrong. But I think that an application developer would get more mileage out of your DBAPI product (coupled with some utility classes to replace Racks) or from ZPatterns than they would from this approach to O-R mapping.
"Phillip J. Eby" wrote:
All this is of course just my opinion, and quite possibly wrong. But I think that an application developer would get more mileage out of your DBAPI product (coupled with some utility classes to replace Racks) or from ZPatterns than they would from this approach to O-R mapping.
Your opinion is highly valued, and I agree there are unknown quantities at this point. But I think the unknowns are not significant enough to drop the idea. The goal is transparent persistence with an arbitrary RDBMS schema. Many developers prefer to code for ZODB but customers demand SQL. This would bridge the gap. Regarding performance, this method is actually ideal IMHO. Data is read once, converted to an object, and kept for later connections, just like ZODB. Also, Jim has suggested database-specific features for querying, etc. and abstracting these does require application-level logic. The goal is not to avoid all application logic, just to move as much as possible to the storage layer. (My statement about this approach not using any application logic was thus inaccurate.) Shane
At 04:13 PM 5/14/01 -0400, Shane Hathaway wrote:
"Phillip J. Eby" wrote:
All this is of course just my opinion, and quite possibly wrong. But I think that an application developer would get more mileage out of your DBAPI product (coupled with some utility classes to replace Racks) or from ZPatterns than they would from this approach to O-R mapping.
Your opinion is highly valued, and I agree there are unknown quantities at this point. But I think the unknowns are not significant enough to drop the idea. The goal is transparent persistence with an arbitrary RDBMS schema. Many developers prefer to code for ZODB but customers demand SQL. This would bridge the gap.
The principal "unknown" is deciding which implementations will be used for which objects. This is an implementation-level decision, but which requires domain-level knowledge. You will find, I think, that this necessarily implies an "application framework" which deals with these decisions. You may choose different ways of specifying and patterning these decision idioms than ZPatterns, but you will likely end up covering the same decision territory if you seek the same kinds of capabilities.
Regarding performance, this method is actually ideal IMHO. Data is read once, converted to an object, and kept for later connections, just like ZODB.
[shrug] Not any different than ZPatterns, except that Racks drop their caches after every transaction to ensure freshness (since few RDBMS have asynchronous cache invalidation messages available). Of course, one can always tell the SQL methods to keep their results longer. Of course, ZPatterns doesn't always "read data once" - it often reads data *zero* times, if it's not needed during that transaction. Ty and I have written applications where certain data was in LDAP and other data in SQL, and either one or the other was needed for most transactions. So ZPatterns only loads the attribute sets you need during that transaction.
Also, Jim has suggested database-specific features for querying, etc. and abstracting these does require application-level logic. The goal is not to avoid all application logic, just to move as much as possible to the storage layer. (My statement about this approach not using any application logic was thus inaccurate.)
Which is why I think all you're going to end up with is an alternative implementation of what ZPatterns already does. :) I'm not opposed to that (not that it would mean anything if I were!), but I'd suggest taking a closer look at what's been done in that arena before investing a lot of time in it. (May I suggest especially taking a look at SkinScript's provisions for storage and retrieval from *any* database, not just RDMBS'?) Anyway, if you can improve on it and put it into Zope proper, great! I'm off the hook for new ZPatterns releases. :) But I think all you're really proposing compared to ZPatterns is to: 1) Simplify Racks' attribute loading to compute all attributes at once (i.e. dropping ZPatterns' load-on-demand attribute groups) 2) Replace the Specialist-per-Interface approach of ZPatterns with a monolithic connection object which has (directly or indirectly) knowledge of storage rules for all application classes (i.e. reducing information hiding and composability of separate application components) 3) Move the implementation of mapping rules, etc. out of the ZODB/through-the-web editing approach of ZPatterns and into Python code (Not sure on this one, you've been kind of vague about this part) 4) Maybe refine the caching compared to ZPatterns so that data can hang around longer than per-transaction - but only if you can get invalidation messages from the source DB, which is a nontrivial exercise in itself. (You might be able to create your own protocol, however, for broadcasting private invalidation messages among peers.) (Interestingly, the above are all directions which we originally thought to go with ZPatterns, or attempted in early drafts, but abandoned for reasons mostly implied above.) 5) Use an API which emulates the existing persistence interface to the extent possible - which for most applications will basically mean that you will be able to set and get attributes the normal Python way - as long as they're relatively simple attributes. (Same as ZPatterns does.) For more complex things, things will be more complex. (Again, same as with ZPatterns.) Anyway, all that having been said, my Yahoo! horoscope said today that I should "Be proud, but don't be obnoxious," so I will stop here. :)
On Mon, 14 May 2001, Phillip J. Eby wrote:
At 04:13 PM 5/14/01 -0400, Shane Hathaway wrote:
Regarding performance, this method is actually ideal IMHO. Data is read once, converted to an object, and kept for later connections, just like ZODB.
[shrug] Not any different than ZPatterns, except that Racks drop their caches after every transaction to ensure freshness (since few RDBMS have asynchronous cache invalidation messages available).
Hey, that would actually work here too. Good idea!
Of course, one can always tell the SQL methods to keep their results longer. Of course, ZPatterns doesn't always "read data once" - it often reads data *zero* times, if it's not needed during that transaction. Ty and I have written applications where certain data was in LDAP and other data in SQL, and either one or the other was needed for most transactions. So ZPatterns only loads the attribute sets you need during that transaction.
This new scheme already does the same, but the approach is to create "ghosts" instead of using hooks. Ghosts have proven to be far more reliable than __getattr__ hooks. The proof of their reliability is in the fact that programmers are very rarely aware of them.
Also, Jim has suggested database-specific features for querying, etc. and abstracting these does require application-level logic. The goal is not to avoid all application logic, just to move as much as possible to the storage layer. (My statement about this approach not using any application logic was thus inaccurate.)
Which is why I think all you're going to end up with is an alternative implementation of what ZPatterns already does. :) I'm not opposed to that (not that it would mean anything if I were!), but I'd suggest taking a closer look at what's been done in that arena before investing a lot of time in it. (May I suggest especially taking a look at SkinScript's provisions for storage and retrieval from *any* database, not just RDMBS'?)
The primary goal is RDBMS integration, but the code will be reusable for other data access.
Anyway, if you can improve on it and put it into Zope proper, great! I'm off the hook for new ZPatterns releases. :) But I think all you're really proposing compared to ZPatterns is to:
1) Simplify Racks' attribute loading to compute all attributes at once (i.e. dropping ZPatterns' load-on-demand attribute groups)
More fundamentally, developers don't have to learn about "racks". :-)
2) Replace the Specialist-per-Interface approach of ZPatterns with a monolithic connection object which has (directly or indirectly) knowledge of storage rules for all application classes (i.e. reducing information hiding and composability of separate application components)
I didn't mean to give that impression. No monolithic connection objects. The experimental code in orconn.tar.gz shows what I have in mind for now. Some slight changes to cPersistence would make it a lot cleaner.
3) Move the implementation of mapping rules, etc. out of the ZODB/through-the-web editing approach of ZPatterns and into Python code (Not sure on this one, you've been kind of vague about this part)
Not exactly. It's easier to write Python code first, that's all.
4) Maybe refine the caching compared to ZPatterns so that data can hang around longer than per-transaction - but only if you can get invalidation messages from the source DB, which is a nontrivial exercise in itself. (You might be able to create your own protocol, however, for broadcasting private invalidation messages among peers.)
We want to open up this possibility, yes.
(Interestingly, the above are all directions which we originally thought to go with ZPatterns, or attempted in early drafts, but abandoned for reasons mostly implied above.)
You don't have the same goals we do. You pursued the thing that was right for you.
5) Use an API which emulates the existing persistence interface to the extent possible - which for most applications will basically mean that you will be able to set and get attributes the normal Python way - as long as they're relatively simple attributes. (Same as ZPatterns does.) For more complex things, things will be more complex. (Again, same as with ZPatterns.)
Attributes will work just like they do in ZODB. Objects are transparently "de-ghostified" when attributes are accessed. It seems to me that ZPatterns makes this process complex by managing each attribute individually through the use of __getattr__(), which this approach does not need. I could be misunderstanding your use of "complex". Shane
My thought on this, is that you will be forced to explicitly consider the nature of this relationship and its storage at the application level. If you write explicitly for ZODB, you might use a BTree, for example. Or perhaps you'd have some kind of global container for the Tasks, with a Catalog to index them, and a method on TaskPerformer to query the Catalog.
How would you propose to map this to an RDMS? Well, substituting catalog queries might be relatively straightforward, but perhaps rather time consuming to develop in a generic way. The BTree might give you a bit more trouble. Probably you'd have to end up using some sort of generic queryable containers that translated to BTrees or Catalogs or RDBMS tables, depending...
And lo! You now have an application framework. Oops.
I'll try to somehow get your and Shane's opinions together a bit: I think Shane's approach does most of the job (of OR mapping) if the objects we are talking about are relatively simple and the application domain is relatively typical for what Zope can do well. E.g. the composite objects we have in mind for the groupware (user credentials coming from LDAP, rest in the ZODB, or file stored in the filesystem metadata in SQL DB, rest in ZODB) can easily be modelled this way. And the existing Zope API (manage_AddX, manage_changeProperties etc.) would do most of the job. But the problems start with querying. Can the ZCatalog do it? Does it make sense to implement a "SQLZCatalog" instead of just offering native SQL query objects or an abstract query language like SkinSkript? To put it simply: The reason why we want to develop a "framework" (or framework extension) is that the Zope framework just doesn't do the stuff we need yet. BTW, the reason why we didn't just use ZPatterns was very similar: ZPatterns makes it easy to write an abstract implementation of a project, but you still have to write all the "glue code" between the storage and the application layer by hand. What we want (and what TransWarp is about, with a slightly different focus) is that we can just map an attribute to a data source and all the glue code (either ZODB or SQL statements) is auto-generated. If the new "framework" just extends the existing Zope API, I don't see a problem. ZPatterns did not, but maybe it is just not that easy to do so (and not all parts of the Zope API really should bear this name ...). The really good thing about the whole discussion is that at the end we will see much better where we should put things. We (the SmartObjects people at iuveno) do not really want to have a completely new world of SmartObjects that will require a new API to learn. We just want to add new functionality to Zope, be it in the core or as mix-in classes, or through-the-web products. Joachim
My thought on this, is that you will be forced to explicitly consider the nature of this relationship and its storage at the application level. If you write explicitly for ZODB, you might use a BTree, for example. Or perhaps you'd have some kind of global container for the Tasks, with a Catalog to index them, and a method on TaskPerformer to query the Catalog.
How would you propose to map this to an RDMS? Well, substituting catalog queries might be relatively straightforward, but perhaps rather time consuming to develop in a generic way. The BTree might give you a bit more trouble. Probably you'd have to end up using some sort of generic queryable containers that translated to BTrees or Catalogs or RDBMS tables, depending...
And lo! You now have an application framework. Oops.
I'll try to somehow get your and Shane's opinions together a bit: I think Shane's approach does most of the job (of OR mapping) if the objects we are talking about are relatively simple and the application domain is relatively typical for what Zope can do well. E.g. the composite objects we have in mind for the groupware (user credentials coming from LDAP, rest in the ZODB, or file stored in the filesystem metadata in SQL DB, rest in ZODB) can easily be modelled this way. And the existing Zope API (manage_AddX, manage_changeProperties etc.) would do most of the job. But the problems start with querying. Can the ZCatalog do it? Does it make sense to implement a "SQLZCatalog" instead of just offering native SQL query objects or an abstract query language like SkinSkript? To put it simply: The reason why we want to develop a "framework" (or framework extension) is that the Zope framework just doesn't do the stuff we need yet. BTW, the reason why we didn't just use ZPatterns was very similar: ZPatterns makes it easy to write an abstract implementation of a project, but you still have to write all the "glue code" between the storage and the application layer by hand. What we want (and what TransWarp is about, with a slightly different focus) is that we can just map an attribute to a data source and all the glue code (either ZODB or SQL statements) is auto-generated. If the new "framework" just extends the existing Zope API, I don't see a problem. ZPatterns did not, but maybe it is just not that easy to do so (and not all parts of the Zope API really should bear this name ...). The really good thing about the whole discussion is that at the end we will see much better where we should put things. We (the SmartObjects people at iuveno) do not really want to have a completely new world of SmartObjects that will require a new API to learn. We just want to add new functionality to Zope, be it in the core or as mix-in classes, or through-the-web products. Joachim
I think this is a great idea! I would definetely like to use and contribute to this effort. Having this kind of flexibily would be fantastic. After demonstratable Python code is working I would request that usability issues (UI Schema mapper, data migration/schema evolution tools, ZEO integration, multi-Storage uses) be addressed sooner than later. John On Monday 14 May 2001 10:47, Shane Hathaway wrote:
Chris Withers wrote:
Shane Hathaway wrote:
I'm telling you there's a lot more you can do with the code that makes
<snip>
The next thing to do is to write a fishbowl proposal.
This sounds cool but made my head hurt :-S
Can you try and bring this back down to the level of us mere mortals by explaining how your OR stuff would let me take a table of data in an RDBMS table and have it appear as objects in the Management Inteferace?
Sorry, this is at a pretty low level and I do need to explain it better.
One would define an "ObjectMappingSchema" whose job it is to store and retrieve objects of a specific type and in a specific location. It would usually grab a database connection object to do its work. When loading, it would perform a query then manually put attributes into a persistent object. When storing, it would grab specific attributes from the persistent object and execute a statement to store those attributes.
So let's say you want a ZODB to store and retrieve users in a specific table while putting everything else in pickles. You would create an instance of PickleSchema, which implements the ObjectMappingSchema interface, and tell it to manage everything *except* the users mapping in BasicUserFolder objects. You would tell it to store and retrieve this object using your UserFolderSchema instead. Your UserFolderSchema would store and retrieve the users from the USERS and USER_PREFS tables. The user table wouldn't require the use of OIDs but would require unique user IDs.
So in the management interface nothing would change. Nor would the application-level Python code. You would only define a layer that maps objects to a relational database. You would still see the user folder as you do now.
Now, it may be useful to provide a management interface for defining the schema mapping. I haven't approached that yet; AFAICT this is where the work done on SmartObjects and DBObjects would be very useful. Initially I was planning for people to code the mapping purely in Python so we could gain experience and find common patterns before inventing a UI.
Shane
_______________________________________________ Zope-Dev maillist - Zope-Dev@zope.org http://lists.zope.org/mailman/listinfo/zope-dev ** No cross posts or HTML encoding! ** (Related lists - http://lists.zope.org/mailman/listinfo/zope-announce http://lists.zope.org/mailman/listinfo/zope )
-- . . . . . . . . . . . . . . . . . . . . . . . . John D. Heintz | Senior Engineer 1016 La Posada Dr. | Suite 240 | Austin TX 78752 T 512.633.1198 | jheintz@isogen.com w w w . d a t a c h a n n e l . c o m
Now, it may be useful to provide a management interface for defining the schema mapping. I haven't approached that yet; AFAICT this is where the work done on SmartObjects and DBObjects would be very useful. Initially I was planning for people to code the mapping purely in Python so we could gain experience and find common patterns before inventing a UI.
Our initial plans for SmartObjects (if I get Stephan right) are to define all objects in Python code (as DBObjects do now), but to also provide two alternative ways of creating the necessary code: a) by importing and "compiling" an XML object description (which might come from a UML tool) b) via a ZMI interface similar to the ZClass GUI (which may or may not use the XML schemas, depending on the actual implementation) Of course it would be necessary to dynamically refresh the newly generated filesystem-based code. The other issue that applies in general to all filesystem-based code is that it is not directly available over ZEO as it is not in the ZODB. But that should not be a very large problem ...
A way to approach how this would be useful would be to contrast different use environments for ZODB code. The same code could be used for: - Unit testing, prototyping, and general exploration could use ZODB with DemoStorage and the default Connection objects. - Production use could make use of ZODB and custom Connection objects that map to/from relational tables. John On Monday 14 May 2001 09:51, Chris Withers wrote:
Shane Hathaway wrote:
I'm telling you there's a lot more you can do with the code that makes
<snip>
The next thing to do is to write a fishbowl proposal.
This sounds cool but made my head hurt :-S
Can you try and bring this back down to the level of us mere mortals by explaining how your OR stuff would let me take a table of data in an RDBMS table and have it appear as objects in the Management Inteferace?
I know that's horribly oversimplified, but some of us only have small brains ;-)
cheers,
Chris
_______________________________________________ Zope-Dev maillist - Zope-Dev@zope.org http://lists.zope.org/mailman/listinfo/zope-dev ** No cross posts or HTML encoding! ** (Related lists - http://lists.zope.org/mailman/listinfo/zope-announce http://lists.zope.org/mailman/listinfo/zope )
-- . . . . . . . . . . . . . . . . . . . . . . . . John D. Heintz | Senior Engineer 1016 La Posada Dr. | Suite 240 | Austin TX 78752 T 512.633.1198 | jheintz@isogen.com w w w . d a t a c h a n n e l . c o m
Hello, i want to thank DC (jim, shane, and paul) for inviting me to come to the new DC offices. i had a great time and learned a bunch... and met the BFDL. i gave a quick overview of the smartobjects design/framework and jim and shane presented some ideas for an or mapping for the zodb. to be honest it seemed both extremely elegant and slightly disturbing at the sametime. I've grown accustomed to treating the ZODB as a magic black box, that just works. integrating an or layer at this low level disturbs me as it meant that application developers would need to tinker at the lowest level of the zope framework, plus the dangers that to properly support application development the zodb would need to be bloated with accessory apis. at the same time it offers a really elegant way to make the or mapping available to python programmers on a pure python level (which is something i'm also interested in). for both approaches there are some similiar problems that must be tackled, namely definition of the or mapping, generation of stub python classes, integration with zope notions (permission, etc). so hopefully discussion here will flesh out, to the point of making a choice clear. i'm primarily focused on doing the or mapping definition in xml, as its language neutral and allows me to leverage other tools (like torque java.apache.org/turbine/torque.html) to assist in application development/porting. i've thought about it for a few days, and while i'm not convinced completely that a zodb approach is the best method, i'm going to start work on the some of the support classes nesc. for moving forward, namely some of the xml model/generators outlined in my design (embryonic) (http://www.zope.org/Members/k_vertigo/smartobjects) most of my concerns echo the issues that philip has brought up re the need of application developers. i do feel some if not most of these issues can be solved, i'm just wondering if for zope this is the best solution. for pure python development it seems alright. for zope, i wonder if a higher level application framework is more suitable. to clarify some existing points in this thread to date. - (IMO) zpatterns is about two things, delegation and data abstraction at the application level. the zpatterns framework doesn't assist in automating this and its up to the integrator/developer to manually wire together all the app/business objects. the smartobjects framework is about automating development and presenting easier development interfaces for both developers and designers. this isn't to say that such automation and interfaces couldn't be built on top of zpatterns, just to say that there are some different design goals. - one of the primary goals of smartobjects is easy integration with zope in a manner that is natural for zope developers. this does not mandate a new db access api, its about facilitating automation of this api so that the objects can be treated as normaly zope objects. both approaches (smartobjects and or zodb) must still tackle similiar fundamental issues to integrate with zope. on the zodb level this seems complicated to me (which betrays my incomplete understanding of the zodb internals), plus it requires some additions to the zodb interface to allow effective application usage (namely a generic query mechanism, and probably interogation of a class information). - one of my main objectives in developing an or mapping is to use the acs4 (www.arsdigita.com) applications natively (or as close as possible) within zope. - my smartobjects design (www.zope.org/Members/k_vertigo/smartobjects) is mine, in that smartobjects is primarily a iuveno project this does not mean it is smartobjects. stephen richter (of iuveno) liked parts of the design and will likely use it parts of it in a iuveno implementation but what comes out of iuveno is not limited or beholden in anyway by my proposal. the key focus of the proposal is a metamodel (schema) based approach. questions - i think any integration of or layer should be complementary to the zope framework. shane's existing examples and discussion of an or layer seem to be more geared towards replacing the zodb, than complementing. is the idea to use some sort of zodb product as a mounted storage?, or is this a replacement strategy. cheers kapil
On Mon, 14 May 2001, ender wrote:
i want to thank DC (jim, shane, and paul) for inviting me to come to the new DC offices. i had a great time and learned a bunch... and met the BFDL.
It was good to talk with you!
i gave a quick overview of the smartobjects design/framework and jim and shane presented some ideas for an or mapping for the zodb.
to be honest it seemed both extremely elegant and slightly disturbing at the sametime. I've grown accustomed to treating the ZODB as a magic black box, that just works. integrating an or layer at this low level disturbs me as it meant that application developers would need to tinker at the lowest level of the zope framework, plus the dangers that to properly support application development the zodb would need to be bloated with accessory apis. at the same time it offers a really elegant way to make the or mapping available to python programmers on a pure python level (which is something i'm also interested in).
Jim's ideas are often disturbingly elegant and difficult to understand all at the same time. :-) This reminds me of a recent slashdot thread that talked about a photographer in the days of black and white film who had devised a scheme for preserving color, and the article presented the results. It was disturbing for readers to realize that life in past centuries actually was in color! Programmers really wouldn't have to tinker with this layer. They would just write a schema and a little glue code that this layer understands.
for both approaches there are some similiar problems that must be tackled, namely definition of the or mapping, generation of stub python classes, integration with zope notions (permission, etc). so hopefully discussion here will flesh out, to the point of making a choice clear.
i'm primarily focused on doing the or mapping definition in xml, as its language neutral and allows me to leverage other tools (like torque java.apache.org/turbine/torque.html) to assist in application development/porting.
i've thought about it for a few days, and while i'm not convinced completely that a zodb approach is the best method, i'm going to start work on the some of the support classes nesc. for moving forward, namely some of the xml model/generators outlined in my design (embryonic) (http://www.zope.org/Members/k_vertigo/smartobjects)
most of my concerns echo the issues that philip has brought up re the need of application developers. i do feel some if not most of these issues can be solved, i'm just wondering if for zope this is the best solution. for pure python development it seems alright. for zope, i wonder if a higher level application framework is more suitable.
It all depends on what layer you're developing for, I guess. In the web interface you can't use rich object hierarchies as fluidly as you can in Python. On the web level you don't care as much about transparent persistence, but in Python you do.
to clarify some existing points in this thread to date.
- (IMO) zpatterns is about two things, delegation and data abstraction at the application level. the zpatterns framework doesn't assist in automating this and its up to the integrator/developer to manually wire together all the app/business objects. the smartobjects framework is about automating development and presenting easier development interfaces for both developers and designers. this isn't to say that such automation and interfaces couldn't be built on top of zpatterns, just to say that there are some different design goals.
Let's say you want CMF objects to be stored in an RDBMS. You don't want to use ZODB stubs and you want a tree of persistent objects. ZPatterns can't help you. Can SmartObjects?
- one of the primary goals of smartobjects is easy integration with zope in a manner that is natural for zope developers. this does not mandate a new db access api, its about facilitating automation of this api so that the objects can be treated as normaly zope objects. both approaches (smartobjects and or zodb) must still tackle similiar fundamental issues to integrate with zope. on the zodb level this seems complicated to me (which betrays my incomplete understanding of the zodb internals), plus it requires some additions to the zodb interface to allow effective application usage (namely a generic query mechanism, and probably interogation of a class information).
Here's another way to look at this: if SmartObjects doesn't use code from the ZODB, it could be reinventing parts of it. It seems like a good idea to get to know ZODB well before proceeding too far.
- one of my main objectives in developing an or mapping is to use the acs4 (www.arsdigita.com) applications natively (or as close as possible) within zope.
Sounds great!
- my smartobjects design (www.zope.org/Members/k_vertigo/smartobjects) is mine, in that smartobjects is primarily a iuveno project this does not mean it is smartobjects. stephen richter (of iuveno) liked parts of the design and will likely use it parts of it in a iuveno implementation but what comes out of iuveno is not limited or beholden in anyway by my proposal. the key focus of the proposal is a metamodel (schema) based approach.
Take a look at my proposal and see how closely our goals match. http://dev.zope.org/Wikis/DevSite/Proposals/ORMappingDB
- i think any integration of or layer should be complementary to the zope framework. shane's existing examples and discussion of an or layer seem to be more geared towards replacing the zodb, than complementing. is the idea to use some sort of zodb product as a mounted storage?, or is this a replacement strategy.
Either or both. :-) One could: - store most objects in ZODB while putting certain pieces of the hierarchy in the RDBMS, - mount an RDBMS, or - store objects for which a schema has not been written in a special table, either pickled or exploded. Shane
participants (6)
-
Chris Withers -
ender -
Joachim Werner -
John D. Heintz -
Phillip J. Eby -
Shane Hathaway