On Thu, 2002-08-01 at 15:04, Shane Hathaway wrote:

There are a few products on zope.org for integrating with CVS. Look through the downloadable products. I think there are CVSFolder and ZCVSFile (or something like that), maybe more.

However, most version control tools are difficult to integrate into applications in a platform-neutral way. To integrate with CVS, for example, Zope spawns a CVS client, but the CVS client runs with the privileges of the Zope process rather than the user's privileges. It works but it's not very clean.

Subversion, however, gives us a new opportunity. The first alpha has been released. Communicating with a Subversion repository involves HTTP, WebDAV, and DeltaV--all documented, open standards. So Zope could talk to a Subversion repository in a very transparent, well-integrated way. Anyone out there looking to write a nice, useful product for Zope? A Zope->Subversion adapter would be a great choice. :-)

http://subversion.tigris.org

Shane

Hi Shane. I've been thinking about Zope versioning, and I also did a bit of list searches for past discussions on this general topic. One problem that seems pertinent to really any external-to-zope versioning system, including Subversion, is dealing with non-folder object managers. If you want to manage the contents of these special object managers individually in your versioning system, you're running into some trouble, some parts of which seem insurmountable to me at the moment (from my admittedly limited knowledge ;-). I find myself agreeing with earlier posters (earlier as in 2001--Paul, for one, I think) who said that Zope itself might need to support full versioning, a la DeltaV or somesuch, itself. This makes sense to me, but I didn't find any historical documents on zope.org as to progress or as to reasons for abandonment of this approach. What's the deal? Would this be a huge effort, or is it definitively problematic for a reason I don't see, or is it stalled for some other reason? Thanks Gary

On 1 Aug 2002, Gary Poster wrote:

Hi Shane. I've been thinking about Zope versioning, and I also did a bit of list searches for past discussions on this general topic. One problem that seems pertinent to really any external-to-zope versioning system, including Subversion, is dealing with non-folder object managers. If you want to manage the contents of these special object managers individually in your versioning system, you're running into some trouble, some parts of which seem insurmountable to me at the moment (from my admittedly limited knowledge ;-).

It's really only a theoretical problem. To store the extra data about folderish objects, you can save the data in a hidden file called, for example, ".properties". The theoretical problem is that someone might give an object that name, since it's perfectly legal. In practice, you can just prevent people from creating Zope objects with a name that starts with a dot. 99% of the users won't mind at all, and those that do can use two dots instead. :-)

I find myself agreeing with earlier posters (earlier as in 2001--Paul, for one, I think) who said that Zope itself might need to support full versioning, a la DeltaV or somesuch, itself. This makes sense to me, but I didn't find any historical documents on zope.org as to progress or as to reasons for abandonment of this approach.

I've thought a lot about Zope object versioning, and I can't see any reason that Zope can't use an external repository. It would help a lot to work with a repository that is transactional and can version directories--but guess what, that's exactly what Subversion is good at!

What's the deal? Would this be a huge effort, or is it definitively problematic for a reason I don't see, or is it stalled for some other reason?

It's only stalled because we're all busy working on other cool stuff. :-) Shane

On 1 Aug 2002, Gary Poster wrote:

Given a hypothetical folder-like instance called "myFLI", we would presumably want, in CVS (or Subversion, or whatever) a folder named "myFLI" containing the children and a file named, to borrow your example, "myFLI.properties.zexp" that *only* contains the non-ObjectManager-children properties, whatever they are. But, as I understand it, when you pickle an object for storage as a zexp--in the way the ZCVSFolder does it, for instance--you are pickling the object *and* its (ObjectManager) children: not what we want.

This is the bigger stumbling block for me. Is this fixable? Overriding __getstate__ (I assume?) just for this seems fragile (can we guarantee the source of the ObjectManager children in the object, for instance? I don't think so). So that was my concern.

AdaptableStorage, which I just presented at OSCON 2002, faced the same problem, but I wrestled until I found a solution. A lot of it is based on ideas from ZODB. Here's my train of thought. I'm trying to refine my explanation each time I give it. 1) You don't have to store or version pickles. Instead, you can ask a series of "serializers" to convert your object to a simpler representation, then you can store or version the simpler format. 2) The simpler representation can encode inter-object references just like ZODB does. 3) As the serializers convert the object to a simpler representation, they can report to a "serialization tracker" which parts of the object have been recorded. 4) To finish idea 2, when the serializers record an inter-object reference, they are required to report the reference to the serialization tracker. This tells the controlling software to look at the referenced object, and if it is a new object not previously recorded, it is added to the stack of objects to record. This is similar to what ZODB.Connection.commit() does. 5) One kind of serializer is the "leftover pickle" serializer, which records the state of attributes and subobjects not recorded by other serializers. This serializer makes it safe to version any kind of object, with a caveat that the leftover data is stored in a binary format. 6) Serializers should also be deserializers, helping ensure that anything that can be serialized can also be deserialized. 7) The thing that I spent months (maybe even years) pondering was the simple format. I knew that object serializers could be very useful for storage anywhere, as well as versioning and merging, if we could just come up with a common format that would let us connect serializers and storage adapters together. I tried XML and DOM, but it was too cumbersome, and I tried several kinds of custom classes. Nothing really fit the bill. I finally gave up, but by giving up I found the solution. Sometimes having a deadline really helps! I started using sequences of records or, in most cases, tuples of tuples. I rediscovered what RDBMS experts have known all along--that data can be easily represented as sequences of schema-bound records. So now all the serializers in AdaptableStorage serialize to a sequence of records. I admit that the requirement is a stretch in some places, since I ended up writing code like "return ((value,),)", but there are great benefits to having a common, simple serialization format, and Python provides just the right ingredients. 8) Serializers are configurable components. Through the configuration mechanism, we will be able to add serializers to the component system in Zope 3. We will configure which serializers should apply to which kinds of objects. Many applications will use them, including object persistence mechanisms, version control adapters, merging tools, etc. It will open up a lot of possibilities, I think. I'm sure I left a few things out, so ask questions about the unclear parts. It's probably more info than you were expecting. ;-) Shane

Chris Withers wrote:

Shane Hathaway wrote:

<snip>

Wow! That was very very cool. I look forward to helping make this all a reality in Zope 3 ;-)

...

I'm sure I left a few things out, so ask questions about the unclear parts. It's probably more info than you were expecting. ;-)

Hmm, okay, the only problem I can think that still needs solving is this: Say you're serializing to a relational database. All well and good.

...until another app comes along and modifies your relational table (pretty common usecase, othwerwise people wouldn't care as much about the storage layer...)

How does the serializer/deserializer find out something has changed and propogate this back up, invalidating any cached object, etc?

I've been trying out a limited-duration cache strategy. The simplest approach is to clear the cache between transactions. Alternatively, you can clear the cache periodically. For a lot of applications this is adequate. Another approach, if you can afford it, is triggers. To do this, you have to know a lot about your database, since there is no standard way. But as I learned from AdaptableStorage, no matter what you do, you need to have conflict detection. Otherwise Zope will merrily delete a directory you just added externally, for example. This was a hard problem to solve until I stumbled on the idea of putting half of an MD5 sum in _p_serial instead of the date. (Only half because the current C code only allows 8 bytes in _p_serial. ;-) ) This works well enough to detect nearly all conflicts, even though it might not be the speediest solution. And as it turned out, as long as I had conflict detection, it didn't matter as much that the database didn't have the most recent state all the time. Zope detected conflict errors and retried the transactions, and the second time always worked. It was good. :-) Finally, here's a theoretical solution that I think would be ideal for a lot of people: if we could just ask the RDBMS for its current transaction ID, Zope could keep track of the last transaction ID it knew about, and clear the caches when another process made a change. This solution may yield the highest performance for Zope-centric applications. It would also remove the need for my MD5 hack. ;-) Shane

Gary Poster wrote:

I agree with Chris: *wow*!

[re the original explanation] ...

...

I'm sure I left a few things out, so ask questions about the unclear parts.

Everything was wonderfully clear, except that the actual mechanism to convert the nested tuples flexibly to RDBMS record sequences escaped me a bit. This probably gets in the realm of "look at the code" which is what, in fact, I will hope to do.

I created another set of components called "record storages". Their job is to load and store record sequences in a schema that matches that of a serializer. In the current AdaptableStorage code, most of them are still called a "table", the old name for "record storage", but when I started serializing to the filesystem I realized that "table" wasn't general enough! Most record storages store data in some kind of database, but other record storages act as a proxy for other record storages, converting data in one schema to another. This theoretically enables application developers and database administrators to work independently, bringing their work together through proxy record storages. Record storages are similar to SQLMethods, with the difference that record storages provide a way to both read and write a set of records. Conventionally, SQLMethods either read or write data, but not both (Though it's possible to write a SQLMethod that can read or write depending on the argument signature, that's not what they were designed to do. Instead, a record storage might consist of two or more SQL methods.) Some record storages might actually perform complex database queries that are not sensible to store. In that case, you can just raise a ReadOnlyError on an attempt to write. Shane

On Wed, 7 Aug 2002, Anthony Baxter wrote:

...

...

...

Shane Hathaway wrote I created another set of components called "record storages". Their job is to load and store record sequences in a schema that matches that of a serializer. In the current AdaptableStorage code, most of them are still called a "table", the old name for "record storage", but when I started serializing to the filesystem I realized that "table" wasn't general enough!

Plug: people might want to browse the online draft of Martin Fowler's "Patterns of Enterprise Application Architecture" for some nice ideas in this sort of framework. There's a whole section on Object / Relational Mapping Patterns. And unlike the GoF book, most of the patterns aren't just workarounds for C++ deficiencies </cheapshot>

http://martinfowler.com/isa/index.html

I've read this. It helped me get started.

...

Most record storages store data in some kind of database, but other record storages act as a proxy for other record storages, converting data in one schema to another. This theoretically enables application developers and database administrators to work independently, bringing their work together through proxy record storages.

Although this does mean you could end up in a really nasty nasty mess when trying to figure out responsibilities for the data changes - particularly if you're adding a new bit of data that needs to flow through the system from RDBMS to ZODB.

Most of the time there's an easy solution. See the other messages in this thread.

...

Record storages are similar to SQLMethods, with the difference that record storages provide a way to both read and write a set of records.

I think this sounds like a Data Mapper or a Table Data Gateway in the Fowler book.

I don't think any of Fowler's patterns correspond directly with a record storage. Data Mapper is the whole OR mapping, including all components. Table Data Gateway is too specific in the wrong direction. "Gateway" is the closest, but it's too general. And a record storage doesn't necessarily store data in a relational database. OTOH I could be wrong. :-) Shane

Chris Withers wrote:

Shane Hathaway wrote:

...

Finally, here's a theoretical solution that I think would be ideal for a lot of people: if we could just ask the RDBMS for its current transaction ID, Zope could keep track of the last transaction ID it knew about, and clear the caches when another process made a change. This solution may yield the highest performance for Zope-centric applications. It would also remove the need for my MD5 hack. ;-)

But would it work for non-transactional storages like file systems?

Nope, unless it's a journaled filesystem and we pulled out the journal log entry number. ;-) Shane

Chris Withers wrote:

Shane Hathaway wrote:

...

...

But would it work for non-transactional storages like file systems?

Nope, unless it's a journaled filesystem and we pulled out the journal log entry number. ;-)

So would it be good to have all three kinds of hooks then?

Yes. I've looked, but I can't find a way to ask Postgres for the transaction ID, though I did find LISTEN/NOTIFY which ought to provide good invalidation notification. Shane

Shane Hathaway writes:

It's really only a theoretical problem. To store the extra data about folderish objects, you can save the data in a hidden file called, for example, ".properties". The theoretical problem is that someone might give an object that name, since it's perfectly legal. In practice, you can just prevent people from creating Zope objects with a name that starts with a dot. 99% of the users won't mind at all, and those that do can use two dots instead. :-) The real problem (in my view) are ObjectManagers that are contained in Non-Object-Managers. Furthermore, splitting/folding ObjectManagers in the sense that you start/stop managing the individual content.

Dieter