[Zope] Re: Ape backwards

[azazel]

Il giorno ven, 11-02-2005 alle 14:31 +0100, martin f krafft ha scritto:

> I want records created in the database outside of Zope to show up as objects in Zope
automatically. The same applies to deletions.
>
> I want objects created through Zope to create appropriate records in the RDB. The same
applies to deletions.
>
> I want to be able to edit objects and have the changes percolate to the database. I
also want changes in the database to show up in the objects.
>
> Is this what Ape is all about?
>

Yes, even if the default SQL storage layout doesn't help you on this, it's schema is
minimalistic and take cares of storing objects in a tree-oriented way.

The main part of the sql schema are:


- the "classification" table used by the Classification gateway to store classification
data, wich maps objoid (the one you find on the attr _p_oid of every Persistent obj) to
a class name and a mapper name which knows how to serialize and deserialize that class;

- oid_seq is the generator of all the objoids;

- the "folder_items" table contains parent/child tree relationship and child id which
reflect the one you see via ZMI;

- the "object_data" table contains the value ofd the "main data
attribute" for each obj, like the text for pt and dtml, etc.;

- the "properies" table contains main properties of a zope object common to all
SimpleItem descendants, like id and title. Depending on the mapper used, a class
dedicated table is created to store properties found on object of that class (if you
create a folder, you will get a "folder_properties"). The mapping between class names
and such tables is maintained in the table "property_tables".

- Last but not least, the "remainder" table contains base64 encoded pickles of all the
members of an object not serialized by "previous" serializers. Remainder serializer has
the lowest priority, so it runs at the very end of the serialization /deserialization
process;

There are other tables used to store user and security stuff.

That said , normal serialization involves some operations:

1) The oid-generator is queried, unfortunately without so much
"context", so you can't generate table dedicated oids wich matches classic RDB surrogate
keys;

2) The classifier classifies the obj and stores this information using the
classification gateway;

3) The composite mapper takes care of serializing the obj, delegating the operation to
serializers depending their priority;

4) finally a blind serialized is called to check if all has been
serialized;

Obviously, the deserialization process happens reversed. What is not  so obvious (even
if it should, really) is that the main parts of the
deserialization process are at storage level:

- the creation of a stub instance (empty) of the class the
classification has found;
- the merge operation of that stub instance with the deserialized
members;

This means that is really possible to control the process of instance creation, and that
is possible to create an instance having starting only by classic RDB data and
developing deserializers that take care of filling missing parts, remembering that
deserialized members really overwrite preexisting (class) members at instance
construction.

> > I thought you were asking if you could make a relational table visible as Zope
objects using APE, which it can theoretically do, and which I'm VERY interested in
;-)
>
> Yes, this is what I am asking, I think. Are there any examples of this? Ape looks
complicated and before diving into the theory, I'd rather explore it in action first.

> So I want a way to associate a folderish Archetype (essentially just a folderish CMF
type) with a RDB table so that an instance thereof obtains and manages some of its
associated data from/in the RDB. At the same time, the instance should also feel like
a regular ZODB object, which I can copy/paste, and in which I can store objects for
management by the ObjectManager.
>
> Will Ape help me here?

Yes!

I had your same idea so i setup a test sandbox with Archetypes 1.3 head  Ape, PostgreSQL
and ArchGenXML. I've setup an ape mount point under a plone site declaring (in
zope.conf) that i wanted to use a
Products.CMFPlone.PloneFolder.PloneFolder instance to be used for the mount point, so i
can use it normally like a plone folder (unfortunately i wasn't able to instantiate
correctly an Archetypes based folder). Then i built ArchGenXML's sample project
outline_od.xmi  running AGXML with "--ape" on command line to have default AT Ape
mappers (wich then make use of stuff in Archetypes.Apexxx.py module) built automatically
and i've installed the product in the portal.

Default AT serializers use a dedicated table named like the class to store AT fields
values. Despite of what the Archetypes.Apexxx module says on the top, i found that the
references are working correctly.

After that i wanted to seed if thougths on the deserialization process where correct, so
instead of creating and Outline instance via zmi, i used normal "psql" command line tool
(for postgres).

I filled values for all the main tables: classification, folder_items, object_data,
properties and outline (all using oids generated by the oid_seq). Leaving ther other
untouched

I then have flushed the storage cache to keep it in synk with the sql backend

After that i browsed to the folder where i attached my sql built
instances (in folder_items) and voilà, the instances are there!

The first time i edited them, i got the remaining data (reminder,
security, ecc) filled on the RDB.

This is just a test done with the default sql storage configuration, but it proofs that
with a custom developed ape storage one can reach a real tight integration between ZODB
and relational databases.
However i still need how to answer to some of the problems that arised:

- The cataloging machinery nor the uid stuff are correctly initialized. They gets
initialized the first time  i edited the instances. I still have to find a good way to
trigger such inits...

- On my SQL data i use one id sequence per table where i use surrogate keys. I need to
find a good schema layout to merge such a classic layout with a more zope storage
oriented one using maybe oids like
"<table_name>id". It would be useful also to better control AT UID creation and
attribution so that they meaningful at SQL level. This will permit the automatic
"rendering" of table relations using special AT reference fields;

The use of unrestricted Python as procedure language on RDB can help on the first issue,
triggering XML-RPC or simply http calls on object creation...


I hope this helps, i'm sorry for my bad english.
I would be useful share our thougths in an appropriate area of some site...:)

bye,
Alberto