[Zope-dev] Re: Zope and Storm (SQL)
Christian Klinger
cklinger at novareto.de
Mon May 26 05:08:57 EDT 2008
Jürgen kartnaller schrieb:
> On Sat, May 24, 2008 at 2:24 PM, David Pratt <fairwinds at eastlink.ca
> <mailto:fairwinds at eastlink.ca>> wrote:
>
> Hi Jurgen. Thank you for this informative post. I am particularly
> interested in how this fits into the refactoring of legacy code. I
> appreciate your sharing your experiences. I also think everyone has
> been paying much attention to the insight Lovely has been sharing on
> scaling and efficient delivery of Zope over the past couple of years.
>
> As a means of mapping one or more backends without changing the
> logic or code with backend logic, schemas play an important role. I
> can see the benefit of providing plain SQL statements since they are
> clearly understood. The concern I have about not using schemas is
> the loss of integration potential for the different backends using a
> common pattern of mapping zope schema to xml, rdf, rdb, whatever ...
> In your opinion, is this abstraction simply costing too much,
> unnecessary, or a matter of application development and runtime speed.
>
>
> I'm not sure what you mean with schema/xml in context with SQL. You can
> still use your schema as you already do. I just wrote a CSV importer
> based on schema's together with formlib's Editform and AddForm.
> The only diference you have is, that it is not possible to use
> "FieldProperty" in Storm classes.
>
>
>
> For me, the crux of the rdb approach for legacy code is the
> container, location and traversal. You have been very generous with
> your examples. I am really hoping for a clearer idea of handling
> Container, OrderedContainer, and Location which is prevalent in
> legacy code. Overall, I can say that I quite the innovation here in
> getting to a 'leaner' concept of Zope.
>
>
> If you have your legacy code you have a clear definition what you need
> for your container. So it should be straight forward to implement
> IContainer and ILocated.
>
> Without going too deep into this here is some code which should be usable:
>
> class Container(Storm):
> interface.implements(IContainer)
> __storm_table__ = 'containers'
> id = Int(primary=True)
> content = ReferenceSet(id, 'Content.id')
> def __iter__(self):
> return self.content
> def __getitem__(self, name):
> item = self.content.find(Content.name == name).one()
> if item is None:
> raise KeyError
> return item
> self __setitem__(self, name, item):
> item.name <http://item.name> = name # add namechooser things here
> item.parent = self
> def __len__(self):
> return self.content.count()
>
> class Content(Storm):
> id = Int(primary=True)
> name = Unicode()
> parent = Reference(id, Container.id)
>
> Don't kill me if something is wrong here, this is an untested quick hack
> to demonstrate what's possible. Also the IContainer interface is not
> fully implemented.
>
>
>
> Regards,
> David
>
>
Hi,
i have released a zope-stromcontainer on pypi
http://pypi.python.org/pypi/nva.stormcontainer/0.2
Christian
>
> Jürgen kartnaller wrote:
>
> There seems to be some interest on the use of SQL databases with
> Zope.
>
> Lovelysystems is now using SQL databases as the primary storage
> for their applications. We use Zope and Postgres with Storm as ORM.
> The main reason for switching to SQL database were speed issues
> with queries.
>
> Here is a short summary of my thougt's and experiences while
> using Storm and Zope for about 3 Month now.
>
>
> RelStorage:
> Relstorage doesn't solve the speed problems. Doing queries with
> SQL is much faster than doing it with ZODB. If you work with a
> lot and with large BTrees you need to load them all into the
> memory of each Zope client. This has to be done with Relstorage too.
>
>
> Indexes:
> You don't need to implement catalog indexes, this is all done on
> the database side. When implementing and using your content
> types, at first you don't need to think about indexes, later you
> optimize the database without touching your python code.
>
> A speed example :
> We had to find similar users based on items a user has
> collected. Doing this with ZODB took minutes to calculate for
> users with a lot of items. We had to implement a lot of code to
> do the calculation asynchronously to not block the users request.
> Doing the same with SQL was possible with a single (of course
> complex) query within 300ms, no async things needed, just
> implement the query and optimize the indexes on the server,
> finished ! Relstorage will not help you here.
>
>
> Content implementation:
> While we are porting our existing ZODB based packages to SQL, we
> found that implementing them with Storm is as easy as using
> ZODB. We still can use the full power of Zope's component
> architecture. This is because Storm objects are extremely easy
> to implement. You can implement a storm object like a Persistent
> object, just derive from Storm instead of Persistent, add
> __storm_table__ and define the properties as Storm properties.
>
> For me a big mistake when switching from ZODB to SQL is trying
> to use the container pattern at any cost.
> A container is nothing but a 1:N relation and this is exactly
> what an SQL database provides : Relations
>
> class Content(Storm):
> id = Int(primary=True)
> content = ReferenceSet(id, 'Contained.somethingId')
> c = Content()
>
> Now you can
> - add data : c.content.add(content)
> - iterate : for a in c.content:
> - search : c.content.find(...)
> - sort : c.content.find().sort_by(...)
> - do anything a Storm ResultSet is providing
>
> But of course it is possible to put an adapter around the
> Content class which will provide IContainer.
>
>
> Annotation:
> Annotations are 1:1 relations, so it's as easy as the above.
> We use annotations like simple adapters to other tables.
>
> class ToBeAnnotated(Storm):
> interface.implements(ICanHaveData)
> id = Int(primary=True)
>
> Note that the "annotated" storm table is implemented as an adapter :
>
> class Data(Storm):
> interface.implements(IData)
> interface.adapts(ICanHaveData)
> id = Int(primary=True)
> __parent__ = Reference(id, ToBeAnnotated.id)
> def _init__(self, context):
> # a dummy to make the adapter happy
> pass
>
> We can now register "Data" as an adapter.
> We use a special adapter factory like zope.annotation.factory to
> autocreate adapted content.
>
> def contentAdapter(table, autocreate=True):
> # an adapter on content for content contained in other
> tables. Just like
> # the annotation adapter, an instance is created if
> autocreate is True.
> adapts = component.adaptedBy(table)
> if adapts is None:
> raise TypeError("Missing 'zope.component.adapts' on table")
> @component.adapter(list(adapts)[0])
> @interface.implementer(list(component.implementedBy(table))[0])
> def getAdapter(context):
> unsafeContext = removeSecurityProxy(context)
> obj = getStore().find(table, table.__parent__ ==
> unsafeContext).one()
> if obj is None and autocreate:
> obj = table(context)
> obj.__parent__ = context
> return obj
> return getAdapter
>
> Now you can define a factory for the adapter:
>
> dataFactory = contentAdapter(Data)
>
> And register "dataFactory" as an adapter.
>
>
> DublinCore:
> If you want to use the full DublinCore implementation from Zope
> you need to do a generic implementation.
> Usually only parts of the DublinCore interface is used.
> We usually implement IDCTimes and IDCDescriptiveProperties. All
> you need to do for this is :
>
> class DCStormContent(Storm):
> interface.implements(IDCTimes, IDCDescriptiveProperties)
> created = DateTime()
> modified = DateTime()
> title = Unicode()
> description = Unicode()
>
> That's it!
> You can now use IDCTimes and IDCDescriptiveProperties for your
> formlib form_fields.
>
> There are two way's to update "modified" :
> - write an event handler for ObjectModifiedEvent
> - do it on the database side with triggers
> I prefer using the event handler because the database trigger is
> doing the update only when writing to the database which can be
> to late.
>
>
> Schema's and Storm objects:
> We don't use schema's to create our Storm objects or the
> database table from it. Right now for us it is not worth the
> time to implement such a feature.
>
>
> Traversing and URL's:
> Usually our customers whant to have special URL's for their
> pages. In any way (ZODB or SQL) we need to implement special
> traverser's to provide the URL's. Usually we use z3c.traverser
> to do this.
> Because of the special URL's we also need to implement absolute
> URL adapters.
>
>
> Transaction handling:
> Storm already has a DataManager for zope's transaction package.
> All you need to do is to register a utility for each of the
> database you want to use.
>
>
> ZMI:
> Hmm, don't work out of the box. If really needed we build
> traversers for the ZMI.
>
>
> Data transparency:
> At any time you can use any database administration tool you
> like to directly view and/or manipulate you data in the database.
>
>
> Jürgen
>
>
> ------------------------------------------------------------------------
>
> _______________________________________________
> Zope-Dev maillist - Zope-Dev at zope.org <mailto:Zope-Dev at zope.org>
> http://mail.zope.org/mailman/listinfo/zope-dev
> ** No cross posts or HTML encoding! **
> (Related lists -
> http://mail.zope.org/mailman/listinfo/zope-announce
> http://mail.zope.org/mailman/listinfo/zope )
>
>
>
>
> ------------------------------------------------------------------------
>
> _______________________________________________
> Zope-Dev maillist - Zope-Dev at zope.org
> http://mail.zope.org/mailman/listinfo/zope-dev
> ** No cross posts or HTML encoding! **
> (Related lists -
> http://mail.zope.org/mailman/listinfo/zope-announce
> http://mail.zope.org/mailman/listinfo/zope )
More information about the Zope-Dev
mailing list