[Zope-PTK] Best practices for Zope development, staging, and production

[Spicklemire, Jerry]

Hi Best Practitioners,

Won't it be great when an acknowledged Zope Development Model is available,
verified, and easily to follow with a fully documented "Coping with Zope"
cookbook? As it is, we're all making this up as we go. Isn't Open Source
exciting!?

For thoughts on the details of staging in Zope, see:
 
http://www.zope.org/Members/tseaver/Projects/HighlyAvailableZope/DevTestProd

The development of Mountable Databases, and Mountable Storage provide part
of the "The Good News" side of the story.

 See: http://www.zope.org/Members/anthony/software/MountedFileStorage/

 See: http://www.zope.org//Members/jim/ZODB/MountedDatabases

Near as I can tell though, no amount of technology will be adequate to stand
in for a good understanding of a solid Zope Development Model. Now, having
stated that premise, let me insert my standard disclaimer: 

	Good Judgement is a byproduct of Experience . . . 
	 . . . which is a byproduct of Poor Judgement

In other words, we're mostly on our own here, but thankfully many Zopetistas
such as Joachim have set the tone by sharing solutions they have come up
with at iuveno.de, and elsewhere.

My more recent observations follow:

We've all read and no doubt accept that the key to effective Web Development
is to Separate Object Layers, such as Logic, Content, and Presentation. 

The first critical task then, is to establish precise definitions of the
terms; Logic, Content, and Presentation. Conceptually, this is a piece of
cake. In practise though, what looks like Content to a Web Designer, looks
like Presentation to a Coder. Likewise, what looks like Logic to a Web
Designer may also look like Presentation, to a Coder.

Here's an mini-example, taken from a post on this list, in the form of a
DTML "Method"*:

"  <tr>
   <td>
    <a href="&dtml.url_quote-objURL;"><img src="&dtml-objIcon;"></a>
   </td>
   <td>
    <a href="&dtml.url_quote-objURL;">&dtml-title;</a>
   </td>
   <td>
     <dtml-var meta_type>
   </td>
   <td>
     <dtml-var Date>
   </td>
  </tr>"

* Any reasonable person might assume that a DTML Method represents only
"Logic", since the term Method is OO Speak for "a way a _program_element_
does something". Looking at the example above shows that, in Zope (and other
Web scripting tools) the "something" that gets done is not just calculations
(Logic), or text extraction (Logic), but also HTML generation
(Presentation), such as table creation.

The point is, any Web Designer will take one look at :

	<a href="&dtml.url_quote-objURL;"><img src="&dtml-objIcon;"></a>

and know with abolute certainty that, "There is Code in There" (and
instantly recognize this as distasteful, and to be avoided), whereas:

     <dtml-var meta_type>

may be more tolerable, since it is simpler, though no less incomprehensible.

So, at least within your team, for use on your projects, you need to get
your definitions clear at the earliest possible point. The better you are at
nailing this, and at the very least understanding the (hopefully rare)
exceptions, the less you'll need to re-build later when you realize you
didn't get it exactly right. If it turns out you actually do get it right,
and never have to re-do anything, PLEASE, tell us how you did it!

Before you get too comfortable with that tiny bit of wisdom, realize this is
just an example, selected to clarify a minor detail. The ultimate conclusion
drawn from this narrow point of view, is that diligent separation of Logic,
Content, and Presentation implies breaking out tiny little snippets of DTML
as single lines or even named instances of standard Zope objects, so that
they can be pointed to from inside DTML Documents as :

	<dtml-var snippet_name>

This practise would no doubt result in a site that is insanely complex, with
massive folders holding large numbers of mini-objects, and likely drive
performance and resource requirements both in precisely the wrong direction.

One more dead end. So, what should we be doing?

For myself, I can't claim to know the best answer, but can confirm that it
takes an inordinate amount of mucking around trying stuff, just to recognise
the questions.

The XHTML Template Project is aimed squarely at this topic, so keep yours
eyes on HiperLogica's HiperDOM as it continues to develop.

 See: http://www.zope.org/Members/lalo/HiperDom

Some other Templating solutions are represented by:

	http://www.zope.org/Members/Hoekstra/ZECF

and for additional thoughts on why Templating matters, See:

 http://www.zope.org//Wikis/DevSite/Proposals/ZopeTemplateObjects

 
http://www.zope.org//Wikis/DevSite/Proposals/DocumentsAndTemplatesDiscussion

 http://lists.zope.org/pipermail/zope-ptk/2000-September/001668.html

Let's jump from the micro example we just looked at, to consider the other
end of the spectrum, "the Big Picture".

This is where the Mounted Database / Storage pieces start to look
attractive. If we arrive at clear definitions of what Logic, Content, and
Presentation are, we will be that much closer to being able to segment each
into discrete designated storages. The reason this is appealing is the sheer
simplicity of updating one major component of a site , e.g. the Logic,
independently from the Content, or Presentation. 

If all Logic is held in a separate storage, completely replacing that
storage with an updated version is equivalent to updating all Logic used by
the site. Of course the same would apply to Presentation storage, and
Content storage. I'm not saying I have found the perfect way to do this, but
that the concept enticing, and is about as crystaline as it gets.

Somewhere in between the two extremes illustrated in the examples above, is
Enhanced Versioning, as seen in the ZCVSMixin product.

	See: http://www.zope.org/Members/sspickle/ZCVSMixin

ZCVSMixin simplifies updating Zope Objects, such as an entire folder,
including all the contained objects (except subfolders, for now). But Wait!
You also get all the handy differencing, timestamping, etc. of CVS, and an
Enhanced Zope interface that makes it much easier to use than you might
imagine.

As for storing Content objects in a RDBM, this idea is far from the worst
case, but as mentioned in the top link above, you seem to lose the benefit
of the Object Friendly nature of ZODB. In other words, text blobs stuffed
into Oracle fields aren't really very smart about their own nature.
Meanwhile, stuffing the same long character strings into ZDB certainly
promises to allow easy attachment of all the right properties and
intelligence that one would hope for. A first pass at this little snag is
embodied in the concept of "Pluggable Brains", as applied to text blobs. 

Some of the thorny gotchas lurking along this path are also part of the
reason the ZPatterns Project was initiated.

 See: http://www.zope.org/Members/pje/ZPatterns

One of the mind bending features of ZPatterns is enabling storage of
Properties as Objects in their own right, all but independent of the objects
they are properties of. Think about that for a second or two. Odd as it may
seem, this is a valuable contribution to the process of reaching a sound
Object Pattern for Zope Development. As for complete and deep explanations
of how all this fits together into a comprehensive Zope Development Model,
well, I'll let you know when I get there. Just remember to throw down a
rope, if you get there first!

BTW, none of the products or concepts mentionedd here are mutually
exclusive, and the most feasible solution may well involve a twisted amalgam
of several existing tools.

There must be many other folks who are up against this, and hopefully we can
all continue to share ideas, successes, and even warn each other of the dead
ends we've had to back out of. Sometimes the only way to measure progress is
to keep count of the number of ways you've found that absolutely do not
work!

Later,
Jerry S.