[Gsoc] Respector: A Grok Introspector Replacement

[Uli Fouquet]

Hi there,

I followed the interesting discussion on the list. Especially
ZODB/registry/component browsers seem to be a hot thing currently ;-)

Because my last years' work on the Grok introspector is affected at
least indirectly by that development and because I am considering to
apply one more time for GSOC, I would like to put in some thoughts
concerning the Grok introspector, but also all kinds of 'runtime
scanners'.

As usual, please forgive the length!

Respector: A Grok Introspector Replacement
******************************************

The Grok introspector I developed during GSOC 2007 is still missing
some fundamental functionality and suffers from some fundamental
design flaws.

After (re)thinking about it and reading all the ideas mentioned in
this list (most notably: ideas collected and put in by Lennart and
Martin Aspeli), I came to the conclusion that a proper
reimplementation might be the best way to get a more powerfull tool
here. A reimplementation, that is based on a more general engine.

This might inlude improvements on the ZODB broken object model.


What does 'respector' mean here?
=================================

The word 'respector' is built from 'replacement', 'debugger' and
'introspector'.

It denotes both of the (currently) two distinct parts of 'runtime'
orientation for users in Grok, the ZODB browser as well as the
package/component browser. While the first one gets and displays
information about ZODB objects, the latter delivers (or: should
deliver) information about loaded packages, interfaces, directives
etc. It should be kept in mind, that ZODB and registry objects might
be only a part of the things users might want to examine in a runtime
environment.


Problems with current approach
==============================

The current approach of `docgrok`, the tool that does introspection
for Grok, suffers from some shortcomings:

- it is bound quite strictly to Grok, which means: it requires a full
  fledged Grok instance running to work.

- it makes strong assumptions about the environment it runs in, namely
  the existance of `apidoc` and other voluminous packages.

- it comes as a part of the grok core, which bloats the number of
  packages needed to run grok, even if the introspecting functionality
  is not wanted (for example in production environments).

- it might be considered a security risk, because it frankly passes all
  information to (authenticated) users without a possibility to
  restrict access. There is no possibility to create something like a 
  'developer user'.

- it does not provide an interface other than the Web-UI, nor does it
  provide entry points for, say, remote debugging without a browser.

- its API might be too complex. While it is possible to register
  arbitrary types of things to be displayed by `docgrok`, the way to
  do it might be plainer.

- it makes too little use of other tools already available (basically
  it wraps only `apidoc`).

Apparently `docgrok` also lacks the ability to display important data
like directives used in a view, available layers, default values
etc. This, however, is more a question of diligence, because in
principle all this is supported by the current approach. It would only
need a concrete implementation for concrete types of things.


Renewing the Grok admin app
===========================

Some of the shortcomings mentioned above can be tackled by simply
factoring the admin-UI out of the Grok core/trunk. Such, a user can
decide whether s/he needs this functionality or not on his/her own.

To cope with the deeper design flaws, I can think of a two-step
strategy:

1) factor out introspection stuff from the then standalone admin app
   into an own package or subpackage that can be used also without
   grok/grok.admin.

2) give it a new, more general, easy to use base.

All this should not be too complex, except the 'general base' stuff.


A more basic foundation for grabbing runtime information
--------------------------------------------------------

Reading all the proposals and ideas on the grok-dev and GSOC lists I
see, that there is a high demand for a better integrated
'runtime-scanner' which can be used in different environments (Plone,
Grok, etc.) and with different frontends (webbrowser, shell, wxpython,
etc.).

If such a tool exists, also Grok should of course make use of it.

From the experiences from last GSOC I might mention some things to pay
attention to when creating such a tool.

A basic runtime scanner

- should be extensible in the way, that it can handle different kinds
  of data sources such as the ZODB, registry etc. It should be
  possible to implement new scanners based on a well-designed API if
  appropriate, for example to add support for RDBMs one day.

  This means, that in the beginning a basic scanner would be needed,
  that makes nearly no assumptions about existing things in the
  environment, but is extensible to provide support for example for
  scanning the ZODB. The zodbscanner in this case would be a special
  case of the base scanner.

- should separate the tasks of information retrieval and information
  presentation. Nearly all tools existing in that area do that
  internally, namely `docgrok`, `apidoc` and `zodbbrowser`. But they
  all make assumptions about the used frontend (browser, wxpython-GUI)
  and offer only support for that kind of 'display'.

  A well designed runtime-scanner might be prepared for collaboration
  with unknown presentation 'plugins'. The separation of retrieval and
  presentation would ease the implementation of specialized frontends
  for Plone, Grok, etc.

- might be prepared for modification of objects. This means, it might
  provide a 'communication channel' to tell a frontend, whether a
  certain attribute/object/thing is modifiable and perform this
  modification on request.

  This way, the scanner would turn into a debugging tool.

All this is _much_ work, probably too much for a single GSOC, but at
least a plain but well thought base implementation should be possible.

When the base API exists, numerous scanner extensions/frontends could
be implemented independant from each other later on.


A renewed runtime scanner for Grok
==================================

Retrieving information
----------------------

Given that such a general base exists, Grok certainly needs a
specialized version that diggs for

- directives used (also default ones, not explicitly used)

- views/layers (also ones not currently used but available)

- other stuff

This would basically mean to implement a scanner, that extends the
base scanner, because every scanner can only look for things it is
aware of. This might make use of existing tools like `apidoc` to
avoid reinventing the reel.


Presenting information
----------------------

Once, the scanner works, an appropriate 'frontend' would be needed,
that naturally would be a set of views.

For Grok there might be much code of the current docgrok
implementation that can be reused for that.

I could also think of a wxpython-based remote tool for debugging,
which would require an appropriate 'frontend' as well.


Modify information
------------------

It would be nice to be able to change values of certain objects
on-the-fly.

A nice feature (not only for Grok) would be the ability to edit broken
objects. This would require a generic broken objects implementation as
proposed on http://wiki.zope.org/gsoc/SummerOfCode2008. I already
implemented support for deletion of broken objects in Grok.


Is all that necessary?
======================

No, it's not. For Grok we can archive the same functionality with the
existing docgrok stuff. It, however, makes sense from my point of
view, because a unified runtime scanner would distribute the effort for
keeping everything up-to-date on many shoulders and avoid
double/triple work. Why should the same ZODB scanning be performed by
four different packages, when in fact they all try to grab the same
information? Unfortunately, this is the current state.

Maybe something like a base runtime scanner already exists due to
efforts starting at the Plone summit. The apidoc package is usable
anyway and also the z3c.zodbbrowser might be of use, if it would
separate the information retrieval and presentation (currently wxpython
based, right?) even more accurate.

The Grok admin UI needs a redesign anyway to address all the issues
mentioned above. So why not go back to the roots?

Furthermore this addresses a topic, which crosscuts several planned
projects for upcoming GSOC. Why not try a collaborative effort? Is there
a better opportunity?

Kind regards,

-- 
Uli

-------------- next part --------------
A non-text attachment was scrubbed...
Name: not available
Type: application/pgp-signature
Size: 191 bytes
Desc: Dies ist ein digital signierter Nachrichtenteil
Url : http://mail.zope.org/pipermail/gsoc/attachments/20080327/b90c595b/attachment.bin