[Zope-dev] Generic filtering of objects

Kent Polk kent@goathill.org
23 Nov 1999 17:53:46 GMT


On 22 Nov 1999 19:40:00 -0600, Robert Leftwich wrote:
>Robert Leftwich wrote:
>>
>>I have a requirement to be able to apply a user-definable filter to all the
>>views on the object database. Obviously this would be best applied at a low
>>a level as possible in order to minimise the impact on the application code
>>and to be made as generic as possible for maximum flexibility.
>>To save re-inventing the wheel, has anyone else had this requirement and if
>>so how did you implement it ?

I'm off in a little different area which is an extension of the
TinyTables concept where I'm wrapping information obtained from
several different locations into an object which is then unrolled
or 'SubMined' as Ty calls it. :^)  The information comes from the
filesystem, the Zope database, and hopefully soon, from other
database queries, etc.

Right now I roll almost all the results into ZRDB.Results objects
so I can more consistently filter them and carry the filter results
along as the user traverses through the data. This has a chance of
allowing me to 'join' my ZRDB.Results with other database results,
etc. I've tried to separate functionality to provide for more
flexibility with selectable filters that are applied to the SubMine
results object via __bobo_traverse__, in addition to DTML-based
rendering filters which are not 'carried around' as the user
traverses the objects. The SubMine filters are listed in order and
can be selectively removed from the 'filter list'. This concept
was derived from the 'Datamining' product posted earlier this year,
along with some initial work that Ty helped me with.

The problem is that the __bobo_traverse__ mechanism I'm using is
fraught with many problems but I can't see any way around them
right now.  This is somewhat applicable to the discussion on schemas
because these objects change on a regular basis (SubMines on every
request), but storing *some* of those changes as transactions in
the Zope database is simply not appropriate. 

Right now, it appears to me that Zope isn't well suited to dealing
with persistent, user-defined volatile filters, schemas, etc.  It
really wants to pass those things off to database-like queries.
For a while I thought that I ought to do just that with my datamining
stuff - just write a separate server that handles the things that
Zope doesn't deal very well with, but the problem is that Zope is
so darned close to what *is* needed that it'd be pretty stupid to
go off and write something else.  I'd rather see if there is a way
to convince Zope to differentiate between objects that need transacted
in the Zope database and objects that just need to be cached in
the Zope database, otherwise retaining the same behavior.

>Phillip J. Eby wrote :
>>
>> Ty Sarna and I have come up with a concept for this, which we have not yet
>> implemented, as we are still finalizing design details.  We
>> believe that it
>> could be made quite general.  The basic idea is that you have a
>>
>(Snip some excellent design/use case discussion [not to mention the odd head
>spin :-) ]...)
>
>I have been thinking about this on and off for the last week or so and I
>wanted to bounce around an idea or two.
>
>Given that one of the main criteria (for me, at least) is the ability to
>apply different filters on a per user basis, I am contemplating adding a
>filter keyword to the in and tree tags. There already exists a very
>specialised filter in both of these tags (the skip_unauthorized flag - which
>is a filter for unauthorised objects), so it should be easy to enhance this
>to support a more generic filter of the type proposed by Philip and Ty.
>
>The factors pushing me toward this solution are that filtering is generally
>a UI-driven requirement and I felt very uncomfortable about retrieving
>user-selected filters from the REQUEST variable at low levels in the code (I
>was experimenting with filtering in the objectId() method of ObjectManager).
>Not to mention that this solution only worked for ObjectManager associated
>data and fails when you have lists or dictionaries or external data access
>such as SQL.
>
>Anyone have any thoughts or comments ?

The SubMine filters that Ty and I worked on attempted to mimic SQL
query commands as it provided a familiar environment. If we are
really talking about general-purpose filters that can operate on
ZRDB.Results objects (in particular), I'd sure like to see what
could be done here.

BTW, Phillip kinda lost me. :^)  Ty - You know what I'm working
on.  Are we talking about anything remotely related? :^)  I'm ready
to ditch what I'm doing if there's a better way out there.

----

FWIW, MS and MIT are offering some funds for developing Open Source
educational resources that I think we stand a chance of taking
advantage of in this area... (http://swissnet.ai.mit.edu/projects/i-campus/)

"I-Campus is a collaborative initiative of MIT and Microsoft Research
to conduct research and create new technologies that will set the
pace for university education in the next five to ten years. The
project aims at fostering excellence in technology-enhanced education,
through producing materials that adhere to open standards, with
results and source code which can be widely published and disseminated.
I-Campus was launched in October 1999 as a five-year research
partnership between MIT and Microsoft, but we plan to engage
additional academic and industry partners as the project evolves."

I am in touch with some people who are considering a proposal that
some of these issues might address.  A few application-specific
things would need to be developed but I think we are headed down
similar paths and ought to consider working together. Any interested
parties?

Paul and the rest of you DC folks, there are a number of issues
with the I-Campus research that are darned close to what the AISD
proposal that we worked on eventually wanted to address. You might
want to take a look and see if there are partnership opportunities
available to pick up where we left off.

Kent