[Zope-CMF] Project-style workflows, DCWorkflow, SQL, etc.

[Kent Polk]

On 1 Oct 2001 15:05:01 -0500, Phillip J. Eby wrote:
> Kent: I hope you don't mind me cc:ing this to zope-cmf, I think many of my 
> comments are applicable there.  Thanks.

(Ack! :^)

> Neither do I, although I have some ideas.  Ty and I came up with a concept 
> once that was based on the notion of "waiters for".  That is, the 
> fundamental notion was dependency.  A task could be "waiting for" a person 
> or persons to do it.  It might be "waiting for" some set of 
> subtasks.  Things "waiting for" persons or groups of persons show up in 
> their worklists.  Things "waiting for" automated processing of some kind go 
> in a processing queue.  Someone who makes a request can also be said to be 
> "waiting for" the request, although more often than not, they will actually 
> be making a request in the context of some other task, which will then be 
> "waiting for" the requested task.

That generally corresponds to the request/accept mechanism that we
needs, and probably deals better with unanticipated workflow action
needs.

> Waited-for items send messages to their waiters, which may cause the waiter 
> to stop waiting, or perform other actions (such as consider itself 
> completed).  In the case of a task performed by a person, they "send a 
> message" by using web forms/methods on the task.  In the case of 
> task-to-task dependencies, the "Observer" protocol could be used, with the 
> waiter subscribing to the waited-for item's events.

As well as the type of thing I'm doing now with xmlrpc clients
which I use to wrap other software applications that the workflow
needs.

> In this model, a "request" is just another waiter/waitee, like a task.  The 
> main difference is that a request has a generic workflow - that of 
> responding to a request to do something.  The receiver of a request (who is 
> what the request is "waiting for") then creates dependencies from the 
> request to the tasks that are needed to fulfill the request.  In a more 
> advanced system, specialized request types could make creation of the 
> standard tasks automatic - although in that case it may make sense to 
> simply customize the workflow and make it purely a task rather than a 
> request.  In general, it has seemed to us, however, that there is value in 
> distinguishing the tracking of a request to do something, from the actual 
> doing of it.

Herein lie my conceptual problems with trying to use a CMF item to
broker workflow actions on the behalf of its 'subworkflows', etc.
I'm convinced that it shouldn't be managed by an item, but by a
workflow management task of some sort, but how would you identify/manage
a wait task in the CMF?

> Anyway, this model is very simple and requires only two basic interfaces; 
> the vast majority of objects in the system will implement both.  People, 
> tasks, requests, etc. are all both waiters/waitees, requesters/responders, 
> dependents/dependencies, or whatever you want to call them.  Most of the 
> objects can be considered state-machines which go through a set of 
> transitions ala DCWorkflow.  Events on waited-for objects can trigger 
> transitions in their waiters, which may use guard conditions that check 
> whether there are any objects still being waited-for.

I think it would be important for the current CMF items to
transparently participate in this sort of workflow when they have
been created by the workflow. Basically, their own workflow interacts
with the project workflow task, as well as the rest of the CMF.

Somehow...

> Although this model is simple, implementations can be arbitrarily 
> complex.  For example, you may want people or "worker" objects to be able 
> to do things like send e-mail notifications (or page/instant-message them) 
> when something is added to the person object's list of objects "waiting 
> for" them.
> 
> DCWorkflow is a bit primitive in some of these areas, but there's nothing 
> in the above model that's inherently beyond its capabilities.  Your 
> principal difficulties would be in making it clear how these things 
> interact with one another, and perhaps creating some libraries or base 
> classes for dependency management, so that the guards, etc. in your 
> DCWorkflows are not too complicated to write.

The complexity at this point can be a serious drawback. The system
needs to be fairly easily comprehendable, and the options I see
for a complex situation worry me.

> Also, one other potential complication would be in the area of users; 
> you'll likely need to extend the notion of a Member or create some kind of 
> "worklist" object that would act as a proxy for the member and implement 
> the "waitee" interface.

Yes, that corresponds the to workflow status query that I described
earlier. I was hoping that we wouldn't need to dig soo deep into
users for this...

...

> These need not be global decisions.  All that the "waiter" concept requires 
> is that a waiter be able to find and link to an existing "waitee" or create 
> a new one.  In the vast majority of use cases, however, one creates a new 
> waitee, or, if the waitee is a person/group, link to an existing one.  It 

Sounds right on track to me.

> tends to be a fairly infrequent action where one needs to create a 
> dependency to an already existing task, so such use cases can usually be 
> addressed on a case by case basis.  For example, in the case of an ISP, 

We have several feedback loops where data is determined to be of
insufficient quality, etc. and it must be reexamined. Some of these
should be arranged for by a workflow filter, but some will be done
by humans. Will have to take a closer look at that.

> they may receive many requests from customers regarding a network outage, 
> and so they might need to make all these requests "wait for" a single NOC 
> ticket, in which case their requests need to have the ability to find and 
> link to NOC tickets.  But then you know in your specific design where to 
> look for NOC tickets.  Make sense?

I think so. Most of our workflow is composed of serial and parallel
actions that are pretty fixed, but we do have to determine a way
to handle transients.

> Thus, at a framework level, it suffices to have a registry (such as the 
> existing Zope/CMF type registries) for selecting types and creating new 
> waiters, and a way to look up users or worklists (such as some extension to 
> the Membership machinery), and perhaps a standard way to find 
> "Organization" or "Team" objects (for department, role, or skill-based 
> worklists, as opposed to specific individuals).

This issue of group or team activities keeps coming up. I wish we
could find a solution to it, and not one that requires that CMF
items be structured in a storage heirarchy to accomplish it.

> Another possibility is ZPatterns, but I wouldn't recommend it, due to its 
> lack of a future.  Steve Alexander has been making heroic efforts to keep 
> it going and even move it forward, but I personally think that the 
> NewReligion will make it ultimately obsolete.

Maybe we can push this aspect off a little further into the future.

> Regardless of your workflow implementation or even specific DB software, I 
> do think that the WarpCORE design patterns will make your SQL design 
> easier; specifically, it's geared to polymorphism, hierarchies, and 
> references to external objects (e.g. data stored in those "evil scientific 
> software" apps you mentioned previously).

How is WarpCore doing? Been too busy to keep up with it.

>>(I don't have anyone around here to discuss this with that understands
>>anywhere as much as I do and I'm woefully inadequate)

I'm trying to talk my boss into hiring some competent help to work
on these issues with me, as well as a bunch of other tasks related
to publishing genomic/phenomic databased data that we are developing.
We have several hundred compute farm processors that we have busy
running data-cleaning models on, so are on the edge of really
beginning to crank out a lot of high-quality data. NEED HELP!
getting intranet software up and running to support/publish it.
It'd sure be nice to pick up a highly-motivated, versatile
Python/Zope/CMF/SQL programmer for this stuff. Maybe we'll be able
to make some progress...