[Grok-dev] Re: Fetching data from a remote site

[Martijn Faassen]

Luciano Ramalho wrote:
> Given an ISBN or title/author info, Kirbi will look for book metadada
> in remote sites such as Amazon.com. I'd like your comments on an
> architecture for doing that in Grok.
> 
> I prefer not to have the precious few threads of my web apps waiting
> on network results, so last year I implemented this as follows using
> Horde:
> 
> - user submits form with ISBN
> - web app checks whether it already know that ISBN; if not, it saves
> it as "pending" and immediately returns a response
> - an external python script periodically asks the web app for pending
> ISBNs, fetches the metadata from the remote site, parses it and
> submits the data to the web app via XML-RPC

I think this is a reasonable approach.

It's probably possible to do this kind of thing integrated into the 
server instead of in a separate script. The benefit of doing this in an 
integrated way would be an easier way to install the application. I'm 
not sure how to accomplish this, but there are likely ways to do it in, 
say, Twisted, which has all kinds of support for asynchronous 
integration. It might be good to ask this question on zope3-user.

> On the other hand, the "pending" ISBNs could be a very nice example of
> using events in Zope 3: an event handler could go and fetch the
> metadata from the remote site. I'd like to hear from the Zope 3
> experts on this.

An event handler would still occupy a Zope thread, so events may not be 
very useful in this design, unless you expect more than one handler may 
(in the future) be listening for ISBN information requests and you need 
that decoupling.

Regards,

Martijn