[Grok-dev] Kirbi progress report

[Luciano Ramalho]

Today I got a prototype of Kirbifetch, the metadata collection engine
for Kirbi working.

The code is in my sandbox:

http://svn.zope.org/Sandbox/luciano/kirbi/kirbifetch/src/kirbifetch/

Like I said, it's a prototype, so there's lots of silver tape putting
it together.

Kirbifetch has a long-running console script (fetch.py) which uses
it's own Twisted reactor to:

- poll a Kirbi method (dumpIncomplete) via XML-RPC to get a list of
ISBNs for books with incomplete metadata

- request the bibliographic records and book convers from remote
sources via HTTP

- parse the XML response from the remote sources

- call a Kirbi method (updateBooks) via XML-RPC to update Kirbi Book
instances with the retrieved metadata

- save the book covers in the proper place inside the Grok standard
static directory

In order to integrate with Kirbifetch, Kirbi also got some new features:

- the book catalog now has a view (kirbi/pac/incomplete) which lists
incomplete book records and those which are pending after being handed
over to Kirbifetch

- there is a new form, addbooks, linked from the main catalog view,
which alows you to paste a list of hundreds ISBNs for processing;
Kirbi then imediately creates the incomplete Book instances, and a
subscriber puts their ISBNs in an incomplete list where kirbifetch
will get them

Programming and testing these asynchronous behaviours was challenging.
BTW, I've done all testing of the asynchronous parts manually so far.
Still have to figure out how to automate such tests. Twisted has
testing infrastructure for that, but I haven't studied it yet.

Martijn told me how to integrate kirbifetch into the kirbi buildout,
but I haven't done that yet.

Tomorrow I've got a full day at the University (my classes have
restarted) but wednesday I am free and should have more news.

Regards,

Luciano