[ZODB-Dev] Webkit Threading and ZODB 3.3a2: problems on Windows

[Matt Feifarek]

Hello.

We've run into an interesting problem using ZODB 3.3a2 and Webware 
Webkit's threaded application server (http://webware.sourceforge.net) on 
Windows (but not Linux).

The symptom that alerted us to the problem was that object attributes 
sometimes -- but inconsistently -- failed to be persisted. (No, this 
problem has nothing to do with mutables. Yes, we're sure.)

First (for the ZODB people), a quick summary of how the Webkit 
application server works: the server maintains a pool of threads to 
service URI requests. Each request is assigned  an available, running 
thread. The app server does one of two things:

1) instantiates a new servlet instance and passes the request to be serviced

    - or -

2) it finds an already-running instance of the relevant servlet and 
passes it
   the request to be serviced

In either case, these threads eventually return a response string to the 
http client.

The crucial subtlety is that a thread does NOT destroy/garbage-collect 
its servlet instance after servicing the request. Rather, the servlet 
instance is "put to sleep" and kept around in case another request is 
for the same servlet. If the application is taking requests for many 
different servlets, the servlet instances do slip in and out of the pool 
as different requests come in, but there is no guarantee that a servlet 
instance will be brand new with every request.

The crux: it appears that Webkit's threading model exposes a ZODB 
problem when a Webkit servlet transacts with a ZODB under Windows.

Changes to persistent objects are made and committed within servlet 
code. If you watch the _p_changed attribute before and after running 
get_transaction().commit(), we assume it should always be 1 before the 
commit() and 0 after the commit. Under our Windows test case, sometimes 
_p_changed is 1 following the commit() suggesting that get_transaction() 
somehow isn't getting the right transaction, and therefore nothing is 
actually being committed after all. The exact same Webkit code does not 
exhibit this problem running under Linux.

We have identified to workarounds on Windows:

1) limit the Webkit app server to one thread; this solves the problem 
but neatly renders the app server useless for production purposes

2) add a seemingly superfluous get_transaction().abort() call just 
before the servlet is put to sleep (and before the database connection 
is closed) by the app server; apparently, if you make this call (even 
when you've made no changes) a side effect is precluding whatever 
circumstances are leading to the real problem

Here are some more details on what we've tested:

Windows:
- python 2.3.3
- ZODB 3.3a2 using standard FileStorage with BTrees
- NOT using ZEO
- Webware 0.8

Linux:
- python 2.3.1
- ZODB 3.3a2 using standard FileStorage with BTrees
- NOT using ZEO
- Webware 0.8

We also noticed that the object returned by get_transaction() has an 
_id. If we watch this value as servlets transact with ZODB, we've 
noticed that the ids are frequently re-used. Perhaps this is relevant?

If anyone would like more information on tracking down this bug, we'd 
love to help. We're afraid that we're probably near the limits of our 
expertise, but can help with debug print statement output or whatever.

Thanks!