[Zope] Zope 2.7.3 + Apelib 1.0 - Intermittent data loss in acl_users obj

[Chris Kratz]

Hello all,

We've been using Apelib with Zope 2.7.3 for several weeks on a production 
site.  We've been having problems with password changes disapearing 
intermittently from one day to the next, and whole user accounts being lost 
periodically.  Up until today, we thought it was a bug in our own code.

After being able to reproduce the problem outside of our code (using the zmi), 
we started looking at Zope and Apelib.  We have come to the conclusion that a 
standard acl_users folder is not always correctly persisted when using 
apelib.  Whether or not we are right, the following is how we understand 
things to be working at this point as well as an ugly hack that seems to fix 
the problem.

We are using the file system persistence method of apelib for our test bed.  
Using the file system persistence and grep, it is a very simple procedure to 
add a user, change a password through the zope zmi and verify that the 
changes to acl_users were persisted by inspecting the file on the file 
system.  There is no pattern, but almost always after several changes to a 
password or adding several users, the changes stopped being persisted to the 
file.  Stopping and starting zope at any point from then on loses those user 
accounts or password changes that were done after the persistence stopped 
happening.  We verified that the objects are correctly changed in memory, 
because subsequent use returns the modified data.  Also, updating the 
properties on an acl_users folder (going to Properties tab and clicking Save 
changes) causes all changes to be written out including password changes and 
user account additions and removals.

We are no experts on this code, but it appears that the serializing code in 
apelib only is inspecting the _p_changed flag on the user folder itself and 
not on the individual user objects persisted within.  In a standard zodb, it 
appears that every user object is persisted separately.  So, a change to the 
user object only marks that user object as changed, not acl_users.  But in 
apelib, the acl_users folder is persisted as a whole with all user objects as 
the contents.  With a standard ZODB, any updates to user objects are 
persisted at the user object level.  The same thing appears to be happening 
with apelib except that it appears that apelib when persisting objects only 
examines the acl_users folder and none of the user objects for changes.

It may be a red herring, but by adding self._p_changed=1 to _doAddUser, 
_doChangeUser and _doDelUsers in class UserFolder, we were no longer able to 
break it (ie changes were always written to file). Since apelib treats a user 
folder and all it's user objects as a single entity on the file system, this 
seems to address the problem.  What's even more troubling is mixing this issue 
with multiple threads which caused all kinds of authentication errors when 
creating new accounts.

We are wondering if anyone else has run into a simliar issue, and/or whether 
there is a better way to solve this then to modify User.py.   It seems like 
the better approach would be to modify apelib, but we haven't been 
successfull up till now.

-Chris