[ZODB-Dev] Exception while closing ZODB

[Ulla Theiss]

Hello Tim,

thank you for the answer!

1. I think, you have understood the problem correctly.

2. The second problem you mentioned, we have solved by using semaphores 
in the calling class.

3. We really need this functionality. Our application is for much more 
than a year in production. Until Zope 2.6.x (ZODB used the class 
LockFile from winlock) we didn't have this problem. But we had a problem 
with big  memory leaks. At the moment we
need reboot Zope 2.6.x about 5 times a day (!). The memory problem is 
solved in Zope 2.7.0. Therefore we really need Zope 2.7.0.

4. Because most of the time the application works fine and the problem 
only comes up with very particular temporarily behaviour, it needs a lot 
of time to find out the problem.

Many thanks again,
Ulla Theiss.

Tim Peters wrote:

>[Ulla Theiss]
>  
>
>>since Version Zope 2.7.0 we have a problem closing the ZODB under
>>Windows.
>>
>>Sometime the class LockFile (in the file .../ZODB/lock_file.py) throws
>>while unlinking the lockfile the Exception:
>>OSError: [Errno 13] Permission denied '....\\var\\...fs.lock'
>>
>>We open and close ZODB-databases in different threads.
>>    
>>
>
>That's dangerous; more below.
>
>  
>
>>A look at the sourcecode in LockFile.close() shows, that first
>>   unlock_file(self._fp)
>>   self._fp.close()
>>
>>is called. Which means other threads and processes are able to use the
>>locked file again. Afterwards the lock-file should be deleted:
>>
>>   os.unlink(self._path)
>>    
>>
>
>Let's look at all the code:
>
>class LockFile:
>    def __init__(self, path):
>        self._path = path
>        try:
>            self._fp = open(path, 'r+')
>        except IOError, e:
>            if e.errno <> errno.ENOENT: raise
>            self._fp = open(path, 'w+')
>        lock_file(self._fp)
>        print >> self._fp, os.getpid()
>        self._fp.flush()
>
>    def close(self):
>        if self._fp is not None:
>            unlock_file(self._fp)
>            self._fp.close()
>            os.unlink(self._path)
>            self._fp = None
>
>As best I understand you:
>
>    thread A created a LockFile, and later called its close() method
>
>    simultaneously, thread B calls LockFile.__init__ with the same path
>
>    thread A completes its unlock_file() and _fp_close() calls
>
>    thread B completes its open() call
>
>    thread A then barfs on its unlink() call (because you can't delete
>    an open file on Windows, and thread B has the file open now)
>
>Is that right?  If so, that's not the only way it can fail.  For example,
>
>    before thread A does unlock_file(), thread B completes open() and
>    tries to do lock_file()
>
>    thread B will barf then (Windows file locks aren't "advisory" -- you
>    can't lock a file that's already locked on Windows)
>
>Example:
>
>  
>
>>>>import ZODB
>>>>from ZODB import lock_file
>>>>f = open('whatever', 'w')
>>>>lock_file.lock_file(f)
>>>>g = open('whatever', 'r')
>>>>lock_file.lock_file(g)  # attempt to lock the same physical file twice
>>>>        
>>>>
>Traceback (most recent call last):
>  File "<stdin>", line 1, in ?
>  File "ZODB\lock_file.py", line 33, in lock_file
>    _LockFile(file.fileno(), 0, 0, 1, 0)
>winlock.error: 33
>  
>
>
>  
>
>>Therefore, if another thread already uses the lock-file, the first thread
>>is unable to delete it and the 'Permission denied' exception is thrown.
>>    
>>
>
>If I'm understanding you, that's one failure mode, but not the only failure
>mode.
>
>  
>
>>As a workaround we put the os.unlink - statement in a try-except-block.
>>
>>def close(self):
>>    if self._fp is not None:
>>        unlock_file(self._fp)
>>        self._fp.close()
>>        try:
>>            os.unlink(self._path)
>>            self._fp = None
>>        except:
>>            pass
>>    
>>
>
>That can't stop the other failure mode above.
> 
>  
>
>>How to correct the error best?
>>    
>>
>
>Trying to open a file in one thread, while simultaneously closing it another
>thread, is the deeper problem here, and seems inherently ill-defined.  Do
>you really need to do that?  Since yours is the first report of this, it
>can't be a popular vice <wink>.
>
>
>  
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://mail.zope.org/pipermail/zodb-dev/attachments/20040707/d50cc3a2/attachment.htm