[ZODB-Dev] FileStorage can't handle > 2GB with Python 2.2.2?

[Tim Peters]

[Greg Czajkowski]
> Hi all, Ive switched over to Python 2.2.2 with ZODB
> 3.1.1, but Im still having problems with the server to
> stop responding with the following error. Causing all
> clients to hang as well.
> I have to kill the server and fsrecover, loose data,
> and restart. Making FileStorage useless for >2GB
> databases.
>
> Any help would be appreciated.

You're running on Windows, and the Microsoft _chsize() function doesn't work
with large files.  I don't know whether there's a way to avoid calls to
file.truncate() from ZODB, but they simply won't work under the 2.2.2 +
Windows + large file combo.

In the (not yet released) Python 2.3, the Python file truncation code on
Windows was rewritten to avoid using Microsoft's C library functions, in a
long-winded way that calls the native Win32 SetEndOfFile() instead.
Truncation of large files isn't a problem then.

So, you'll need to move off of Windows, avoid large files, wait for Python
2.3, talk someone into backporting the 2.3 file.truncate rewrite to the (not
yet released) Python 2.2.3, or hope (this seems least likely to me, alas)
someone knows how to trick ZODB into never calling truncate() on large
files.

> Traceback (most recent call last):
>   File
> "C:\Python22\Lib\site-packages\ThreadedAsync\LoopCallback.py",
> line 114, in poll
>     obj.handle_read_event()
>   File "C:\Python22\lib\asyncore.py", line 396, in
> handle_read_event
>     self.handle_read()
>   File
> "C:\Python22\Lib\site-packages\ZEO\zrpc\smac.py", line
> 94, in handle_read
>     d = self.recv(8192)
>   File "C:\Python22\lib\asyncore.py", line 359, in
> recv
>     self.handle_close()
>   File
> "C:\Python22\Lib\site-packages\ZEO\zrpc\smac.py", line
> 202, in handle_close
>     self.close()
>   File
> "C:\Python22\Lib\site-packages\ZEO\zrpc\connection.py",
> line 462, in close
>     self.obj.notifyDisconnected()
>   File
> "C:\Python22\Lib\site-packages\ZEO\StorageServer.py",
> line 253, in notifyDisconnected
>     self.abort()
>   File
> "C:\Python22\Lib\site-packages\ZEO\StorageServer.py",
> line 486, in abort
>     strategy.abort(self)
>   File
> "C:\Python22\Lib\site-packages\ZEO\StorageServer.py",
> line 694, in abort
>     self.tpc_abort()
>   File
> "C:\Python22\Lib\site-packages\ZEO\StorageServer.py",
> line 640, in tpc_abort
>     self.storage.tpc_abort(self.txn)
>   File
> "C:\Python22\Lib\site-packages\ZODB\BaseStorage.py",
> line 123, in tpc_abort
>     self._abort()
>   File
> "C:\Python22\Lib\site-packages\ZODB\FileStorage.py",
> line 1031, in _abort
>     self._file.truncate(self._pos)
> OverflowError: the new size is too long for _chsize
> (it is limited to 32-bit values)