[ZODB-Dev] Importing large .zexp fails with ZEO

[Paul Winkler]

I'm running ZEO 2 from (according to ZODB/__init__.py) ZODB 3.1.2b2.
(I'm pretty sure i installed this from the 3.1.2 tarball.)

I have a .zexp that's about 569 MB.  I tried to import this into
a different zope instance, and it fails with ...
  File /zope/Zope_software_home/lib/python/ZEO/TransactionBuffer.py, line 99, in _store
IOError: [Errno 28] No space left on device

This is odd because prior to attempting this I had 1.6 GB free in /tmp.
(It's an ext3 partition, linux.)

Looking in TransactionBuffer, adding a bit of logging there,  reading 
the Ext3 FAQ, and playing with lsof and df, I'm pretty sure that 
TransactionBuffer opens a temp file in /tmp which is somehow deleted but not 
actually closed until zope exits.  So the space is not reclaimed by
the filesystem.

So, even if I import something large enough to be buffered but small 
enough to not trigger the IOError, the space in /tmp is never 
reclaimed while zope is up.  This is no good. If I repeatedly import 
the same .zexp, the free space in /tmp declines each time until eventually 
I will get the IOError.

So, questions...

1) What is the relationship between the temp space required by 
TransactionBuffer and the size of the .zexp? When I changed my TMPDIR,
restarted, and successfully imported the 569-MB file, the used 
temp space was well over 2 GB!  But when I imported a 120-MB file,
the used temp space was just over 120 MB.

2) TransactionBuffer.clear() seeks to zero in the file and resets its
size counter, but doesn't close the file.
Any reason not to close the file and open a new one here?


-- 

Paul Winkler
http://www.slinkp.com
Look! Up in the sky! It's MEGA SPICY DUDE!
(random hero from isometric.spaceninja.com)