[ZODB-Dev] ZEO client cache tempfile oddness

[Paul Winkler]

On Thu, Apr 05, 2007 at 07:41:28PM -0400, Tim Peters wrote:
> [Paul Winkler]
> >Can somebody explain to me the intent of this dance?
> >(from ZEO/cache.py):
> 
> The ZEO client cache is stored in a fixed-size disk file.  When a ZEO
> client needs to create this file for the first time, it's trying to
> ensure there's enough space on disk for it at the start, and reserve
> that disk space then, rather than risk dying with a "no space left on
> device" error umpteen hours later.  Alas, there is no portable way to
> do so (the C standard says nothing about physical devices such as
> disks).

So it might work as intended on some systems but inspire false
confidence on others. That's sad, but I guess we can't do much about
it.

> >            # Make sure the OS really saves enough bytes for the file.
> >            self.f.seek(self.maxsize - 1)
> >            self.f.write('x')
> >            self.f.truncate()
> >
> >The comment seems self-explanatory, except that I'm not sure what
> >"saves" means. Does the behavior vary on different filesystems?
> 
> Yes.
> Sounds like it optimizes for sparse files.  Not all filesystems do.

OK... I'm still wondering on which filesystems this code actually does
guarantee sufficient space. Our experiments suggest that ext2, ext3,
and reiserfs optimize for sparse files so there is no such guarantee.
AFAICT from some quick googling and wikipediaing, the same is true for
NTFS, XFS, JFS, ZFS. I suspect we've accounted for the majority of the
production Zope installations in the world.  The only fs I found that
has no sparse file support is HFS+.  Not sure about UFS (I found
people claiming both no and yes).

> If you care, the intent could probably be better served by changing
> the code to write a junk byte at every (say) thousandth byte offset
> throughout the file (at least one non-NUL byte per disk block).  Alas,
> there would still be no guarantee that there's actually enough space
> on the physical disk to store it.

OK, then I don't see how we could get a real guarantee without
actually writing junk to every byte, which might be just a little
slower :)

Short of that, since crystal ball technology is still not widely
deployed outside the Python Secret Underground, we can't know if
currently free space will still be available when we need it.

Thanks for shedding some light,

-- 

Paul Winkler
http://www.slinkp.com