[ZODB-Dev] Use of fsync in FileStorage

[Tim Peters]

[Toby Dickenson]
> You posted this link - was that the thread you meant?

Right.

>> According to point #5 in
>>  
>> http://mail.python.org/pipermail/python-dev/2003-September/038375.html
>> Linux will allow fsync() to return even if the data isn't actually on
>> disk.

> Yes, linux can allow fsync to return before the data is on disk, if the
> disk has a volatile write cache enabled. Some IDE disks have it on by
> default, and some do not allow it to be turned off.
>
> Im not sure this is news. I believe these problems are limited to the
> low-end IDE products, not server-class disks where we expect ZODB to be
> robust.

I'm more interested in facts than news.  I don't know what the facts are
here.  What I've heard so far is that:

- You need the right flavor of the right OS.

- You need the right filesystem implementation.  For example, here:

    http://mail.zope.org/pipermail/zodb-dev/2003-January/004211.html

  you appear to be saying that reiserfs is good enough, but ext2 isn't:

    On ext2 there are certain operations that can not be performed
    safely, without the risk of a crash or power loss dumping files in
    /lost+found.

- You need the right hardware (disks, controllers).  I haven't seen
  a way to determine which HW is, and which isn't, acceptable.

- You may need to configure your HW (such as disabling disk write caching).

None of that is news, and indeed it's sketching an outline of what the POSIX
spec asks for in a conformance document, explaining how users *can* get data
stored to disk.  If most of it is true, there are no easy answers here --
adding more fsync()s may help some people in rare cases, at some cost to
all.

I don't know that Zope Corp has any position on what kind of disks ZODB
users have to use, and I'm sure it doesn't have a position on which
filesystem they have to use.  Perhaps so long as DirectoryStorage
effectively requires reiserfs, you gain the advantage of a particularly
robust filesystem to build on (btw, googling today turned up a lot of copies
of old msgs from you on reiserfs-list, quizzing the developers about fsync()
semantics -- but mostly not getting answers <wink>).

If someone wants to develop, test, and submit FileStorage patches that make
a real improvement on their box, they'll be considered.  Short of that, I'm
too overburdened to worry about problems we haven't seen.

BTW, I would encourage anyone running a serious site to invest in a good
uninterruptible power supply, so they can stop worrying about power outages.
That's good server-class practice too!