[ZODB-Dev] Re: Use of fsync in FileStorage

[Chris McDonough]

Stats for an "enterprise class" 3.0GHz dual-processor system set up with
hyperthreading, running Linux (RHEL 3.0; kernel 2.4.21), on an "Emulex
LightPulse LP982" controller which fronts for a SAN-attached RAID array
that appears to the kernel as a SCSI device (I don't have physical
access to it so I don't know quite what this means but it sounds
impressive) using 10,000 commits:

2 fsyncs (added an additional fsync to tpc_vote, per Sidnei's code):
  38.591 seconds, 259.128 txn/sec

1 fsync (default Zope 2.7.1 code):
  23.4319 seconds, 426.769 txn/sec

0 fsyncs (remove fsync from _finish):
  3.91192 seconds, 2556.29 txn/sec

Ran each of these tests twice with no significant variation.

This would seem to break the pattern of 1-vs-2 fsyncs not behaving much
differently on Linux, as there is a 60% difference in speed between 1
and 2 fsyncs in this setup.  I have no idea why, although this seems to
be the first test where the disks are network-attached as opposed to
bus-attached, so maybe that has something to do with it.

- C

On Fri, 2004-07-30 at 00:09, Tim Peters wrote:
> [Casey Duncan]
> > Some timing on a 700MHz notebook with a 30gig drive running FreeBSD
> > 5.1/ufs:
> >
> > Stock ZODB:    57.9809 seconds, 172.471 txn/sec
> > Without fsync: 36.0513 seconds, 277.383 txn/sec
> >
> > (I didn't bother with two fsyncs)
> 
> I hope you change your mind and try that.  So far it looks like an
> additional fsync causes major degradation (vs the status quo) only on my
> box, but I'd like to get as much evidence for that as reasonably possible.
> 
> > The former was I/O bound and the latter CPU bound.
> >
> > I definitely think this means we should consider a knob for those willing
> > to run fast and loose versus those wanting to trade performance for
> > better data integrity.
> 
> The rub is that you might not notice any difference -- the test driver here
> doesn't emulate running a real app, it just beats on a FileStorage as
> intensely as possible, doing nothing else.  FileStorage probably isn't "the
> bottleneck" in any Zope application today, so I expect it to have a minor
> effect -- possibly even an unmeasurable one.
> 
> Example:  the difference between 0 and 1 fsyncs is a factor of *100* in txn
> rate on my box in this test.  If I disable fsyncs, the ZODB/ZEO --all tests
> finish 3 minutes faster on this box.  That's worth something to me, but they
> still take 20 minutes to run, and dropping fsync on this box gives by far
> the biggest win (and adding another fsync the biggest loss) across all boxes
> reported on so far.
> 
> The test suite doesn't emulate a real app either, of course.  It does seem
> worth trying 0, and 2, in a real app.
> 
> > For development, testing, loading and database intensive tasks like
> > cataloging the "fast and loose mode" could be pretty desireable, with the
> > caveat that there may be little real difference between the two with a
> > decent disk controller.
> 
> Or at all.  Note that because Jeremy put the existing fsync in tpc_finish,
> dropping that fsync won't do anything primary to speed tpc_vote (it may have
> good secondary effects), and tpc_vote time is "a classic" ZEO bottleneck
> under heavy load.
> 
> No way to tell for sure without trying it (for more values of "it" than I
> can count ...).
> 
> _______________________________________________
> For more information about ZODB, see the ZODB Wiki:
> http://www.zope.org/Wikis/ZODB/
> 
> ZODB-Dev mailing list  -  ZODB-Dev at zope.org
> http://mail.zope.org/mailman/listinfo/zodb-dev
>