[ZODB-Dev] Errors running a Stress Test
Tim Peters
tim at zope.com
Tue Mar 30 21:09:55 EST 2004
[Tim sez pack is generally called infrequently, and that it's good to
pack to some time in the (recent) past instead of to "right now";
calling it frequently, and from multiple processes simultaneously, and
packing to "right now", cause several of the odd tracebacks in Bradon's
tests; and we don't have any plans to change those
]
[Brandon Ehle]
> I guess that pack means something different to me, because on average,
> the FileStorage is growing on me by about 50MB / hour on a machine
> with a fast but small 10,000 RPM drive. Thus my stress test is
> designed to mimic my regular usage, but uncover errors faster.
FileStorage obviously wasn't designed for small disks, but I don't think
there's anything "wrong" with your usage. You can still pack to some time
in the recent past (say, pack to 5 minutes ago, if transactions typically
take less than a minute), and you can still arrange to run no more than one
pack at a time. Then you won't see the kinds of tracebacks explained in the
last mail.
> I currently need to pack several times a day (about once a hour
> currently) to prevent two things:
>
> 1) Not run out of disk space.
Packing to several minutes in the past should make little difference to
that.
> 2) To make the pack do less work in a single operation so that my
> transaction throughput doesn't suffer and cause conflicts.
I don't follow that one. Pack time is roughly proportional to the sums of
the sizes of the "before" and "after" files. If I pack 12 times in 12
hours, or one time in 12 hours, I get rid of the same total number of
expired objects, but pay repeatedly to copy the then-still-live objects in
the former case. So I expect that *total* system burden would lessen by
packing less often. OTOH, a large pack sucks up more disk cache, etc. If
you've tried it both ways and measured it objectively, I'll agree to believe
whatever your clock said <wink>.
More information about the ZODB-Dev
mailing list