[Zope-dev] Non-undoable storage

[Shane Hathaway]

"Morten W. Petersen" wrote:
> during testing of a mail product I've discovered that the Data.fs file may
> bloat considerably after storing 50 messages.  Packing the database will
> reduce the Data.fs file to 20 MB (from 40 MB).  Another thing is that
> storing 50 messages takes a *long time* on a 600Mhz 256 MB RAM system.

Did you catalog each message?  What version of Zope?

> The thing that struck me was that transactions may cause a lot of
> overhead, both in database size and for execution speed, so, the thing I'm
> wondering about is if there exists a non-undoable storage, or something
> similar that doesn't support transactions..

Yep, see BerkeleyStorage.

> Another thing I noticed BTW, is that retrieving 500 messages and packing
> the database on every 50th message (all messages retrieved during one
> request) would lead to off the charts memory usage, the process ended up
> using 700 MB of RAM (had to add swap files as the footprint grew!)

You could either:

1) Import messages in a single transaction with subtransactions.  This
would take less time and incur less memory consumption and database
bloat.  We at DC have experimented with bulk-loading mail into Zope and
this is what works the best.

2) Change the database cache parameters to keep the RAM cache as low as
possible (set the timeout to 3 seconds and the target size to 10).

3) Manually zap the caches periodically, which is a capability of Zope
2.4.x.

Shane