Shane Hathaway wrote:
"Morten W. Petersen" wrote:
during testing of a mail product I've discovered that the Data.fs file may bloat considerably after storing 50 messages. Packing the database will reduce the Data.fs file to 20 MB (from 40 MB). Another thing is that storing 50 messages takes a *long time* on a 600Mhz 256 MB RAM system.
Did you catalog each message? What version of Zope?
Hold your tongue, Shane! ;-)
The thing that struck me was that transactions may cause a lot of overhead, both in database size and for execution speed, so, the thing I'm wondering about is if there exists a non-undoable storage, or something similar that doesn't support transactions..
Yep, see BerkeleyStorage.
Another thing I noticed BTW, is that retrieving 500 messages and packing the database on every 50th message (all messages retrieved during one request) would lead to off the charts memory usage, the process ended up using 700 MB of RAM (had to add swap files as the footprint grew!)
You could either:
1) Import messages in a single transaction with subtransactions. This would take less time and incur less memory consumption and database bloat. We at DC have experimented with bulk-loading mail into Zope and this is what works the best.
See ZCatalog's ZCatalog.py/catalog_object method for an example of using subtransactions. Note that the cacheFullSweep call in there is necessary to see any RAM savings.
2) Change the database cache parameters to keep the RAM cache as low as possible (set the timeout to 3 seconds and the target size to 10).
3) Manually zap the caches periodically, which is a capability of Zope 2.4.x.
Shane
_______________________________________________ Zope-Dev maillist - Zope-Dev@zope.org http://lists.zope.org/mailman/listinfo/zope-dev ** No cross posts or HTML encoding! ** (Related lists - http://lists.zope.org/mailman/listinfo/zope-announce http://lists.zope.org/mailman/listinfo/zope )