At 10:04 AM 10/22/99 -0400, Michel Pelletier wrote:

...

-----Original Message----- From: Andrew M. Kuchling [mailto:akuchlin@mems-exchange.org] Sent: Thursday, October 21, 1999 9:14 PM To: zope-dev@zope.org Subject: RE: [Zope-dev] ZODB and # of objects

Michel Pelletier writes:

...

Are you using substransactions? If you are using 2.0.x, then you might not notice any improvement with sub-transactions because of a deactivation bug that was fixed just after the release (sigh). If you

Trying the CVS version doesn't seem to help; I suspect that the problem is the growing in-memory index of FileStorage objects, and the fix is to use an alternate Storage class that scales better.

Ty has the Berkely Storage, that might work quite well.

Keep in mind that this storage is still highly experimental, and also that it does not currently support Undo or Versions. (Although Ty might have Versions done in time for next week's Zope News... ;) ) But it does offer superior startup time to FileStorage for large databases and doesn't have the memory growth overhead that FileStorage does. On the other hand, BerkeleyStorage does *not* use temporary files for transaction logging, but keeps everything in memory until the transaction is committed, so if you have transactions involving large numbers of objects, Berkeley may not perform as well, memorywise. The long-term (i.e., a few weeks) plan for BerkeleyStorage is to support Undo, but only inside of Versions. We believe this will produce an ideal trade-off for applications where "run-time" activities would cause undue file-size growth under FileStorage, but where site maintainers/admins/content managers still need to have some undo capability. The other thing we'd like to do is figure out a way to have Versions manage conflicts optimistically, allowing non-versioned objects to be changed while a version is being worked on. This would allow tests of cataloging things, etc. without locking the catalog outside the version. ConflictError could then be raised on version commit if the appropriate transactions aren't undone. The undo data provided by the version could include indicators of which transactions involve objects which have been modified on the 'main line' and need to be undone for commit to occur. This is a bit more ambitious a project, as it would probably also require the ability to "revert" individual DB objects to their un-versioned state, independently of transactional undo, and/or to request a "hard" lock on an object.