Erik Enge writes:
I've just finished adding a somewhat small number of objects: 5000. For every 1000th object, the Data.fs seemed to grow to about 900MB; that's when things started going slow, in a non-linear fashion (this is more a hunch than something I payed much attention to).
I paused the script (fancy Unix-command: "^Z") for every 1000th object, packed the database (which shrunk to 19.5MB! Hmpf.) and restarted the script (again, fancy Unix-command: "fg"). Then I was back to the same speed as I initially had. Are the imported object "CatalogAware"? This would imply that each object's import rewrites part of the catalog indexes and lets your ZODB grow drastically.
The old (pre 2.3.1) catalog implementation was know not to be very storage friendly. If a significant portion of the catalog indexes would be affected by imports, then you would see a quadratic storage increase. The new BTree implementation in 2.3.1 might reduce the problem. Dieter