[ZODB-Dev] From memory Problems to Disk Problems :-(

[Chris Withers]

Hi,

Well, moving from FileStorage to BerkleyStorage hasn't solved the problem, but
it has made it different :-S

The import script only uses 100Mb odd of RAM now, but something that the BDB
tables involved are doing is causing the cache to jump up to 536Mb :-(

How can indexing objects cause this to happen?
What should I be doing?
I'm currently indexing the objects in batches of 50 before committing, each
commit takes roughly 25 minutes(!!!) so I'm getting a little concerned.

How can I check how much data is being written to the ODB? What should I be
looking to optimise with for BDB? Lots of small objects or fewer big objects?
I'm only using the BTrees module here and I'm getting to a point of despair with
using it...

Should I just give up and fidn some way of indexing into SQL or can anyoen offer
any help here?

I can give any details necessary, just ask...

*sigh*

Chris