[Zope] need advice on mass data processing

[Dieter Maurer]

Allen Huang wrote at 2007-1-8 19:28 -0800:
>currently I have written my program so it will do an entry by entry processing with zope. This operation is like this
> 
>1. read data (the data file)
>2. create product (a python product that store three field data: one string and two float data)
>3. update product (update the three field entries)
> 
>when I first tried it out with the first 1000 entries it took about 30 seconds. That means its going to take 50 ~ 60 minutes for 110000 entries. 

You need to be especially careful in which container you dump
your entries. You *MUST NOT* use a standard folder for this.
It is not fit for larger amounts of entries. Use instead
a "BTreeFolder2".


Another way to speed up the creation process is to batch
the creation and commit a transaction only for every X entries.


Using these techniques, I expect that you will be able
to create about 100 entries per second.



-- 
Dieter