Allen Huang wrote at 2007-1-8 19:28 -0800:
currently I have written my program so it will do an entry by entry processing with zope. This operation is like this
1. read data (the data file) 2. create product (a python product that store three field data: one string and two float data) 3. update product (update the three field entries)
when I first tried it out with the first 1000 entries it took about 30 seconds. That means its going to take 50 ~ 60 minutes for 110000 entries.
You need to be especially careful in which container you dump your entries. You *MUST NOT* use a standard folder for this. It is not fit for larger amounts of entries. Use instead a "BTreeFolder2". Another way to speed up the creation process is to batch the creation and commit a transaction only for every X entries. Using these techniques, I expect that you will be able to create about 100 entries per second. -- Dieter