[Zope] need advice on mass data processing
Andreas Jung
lists at zopyx.com
Tue Jan 9 00:55:34 EST 2007
--On 8. Januar 2007 19:28:32 -0800 Allen Huang <swapp0 at yahoo.com> wrote:
> I have a data file that has over 110000 entry of 3 column data (string,
> float, float)
> currently I have written my program so it will do an entry by entry
> processing with zope. This operation is like this
> 1. read data (the data file)
> 2. create product (a python product that store three field data: one
> string and two float data) 3. update product (update the three field
> entries)
Please name things the right way. A "Product" is basically a Zope/Python
package that contains definitions of classes, scripts, templates etc.
You mean instances of a particular class?
>
> when I first tried it out with the first 1000 entries it took about 30
> seconds. That means its going to take 50 ~ 60 minutes for 110000 entries.
You're creating 110k instances for storing a string and two floats?
If yes, that's stupid idea.
You can persistent large amounts of data within a single instances
by using Zope BTrees.
> It not every day that you have to process over 110000 data entries but
> processing over 60 minutes is still kind of long.
>
What kind of processing?
-aj
-------------- next part --------------
A non-text attachment was scrubbed...
Name: not available
Type: application/pgp-signature
Size: 186 bytes
Desc: not available
Url : http://mail.zope.org/pipermail/zope/attachments/20070109/2dfc9232/attachment.bin
More information about the Zope
mailing list