Sorry Andreas.. My product is rewritten in python placed in the product folder. So yes, it is a instance of a class. In the future, I will be storing more data into one instance from other dbase file (The total columns may vary). I'm still in the developing stage and this is just a test run to know that is the length of processing time I'm look at. I want each entry to be an instance because I'm planning to create other interactive functions (edit and query(ZCatalog maybe, I'm not sure yet) and more ) for particular info. I've never thought of using a BTree because I don't know enough about it. I'll look into it but will BTree still be a better choice than making instance if I'm going to make interactive functions? Any other suggestions? ----- Original Message ---- From: Andreas Jung <lists@zopyx.com> To: Allen Huang <swapp0@yahoo.com>; Zope <zope@zope.org> Sent: Tuesday, January 9, 2007 1:55:34 PM Subject: Re: [Zope] need advice on mass data processing --On 8. Januar 2007 19:28:32 -0800 Allen Huang <swapp0@yahoo.com> wrote:
I have a data file that has over 110000 entry of 3 column data (string, float, float) currently I have written my program so it will do an entry by entry processing with zope. This operation is like this 1. read data (the data file) 2. create product (a python product that store three field data: one string and two float data) 3. update product (update the three field entries)
Please name things the right way. A "Product" is basically a Zope/Python package that contains definitions of classes, scripts, templates etc. You mean instances of a particular class?
when I first tried it out with the first 1000 entries it took about 30 seconds. That means its going to take 50 ~ 60 minutes for 110000 entries.
You're creating 110k instances for storing a string and two floats? If yes, that's stupid idea. You can persistent large amounts of data within a single instances by using Zope BTrees.
It not every day that you have to process over 110000 data entries but processing over 60 minutes is still kind of long.
What kind of processing? -aj __________________________________________________ Do You Yahoo!? Tired of spam? Yahoo! Mail has the best spam protection around http://mail.yahoo.com