[Zope] need advice on mass data processing

[Allen Huang]

Sorry Andreas..

My product is rewritten in python placed in the product folder. So yes, it is a instance of a class. 

In the future, I will be storing more data into one instance from other dbase file (The total columns may vary).

I'm still in the developing stage and this is just a test run to know that is the length of processing time I'm look at. I want each entry to be an instance because I'm planning to create other interactive functions (edit and query(ZCatalog maybe, I'm not sure yet) and more ) for particular info.

I've never thought of using a BTree because I don't know enough about it. I'll look into it but will BTree still be a better choice than making instance if I'm going to make interactive functions?

Any other suggestions?



----- Original Message ----
From: Andreas Jung <lists at zopyx.com>
To: Allen Huang <swapp0 at yahoo.com>; Zope <zope at zope.org>
Sent: Tuesday, January 9, 2007 1:55:34 PM
Subject: Re: [Zope] need advice on mass data processing


--On 8. Januar 2007 19:28:32 -0800 Allen Huang <swapp0 at yahoo.com> wrote:

> I have a data file that has over 110000 entry of 3 column data (string,
> float, float)
> currently I have written my program so it will do an entry by entry
> processing with zope. This operation is like this
> 1. read data (the data file)
> 2. create product (a python product that store three field data: one
> string and two float data) 3. update product (update the three field
> entries)

Please name things the right way. A "Product" is basically a Zope/Python 
package that contains definitions of classes, scripts, templates etc.

You mean instances of a particular class?

>
> when I first tried it out with the first 1000 entries it took about 30
> seconds. That means its going to take 50 ~ 60 minutes for 110000 entries.

You're creating 110k instances for storing a string and two floats?
If yes, that's stupid idea.

You can persistent large amounts of data within a single instances
by using Zope BTrees.

> It not every day that you have to process over 110000 data entries but
> processing over 60 minutes is still kind of long.
>

What kind of processing?

-aj

__________________________________________________
Do You Yahoo!?
Tired of spam?  Yahoo! Mail has the best spam protection around 
http://mail.yahoo.com 
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://mail.zope.org/pipermail/zope/attachments/20070109/9f165971/attachment.htm