[Zope] Exporting zodb data (Urgent)

[Paul Winkler]

On Tue, Jul 01, 2003 at 12:04:11AM +0530, Sukhwinder Singh wrote:
> The main "/" (root) folder of the application, under which all other files
> are stored shows a tooltip of "Ffolder w/Customizer support".
(snip)

> The objects I am talking about, like employees and students, are shown with
> specialist icons( Icon of a person sitting on a desk). What this means? What
> kind of objects these are?

OK, it appears from these statements that our earlier guesses were wrong. 
Your application is in fact built with some additional zope Products. 
I don't recognize icons by that description, I am guessing the Products
were created specifically for this project.

When you point at an icon in the ZMI, you normally get a tooltip showing
the meta_type of the Product. This identifies the product. You have
a folder of type "Folder w/Customizer Support"; what about these person-at-
desk icons? what tooltip do you get for those?

When you know the meta_type of the product, go to the zope root, click on
Control Panel, then click on Product Management. There should be items there
with similar names to what you see in the tooltips.
You might even want to paste the whole list in a message here - we're familiar 
with the standard products, we could then guess which ones look like they were 
added for your application.

When you know which products are relevant to your needs, look at the little
blue box icon next to them in the Product Management list.
Most of the little blue boxes look "closed" (there is just a small little
thing sticking up off the top of the box... don't ask me what that is
supposed to represent!).  

Do the icons for your products look like all the others, or are they
clearly different - looking like an open box with the lid sticking off to
the left?

Let us know which and we'll proceed from there.

> When I click these objects (for example, employee) and check their
> properties, no property is defined.

OK, I was wrong about that, sorry. It's hard to guess how somebody
built an application when you can't even see it :-)

> But under these objects their are dtml
> methods, python scripts and other dtml files. Some external methods were
> added by
> earlier programmer under these folder/objects  to export data from each of
> these
> objects (specialists). But I don't know how he determined attributes of
> these objects (like employee.id, employee.is_operator) etc.
> 
> These external methods in each specialist call a scipt like this to export
> data:
> 
> /////////////////////////////////////////////////////////////////
> def list_all_employees(self):
>   self.REQUEST.RESPONSE.setHeader('Content-Type','text/csv')
>   start_time=DateTime()
>   container=self

ah I see, that's how "container" got into the External Method.
I was wondering about that...

>   result=[]
>   r=""
>   q=""
>   e_id=""
>   e_name=""
>   e_isop=""
>   e_surname=""
> 
>   employees=container.Catalog()

ok, this application DOES use ZCatalog.
(At least, I am assuming that container.Catalog is an instance
of ZCatalog).  I wonder why the heck it's slow. That's not normal.

>   for employee in employees:
>     e_name = employee.name
(snip)

OK, you are not actually iterating over objects in the folder.
You are iterating over catalog search results. Each search result
contains information about an object - but it is not identical with the
object.  This means that in order for you to use an attribute of
a result object, the catalog must be configured to keep this data
about this object.  This is done on the Metadata tab of the catalog
instance.

So, go to one of these Catalogs, click on it, then click the
Metadata tab.  You should see that there is a Metadata entry for
all the attributes that are accessed in the external method.

>     e_id = employee.id
>     e_isop = "%s" % (employee.is_operator)
>     e_surname = employee.surname
>     q+=e_id+"#"+e_isop+"#"+e_name+"#"+e_surname+"\n"
> 
>   return q
>

I wonder if there's a lot of stuff like this in the application.
String concatenation is O(N**2) in python, and this script
is potentially doing it a LOT depending on the size of the catalog
results. It's much faster to build a list, and then join the list
elements once at the end.  List appends and string joins are O(N).

But as I said before, I can't actually tell if that's the cause for 
the slowness without profiling the app.

> But problem is that he had to add these external methods under each of these
> object himself to export data.

certainly not! for that kind of job, acquisition is your friend.
You could have one external method that calls several functions,
one for each kind of data to export (employees, etc.), much like
you have now. Put the external method at the top of the application
folder tree.

Now, it sounds like for some reason you have multiple catalogs, though
I'm not sure of this. Is there one Catalog for Employees, one for
Students, etc.?  If so, each function would look something like:

def extract_employees(self):
    catalog = self.path.to.where.the.catalog.lives.Catalog
    results = []
    for emp in catalog():
        e_id = employee.id
        ... etc ...
        results.append("%s#%s#%s#%s" % (e_id, e_isop, e_name, e_surname))

    # now join the list elements with newlines
    return "\n".join(results)


> Also how did he determine, whether there was
> an attribute "employee.is_operator" and names of other attributes?

He probably looked at the employee Product. We're getting there...

-- 

Paul Winkler
http://www.slinkp.com
Look! Up in the sky! It's UNDER-PEAR OF THE DEEP!
(random hero from isometric.spaceninja.com)