Exporting zodb data (Urgent)
Hello, (I posted this question to zope-db@zope.org but no one replied so I am posting it here as it is urgent.) I have a web application which is written by some other person using zope/zodb/python. Now client wants to write this application using php/mysql. But existing data stored in zodb database has to be exported to mysql database. Zodb uses fileStorage to store data. I have tried zope's xml export but that doesn't make any sense to me. Now problem is I don't have much knowledge about zope. I have only basic knowledge that everything is stored as an object. I also have some knowledge of python. I have searched the web for any information about zodb but could not find much. I have only 2-3 days so I cannot learn zope then zodb and then python. My problems are: 1. How do I identify which of these objects store data(tables) because zope stores everything (images, dhtml files, folders etc) as objects so I want only objects which store data(rows) and then how many of these tables are there? 2.How do I determine how many fields are there in each data object and what are their names and data types. 3. How do I extract data from these objects and then store that to some intermediate format (csv) and then import it to mysql. So what I want to do is introspection. I don't know where to start. There are many sub-folders under this web applications. I know that I have to use external methods and I have tried it like this: from ZODB import FileStorage, DB storage = FileStorage.FileStorage('data.fs') db = DB( storage ) connection = db.open() root = connection.root() // up to this point it works. now when I try something like: students = root['students'] error message is displayed that students not found. For this reason I want to determine in which objects data is stored so that I can get its values. I also have to determine the name of the attributes (fields) used in this objects so that I can create mysql table fields of the same name and data type. If possible, please give me complete code which I can just copy and paste in external method file. If that is no possible then provide some other working example or explain the steps in detail as I don't have much knowledge about zope and co.
You do have a problem. I suspect you will need to learn a bit about Zope to do much of anything. Have you examined the system using the ZMI? http://yourURL/manage? You can extract most of the information in the system through FTP. Which port depends upon how the Zope system has been started. Typically, if you start XZZope on port 8080, ftp access is available on port 8021. HTH, -d On Fri, 27 Jun 2003, Sukhwinder Singh wrote:
Hello,
(I posted this question to zope-db@zope.org but no one replied so I am posting it here as it is urgent.) I have a web application which is written by some other person using zope/zodb/python. Now client wants to write this application using php/mysql. But existing data stored in zodb database has to be exported to mysql database. Zodb uses fileStorage to store data. I have tried zope's xml export but that doesn't make any sense to me.
Now problem is I don't have much knowledge about zope. I have only basic knowledge that everything is stored as an object. I also have some knowledge of python. I have searched the web for any information about zodb but could not find much. I have only 2-3 days so I cannot learn zope then zodb and then python. My problems are:
1. How do I identify which of these objects store data(tables) because zope stores everything (images, dhtml files, folders etc) as objects so I want only objects which store data(rows) and then how many of these tables are there?
2.How do I determine how many fields are there in each data object and what are their names and data types.
3. How do I extract data from these objects and then store that to some intermediate format (csv) and then import it to mysql.
So what I want to do is introspection.
I don't know where to start.
There are many sub-folders under this web applications. I know that I have to use external methods and I have tried it like this:
from ZODB import FileStorage, DB storage = FileStorage.FileStorage('data.fs') db = DB( storage ) connection = db.open() root = connection.root() // up to this point it works.
now when I try something like:
students = root['students']
error message is displayed that students not found.
For this reason I want to determine in which objects data is stored so that I can get its values. I also have to determine the name of the attributes (fields) used in this objects so that I can create mysql table fields of the same name and data type.
If possible, please give me complete code which I can just copy and paste in external method file. If that is no possible then provide some other working example or explain the steps in detail as I don't have much knowledge about zope and co.
You do have a problem. I suspect you will need to learn a bit about Zope to do much of anything. Have you examined the system using the ZMI? http://yourURL/manage? You can extract most of the information in the system through FTP. Which port depends upon how the Zope system has been started. Typically, if you start XZZope on port 8080, ftp access is available on port 8021.
I have tried to read whatever I could in last two days. But there is not much information available on net about zodb. I have used and explored zope management interface. I also read the zope developers guide. There is not much information about how zope stores objects in zodb and how to access those objects. Other documentation provide information about storing data and then retrieving that data. No documentation is available about how to retrieve already stored data about which I don't have any prior knowledge. As in java, ide's use introspection to determine what are fields and methods of a object (bean). Other DBMS also allow us to determine how many tables etc. are there in database. But how do I do that with zope and zodb? I can access objects If I knew which objects store data and how many fields are there in each object. Looking into every dtml file or other methods isn't possible in this shot period. Even If I did that I may miss some fields which are used in this application. --Sukhwinder Singh ----- Original Message ----- From: "Dennis Allison" <allison@sumeru.stanford.EDU> To: "Sukhwinder Singh" <ssruprai@hotmail.com> Cc: <zope@zope.org> Sent: Friday, June 27, 2003 9:45 PM Subject: Re: [Zope] Exporting zodb data (Urgent)
You do have a problem. I suspect you will need to learn a bit about Zope to do much of anything. Have you examined the system using the ZMI? http://yourURL/manage? You can extract most of the information in the system through FTP. Which port depends upon how the Zope system has been started. Typically, if you start XZZope on port 8080, ftp access is available on port 8021.
HTH, -d
On Fri, 27 Jun 2003, Sukhwinder Singh wrote:
Hello,
(I posted this question to zope-db@zope.org but no one replied so I am posting it here as it is urgent.) I have a web application which is written by some other person using zope/zodb/python. Now client wants to write this application using php/mysql. But existing data stored in zodb database has to be exported to mysql database. Zodb uses fileStorage to store data. I have tried zope's xml export but that doesn't make any sense to me.
Now problem is I don't have much knowledge about zope. I have only basic knowledge that everything is stored as an object. I also have some knowledge of python. I have searched the web for any information about zodb but could not find much. I have only 2-3 days so I cannot learn zope then zodb and then python. My problems are:
1. How do I identify which of these objects store data(tables) because zope stores everything (images, dhtml files, folders etc) as objects so I want only objects which store data(rows) and then how many of these tables are there?
2.How do I determine how many fields are there in each data object and what are their names and data types.
3. How do I extract data from these objects and then store that to some intermediate format (csv) and then import it to mysql.
So what I want to do is introspection.
I don't know where to start.
There are many sub-folders under this web applications. I know that I have to use external methods and I have tried it like this:
from ZODB import FileStorage, DB storage = FileStorage.FileStorage('data.fs') db = DB( storage ) connection = db.open() root = connection.root() // up to this point it works.
now when I try something like:
students = root['students']
error message is displayed that students not found.
For this reason I want to determine in which objects data is stored so that I can get its values. I also have to determine the name of the attributes (fields) used in this objects so that I can create mysql table fields of the same name and data type.
If possible, please give me complete code which I can just copy and paste in external method file. If that is no possible then provide some other working example or explain the steps in detail as I don't have much knowledge about zope and co.
On Sun, Jun 29, 2003 at 01:13:39AM +0530, Sukhwinder Singh wrote:
I have tried to read whatever I could in last two days. But there is not much information available on net about zodb. I have used and explored zope management interface. I also read the zope developers guide. There is not much information about how zope stores objects in zodb and how to access those objects.
It's generally used as a black box. Zope developers typically don't interact directly with the ZODB the way that a java developer might with an RDBMs. The low-level storage details are mostly automatic, and therefore irrelevant to your needs.
Other documentation provide information about storing data and then retrieving that data. No documentation is available about how to retrieve already stored data about which I don't have any prior knowledge.
Explore some instances in the ZMI and if that's not enough, Use The Source.
As in java, ide's use introspection to determine what are fields and methods of a object (bean). Other DBMS also allow us to determine how many tables etc. are there in database. But how do I do that with zope and zodb?
by exploring in the ZMI, by using the source if you want to know how an object REALLY works, and / or by using the interactive debugger to explore things. http://www.zope.org/Members/klm/ZopeDebugging/ConversingWithZope -- Paul Winkler http://www.slinkp.com Look! Up in the sky! It's THE SCHIZOID THANATOS! (random hero from isometric.spaceninja.com)
Sukhwinder Singh wrote at 2003-6-29 01:13 +0530:
... But there is not much information available on net about zodb.
Quantity is seldom a measure for quality... Did you find the ZODB3 description ("zodb3.pdf/zodb3.html")? Did you find the ZODB guide?
... There is not much information about how zope stores objects in zodb
That's because they are stored automatically and naturally. There is nothing the ZODB user must do. Storing and loading happens transparently. You just access objects naturally (as provided by Python) and modify them (there are some caveats for non-persistent mutable attributes). Everything else is done automatically be the ZODB in the background. Your problem is not a ZODB problem. Your problem is your lack of knowledge about the application! Any ZODB object contains data, but not all ZODB objects contain data relevant for your application. You access the content of your objects usually through an API. Which one is determined by your application and not by the ZODB. Usually, you use the "PropertyManager" or "PropertySheet" API to access properties and the "ObjectManager" API to access children. This is because most applications use properties to store elementary data and an object manager to store subobjects. But, applications are free to do it in any other way...
No documentation is available about how to retrieve already stored data about which I don't have any prior knowledge. As in java, ide's use introspection to determine what are fields and methods of a object (bean)
Maybe, you should read about Python's "inspect" module or its "dir" and "vars" builtin functions. Python's inspection support is at least as good as Java's... There is a (small) caveat when using "dir/vars" with objects from the ZODB: The ZODB loads object state only on demand from the database. "dir/vars" does not trigger this mechanism. Therefore, the object may appear empty (when is has not yet been loaded). Accessing any attribute (even one that does not exist) will load the object and "dir/vars" can be used to analyse its state.
. Other DBMS also allow us to determine how many tables etc. are there in database. But how do I do that with zope and zodb?
You do not because the objects are not organized into tables. The closest thing would be to determine the classes of objects stored in the ZODB. This is not supported (as people usually have more knowledge about their application than you have and do not need this service). It would not help you either because the object state is not determined by the class but by the instance itself.
I can access objects If I knew which objects store data
All objects store data... Your application documentation must tell you which objects store data relevant for the application. If it does not, this is not the fault of Zope or the ZODB but of the application development...
and how many fields are there in each object.
"dir" tells you the name of fields in any given object. "vars" gives you name and values...
Looking into every dtml file or other methods isn't possible in this shot period. Even If I did that I may miss some fields which are used in this application.
I fear we cannot help you much with this as we (of course) do not know your application better than you do ;-) Probably your best way: Start with the objects exported by the exporting script you mentioned in another post. These objects form a subset of the one relevant for your application. Analyse these objects with "dir". Many attributes will be internal, all starting with "_" will be. Would you know that your application stores attributes as properties, then the "PropertyManager" API methods would reveal them all. In the future, you should probably more careful when you accept client requests and time frames. Dieter
Did you find the ZODB3 description ("zodb3.pdf/zodb3.html")? Did you find the ZODB guide?
I wasn't able to find zodb guide, only zope guide.
In the future, you should probably more careful when you accept client requests and time frames.
It is not my fault as project was already accepted when I joined this company. Anyway Thank you very much for explaining what I need to do. --Sukhwinder Singh
Dieter Maurer worte at Sunday, June 29, 2003 11:08 AM
for employee in container.employees print employee.name
Go to the "Properties" tab of these objects. Do you see there the relevant properties?
The main "/" (root) folder of the application, under which all other files are stored shows a tooltip of "Ffolder w/Customizer support". There are other folders in this root folder like, Control_Panel, acl_users. When I click the root folder of this application which is "school". Many subfolders and other files are displayed. The objects I am talking about, like employees and students, are shown with specialist icons( Icon of a person sitting on a desk). What this means? What kind of objects these are? When I click these objects (for example, employee) and check their properties, no property is defined. But under these objects their are dtml methods, python scripts and other dtml files. Some external methods were added by earlier programmer under these folder/objects to export data from each of these objects (specialists). But I don't know how he determined attributes of these objects (like employee.id, employee.is_operator) etc. These external methods in each specialist call a scipt like this to export data: ///////////////////////////////////////////////////////////////// def list_all_employees(self): self.REQUEST.RESPONSE.setHeader('Content-Type','text/csv') start_time=DateTime() container=self result=[] r="" q="" e_id="" e_name="" e_isop="" e_surname="" employees=container.Catalog() for employee in employees: e_name = employee.name e_id = employee.id e_isop = "%s" % (employee.is_operator) e_surname = employee.surname q+=e_id+"#"+e_isop+"#"+e_name+"#"+e_surname+"\n" return q def list_all_adverts(self): self.REQUEST.RESPONSE.setHeader('Content-Type','text/csv') start_time=DateTime() container=self result=[] r="" q="" e_id="" e_name="" e_isop="" e_surname="" adverts=container.Catalog() for advert in adverts: a_class = "%s" %(advert.classification) a_id = "%s" %(advert.id) a_desc = "%s" %(advert.description) a_name= "%s" %(advert.name) q+=a_class+"#"+a_desc+"#"+a_id+"#"+a_name+"\n" return q //////////////////////////////////////////////////////////// But problem is that he had to add these external methods under each of these object himself to export data. Also how did he determine, whether there was an attribute "employee.is_operator" and names of other attributes? So now, can I do it without requiring client to install these external methods themselves under each folder? Becuase they don't know how to manage zope. If there was only one script to be installed, then may be able to do it. But for each object it may not be possible. So this is all I know. Can I give them just one script which they can install in root of their application and which when called exports all data to csv or other format? --Sukhwinder Singh
On Tue, Jul 01, 2003 at 12:04:11AM +0530, Sukhwinder Singh wrote:
The main "/" (root) folder of the application, under which all other files are stored shows a tooltip of "Ffolder w/Customizer support". (snip)
The objects I am talking about, like employees and students, are shown with specialist icons( Icon of a person sitting on a desk). What this means? What kind of objects these are?
OK, it appears from these statements that our earlier guesses were wrong. Your application is in fact built with some additional zope Products. I don't recognize icons by that description, I am guessing the Products were created specifically for this project. When you point at an icon in the ZMI, you normally get a tooltip showing the meta_type of the Product. This identifies the product. You have a folder of type "Folder w/Customizer Support"; what about these person-at- desk icons? what tooltip do you get for those? When you know the meta_type of the product, go to the zope root, click on Control Panel, then click on Product Management. There should be items there with similar names to what you see in the tooltips. You might even want to paste the whole list in a message here - we're familiar with the standard products, we could then guess which ones look like they were added for your application. When you know which products are relevant to your needs, look at the little blue box icon next to them in the Product Management list. Most of the little blue boxes look "closed" (there is just a small little thing sticking up off the top of the box... don't ask me what that is supposed to represent!). Do the icons for your products look like all the others, or are they clearly different - looking like an open box with the lid sticking off to the left? Let us know which and we'll proceed from there.
When I click these objects (for example, employee) and check their properties, no property is defined.
OK, I was wrong about that, sorry. It's hard to guess how somebody built an application when you can't even see it :-)
But under these objects their are dtml methods, python scripts and other dtml files. Some external methods were added by earlier programmer under these folder/objects to export data from each of these objects (specialists). But I don't know how he determined attributes of these objects (like employee.id, employee.is_operator) etc.
These external methods in each specialist call a scipt like this to export data:
///////////////////////////////////////////////////////////////// def list_all_employees(self): self.REQUEST.RESPONSE.setHeader('Content-Type','text/csv') start_time=DateTime() container=self
ah I see, that's how "container" got into the External Method. I was wondering about that...
result=[] r="" q="" e_id="" e_name="" e_isop="" e_surname=""
employees=container.Catalog()
ok, this application DOES use ZCatalog. (At least, I am assuming that container.Catalog is an instance of ZCatalog). I wonder why the heck it's slow. That's not normal.
for employee in employees: e_name = employee.name (snip)
OK, you are not actually iterating over objects in the folder. You are iterating over catalog search results. Each search result contains information about an object - but it is not identical with the object. This means that in order for you to use an attribute of a result object, the catalog must be configured to keep this data about this object. This is done on the Metadata tab of the catalog instance. So, go to one of these Catalogs, click on it, then click the Metadata tab. You should see that there is a Metadata entry for all the attributes that are accessed in the external method.
e_id = employee.id e_isop = "%s" % (employee.is_operator) e_surname = employee.surname q+=e_id+"#"+e_isop+"#"+e_name+"#"+e_surname+"\n"
return q
I wonder if there's a lot of stuff like this in the application. String concatenation is O(N**2) in python, and this script is potentially doing it a LOT depending on the size of the catalog results. It's much faster to build a list, and then join the list elements once at the end. List appends and string joins are O(N). But as I said before, I can't actually tell if that's the cause for the slowness without profiling the app.
But problem is that he had to add these external methods under each of these object himself to export data.
certainly not! for that kind of job, acquisition is your friend. You could have one external method that calls several functions, one for each kind of data to export (employees, etc.), much like you have now. Put the external method at the top of the application folder tree. Now, it sounds like for some reason you have multiple catalogs, though I'm not sure of this. Is there one Catalog for Employees, one for Students, etc.? If so, each function would look something like: def extract_employees(self): catalog = self.path.to.where.the.catalog.lives.Catalog results = [] for emp in catalog(): e_id = employee.id ... etc ... results.append("%s#%s#%s#%s" % (e_id, e_isop, e_name, e_surname)) # now join the list elements with newlines return "\n".join(results)
Also how did he determine, whether there was an attribute "employee.is_operator" and names of other attributes?
He probably looked at the employee Product. We're getting there... -- Paul Winkler http://www.slinkp.com Look! Up in the sky! It's UNDER-PEAR OF THE DEEP! (random hero from isometric.spaceninja.com)
Thanks for trying to help me. On Monday, June 30, 2003 1:13 PM Paul Winkler wrote:
When you point at an icon in the ZMI, you normally get a tooltip showing the meta_type of the Product. This identifies the product. You have a folder of type "Folder w/Customizer Support"; what about these person-at- desk icons? what tooltip do you get for those?
I have already wrtten that icon of person-at-desk shows word "specialist"
When you know the meta_type of the product, go to the zope root, click on Control Panel, then click on Product Management. There should be items there with similar names to what you see in the tooltips. You might even want to paste the whole list in a message here - we're familiar with the standard products, we could then guess which ones look like they were added for your application.
There are many products listed here. Product I am interested in is named school with icon of an open box. Other products are with closed box icon. Here is the complete list: ArmTriggersHack (Installed product ArmTriggersHack) 2001-12-13 11:33 -->School 2003-03-17 16:49 ExternalMethod (Installed product ExternalMethod (External Method-1-0-0)) 2001-12-13 11:33 LoginManager (Installed product LoginManager (LoginManager-0-8-8b1)) 2001-12-13 11:33 MIMETools (Installed product MIMETools) 2001-12-13 11:33 MailHost (Installed product MailHost (MailHost-1-2-0)) 2001-12-13 11:33 OFSP (Installed product OFSP (OFSP-1-0-0)) 2001-12-13 11:33 PatchMiniDom (Installed product PatchMiniDom) 2001-12-13 11:33 PlugIns (Installed product PlugIns (PlugIns-0-4-3p1)) 2001-12-13 11:33 PluginIndexes (Installed product PluginIndexes) 2003-03-17 15:58 PythonScripts (Installed product PythonScripts (PythonScripts-2-0-0)) 2001-12-13 11:33 SiteAccess (Installed product SiteAccess (SiteAccess-2-0-0)) 2001-12-13 11:33 StandardCacheManagers (Installed product StandardCacheManagers (StandardCacheManagers-1-1-0)) 2001-12-13 11:33 TransactionAgents (Installed product TransactionAgents (TransactionAgents 0.0.1)) 2001-12-13 11:33 ZCatalog (Installed product ZCatalog (ZCatalog-2-2-0)) 2001-12-13 11:33 ZGadflyDA (Installed product ZGadflyDA) 2001-12-13 11:33 ZODBCDA (Installed product ZODBCDA) 2003-03-21 10:15 ZPatterns (Installed product ZPatterns (ZedPatterns-0-4-3p2-transactionagents)) 2001-12-13 11:33 ZSQLMethods (Installed product ZSQLMethods) 2001-12-13 11:33 ZopeTutorial (Installed product ZopeTutorial (Zope Tutorial 1.0))
This means that in order for you to use an attribute of a result object, the catalog must be configured to keep this data about this object. This is done on the Metadata tab of the catalog instance. So, go to one of these Catalogs, click on it, then click the Metadata tab. You should see that there is a Metadata entry for all the attributes that are accessed in the external method.
Is cataloge an object which represents the zodb (data.fs) file? How can I access this in catalog ZMI?
Is there one Catalog for Employees, one for Students, etc.?
Unless I know what catalog is, I can't answer this question?
def extract_employees(self): catalog = self.path.to.where.the.catalog.lives.Catalog
example path?
e_id = employee.id
How do you know that there was id attribute?
certainly not! for that kind of job, acquisition is your friend. You could have one external method that calls several functions, one for each kind of data to export (employees, etc.), much like you have now. Put the external method at the top of the application folder tree.
OK I put that external method in root folder then? What external method should look like? Like that example above? I think you will eventually solve my problem. I still have 2-3 days. --Sukhwinder Singh
Thanks for trying to help me. On Monday, June 30, 2003 1:13 PM Paul Winkler wrote:
When you point at an icon in the ZMI, you normally get a tooltip showing the meta_type of the Product. This identifies the product. You have a folder of type "Folder w/Customizer Support"; what about these person-at- desk icons? what tooltip do you get for those?
I have already wrtten that icon of person-at-desk shows word "specialist"
When you know the meta_type of the product, go to the zope root, click on Control Panel, then click on Product Management. There should be items there with similar names to what you see in the tooltips. You might even want to paste the whole list in a message here - we're familiar with the standard products, we could then guess which ones look like they were added for your application.
There are many products listed here. Product I am interested in is named school with icon of an open box. Other products are with closed box icon. Here is the complete list: ArmTriggersHack (Installed product ArmTriggersHack) 2001-12-13 11:33 -->School 2003-03-17 16:49 ExternalMethod (Installed product ExternalMethod (External Method-1-0-0)) 2001-12-13 11:33 LoginManager (Installed product LoginManager (LoginManager-0-8-8b1)) 2001-12-13 11:33 MIMETools (Installed product MIMETools) 2001-12-13 11:33 MailHost (Installed product MailHost (MailHost-1-2-0)) 2001-12-13 11:33 OFSP (Installed product OFSP (OFSP-1-0-0)) 2001-12-13 11:33 PatchMiniDom (Installed product PatchMiniDom) 2001-12-13 11:33 PlugIns (Installed product PlugIns (PlugIns-0-4-3p1)) 2001-12-13 11:33 PluginIndexes (Installed product PluginIndexes) 2003-03-17 15:58 PythonScripts (Installed product PythonScripts (PythonScripts-2-0-0)) 2001-12-13 11:33 SiteAccess (Installed product SiteAccess (SiteAccess-2-0-0)) 2001-12-13 11:33 StandardCacheManagers (Installed product StandardCacheManagers (StandardCacheManagers-1-1-0)) 2001-12-13 11:33 TransactionAgents (Installed product TransactionAgents (TransactionAgents 0.0.1)) 2001-12-13 11:33 ZCatalog (Installed product ZCatalog (ZCatalog-2-2-0)) 2001-12-13 11:33 ZGadflyDA (Installed product ZGadflyDA) 2001-12-13 11:33 ZODBCDA (Installed product ZODBCDA) 2003-03-21 10:15 ZPatterns (Installed product ZPatterns (ZedPatterns-0-4-3p2-transactionagents)) 2001-12-13 11:33 ZSQLMethods (Installed product ZSQLMethods) 2001-12-13 11:33 ZopeTutorial (Installed product ZopeTutorial (Zope Tutorial 1.0))
This means that in order for you to use an attribute of a result object, the catalog must be configured to keep this data about this object. This is done on the Metadata tab of the catalog instance. So, go to one of these Catalogs, click on it, then click the Metadata tab. You should see that there is a Metadata entry for all the attributes that are accessed in the external method.
Is cataloge an object which represents the zodb (data.fs) file? How can I access this in catalog ZMI?
Is there one Catalog for Employees, one for Students, etc.?
Unless I know what catalog is, I can't answer this question?
def extract_employees(self): catalog = self.path.to.where.the.catalog.lives.Catalog
example path?
e_id = employee.id
How do you know that there was id attribute?
certainly not! for that kind of job, acquisition is your friend. You could have one external method that calls several functions, one for each kind of data to export (employees, etc.), much like you have now. Put the external method at the top of the application folder tree.
OK I put that external method in root folder then? What external method should look like? Like that example above? I think you will eventually solve my problem. I still have 2-3 days. --Sukhwinder Singh
On Tue, Jul 01, 2003 at 11:35:17AM -0700, Sukhwinder Singh wrote:
I have already wrtten that icon of person-at-desk shows word "specialist"
ok, sorry, i missed that.
There are many products listed here. Product I am interested in is named school with icon of an open box. Other products are with closed box icon. Here is the complete list:
ArmTriggersHack (Installed product ArmTriggersHack) 2001-12-13 11:33
hmm, I have no idea what that one is, or if it's relevant.
-->School 2003-03-17 16:49
ok...
ZPatterns (Installed product ZPatterns (ZedPatterns-0-4-3p2-transactionagents)) 2001-12-13 11:33
hmm. ZPatterns is a rather complex system which I've never used. If your application uses it, i'm afraid I can't help with that. everything else looks like standard stuff.
This means that in order for you to use an attribute of a result object, the catalog must be configured to keep this data about this object. This is done on the Metadata tab of the catalog instance. So, go to one of these Catalogs, click on it, then click the Metadata tab. You should see that there is a Metadata entry for all the attributes that are accessed in the external method.
Is cataloge an object which represents the zodb (data.fs) file? How can I access this in catalog ZMI?
ZCatalog is described here: http://www.zope.org/Documentation/Books/ZopeBook/2_6Edition/SearchingZCatalo... There is evidently at least one instance of ZCatalog catalog in your zope site somewhere, named Catalog, because it's referred to by that name in the External Method you posted. So search for things with the id "Catalog".
Is there one Catalog for Employees, one for Students, etc.?
Unless I know what catalog is, I can't answer this question?
see above.
def extract_employees(self): catalog = self.path.to.where.the.catalog.lives.Catalog
example path?
yes, it was an example. I don't know where this Catalog is in your site. It's evidently in or above the folder that contains the External Method you posted.
e_id = employee.id
How do you know that there was id attribute?
I copied that from the external method you posted!
certainly not! for that kind of job, acquisition is your friend. You could have one external method that calls several functions, one for each kind of data to export (employees, etc.), much like you have now. Put the external method at the top of the application folder tree.
OK I put that external method in root folder then? What external method should look like? Like that example above?
something like that. I can't be more specific because I don't have your application in front of me.
I think you will eventually solve my problem.
Hmmm. I've actually given a lot more help than I planned on already. I don't mind helping but I am starting to wonder why you are being paid for this and I am not :) -- Paul Winkler http://www.slinkp.com Look! Up in the sky! It's PICO-CRANK IN YOUR BRAIN! (random hero from isometric.spaceninja.com)
Under Control_Panel/Products/School. There are classes for each of the object I want to export. Like Students, Employees. Their icons are rectangles with lines in them. Tooltip show Z Class for them. Now with this information how do I export data from these classes by installing just one script in root folder. This script should discover all field names dynamically and then create csv file containing all records. Is it possible now? --Sukhwinder SIngh ----- Original Message ----- From: "Paul Winkler" <pw_lists@slinkp.com> To: <zope@zope.org> Sent: Monday, June 30, 2003 1:13 PM Subject: Re: [Zope] Exporting zodb data (Urgent)
On Tue, Jul 01, 2003 at 12:04:11AM +0530, Sukhwinder Singh wrote:
The main "/" (root) folder of the application, under which all other files are stored shows a tooltip of "Ffolder w/Customizer support". (snip)
The objects I am talking about, like employees and students, are shown with specialist icons( Icon of a person sitting on a desk). What this means? What kind of objects these are?
OK, it appears from these statements that our earlier guesses were wrong. Your application is in fact built with some additional zope Products. I don't recognize icons by that description, I am guessing the Products were created specifically for this project.
When you point at an icon in the ZMI, you normally get a tooltip showing the meta_type of the Product. This identifies the product. You have a folder of type "Folder w/Customizer Support"; what about these person-at- desk icons? what tooltip do you get for those?
When you know the meta_type of the product, go to the zope root, click on Control Panel, then click on Product Management. There should be items there with similar names to what you see in the tooltips. You might even want to paste the whole list in a message here - we're familiar with the standard products, we could then guess which ones look like they were added for your application.
When you know which products are relevant to your needs, look at the little blue box icon next to them in the Product Management list. Most of the little blue boxes look "closed" (there is just a small little thing sticking up off the top of the box... don't ask me what that is supposed to represent!).
Do the icons for your products look like all the others, or are they clearly different - looking like an open box with the lid sticking off to the left?
Let us know which and we'll proceed from there.
When I click these objects (for example, employee) and check their properties, no property is defined.
OK, I was wrong about that, sorry. It's hard to guess how somebody built an application when you can't even see it :-)
But under these objects their are dtml methods, python scripts and other dtml files. Some external methods were added by earlier programmer under these folder/objects to export data from each of these objects (specialists). But I don't know how he determined attributes of these objects (like employee.id, employee.is_operator) etc.
These external methods in each specialist call a scipt like this to export data:
///////////////////////////////////////////////////////////////// def list_all_employees(self): self.REQUEST.RESPONSE.setHeader('Content-Type','text/csv') start_time=DateTime() container=self
ah I see, that's how "container" got into the External Method. I was wondering about that...
result=[] r="" q="" e_id="" e_name="" e_isop="" e_surname=""
employees=container.Catalog()
ok, this application DOES use ZCatalog. (At least, I am assuming that container.Catalog is an instance of ZCatalog). I wonder why the heck it's slow. That's not normal.
for employee in employees: e_name = employee.name (snip)
OK, you are not actually iterating over objects in the folder. You are iterating over catalog search results. Each search result contains information about an object - but it is not identical with the object. This means that in order for you to use an attribute of a result object, the catalog must be configured to keep this data about this object. This is done on the Metadata tab of the catalog instance.
So, go to one of these Catalogs, click on it, then click the Metadata tab. You should see that there is a Metadata entry for all the attributes that are accessed in the external method.
e_id = employee.id e_isop = "%s" % (employee.is_operator) e_surname = employee.surname q+=e_id+"#"+e_isop+"#"+e_name+"#"+e_surname+"\n"
return q
I wonder if there's a lot of stuff like this in the application. String concatenation is O(N**2) in python, and this script is potentially doing it a LOT depending on the size of the catalog results. It's much faster to build a list, and then join the list elements once at the end. List appends and string joins are O(N).
But as I said before, I can't actually tell if that's the cause for the slowness without profiling the app.
But problem is that he had to add these external methods under each of these object himself to export data.
certainly not! for that kind of job, acquisition is your friend. You could have one external method that calls several functions, one for each kind of data to export (employees, etc.), much like you have now. Put the external method at the top of the application folder tree.
Now, it sounds like for some reason you have multiple catalogs, though I'm not sure of this. Is there one Catalog for Employees, one for Students, etc.? If so, each function would look something like:
def extract_employees(self): catalog = self.path.to.where.the.catalog.lives.Catalog results = [] for emp in catalog(): e_id = employee.id ... etc ... results.append("%s#%s#%s#%s" % (e_id, e_isop, e_name, e_surname))
# now join the list elements with newlines return "\n".join(results)
Also how did he determine, whether there was an attribute "employee.is_operator" and names of other attributes?
He probably looked at the employee Product. We're getting there...
--
Paul Winkler http://www.slinkp.com Look! Up in the sky! It's UNDER-PEAR OF THE DEEP! (random hero from isometric.spaceninja.com)
_______________________________________________ Zope maillist - Zope@zope.org http://mail.zope.org/mailman/listinfo/zope ** No cross posts or HTML encoding! ** (Related lists - http://mail.zope.org/mailman/listinfo/zope-announce http://mail.zope.org/mailman/listinfo/zope-dev )
Under Control_Panel/Products/School. There are classes for each of the object I want to export. Like Students, Employees. Their icons are rectangles with lines in them. Tooltip show Z Class for them. Now with this information how do I export data from these classes by installing just one script in root folder. This script should discover all field names dynamically and then create csv file containing all records. Is it possible now? --Sukhwinder SIngh ----- Original Message ----- From: "Paul Winkler" <pw_lists@slinkp.com> To: <zope@zope.org> Sent: Monday, June 30, 2003 1:13 PM Subject: Re: [Zope] Exporting zodb data (Urgent)
On Tue, Jul 01, 2003 at 12:04:11AM +0530, Sukhwinder Singh wrote:
The main "/" (root) folder of the application, under which all other files are stored shows a tooltip of "Ffolder w/Customizer support". (snip)
The objects I am talking about, like employees and students, are shown with specialist icons( Icon of a person sitting on a desk). What this means? What kind of objects these are?
OK, it appears from these statements that our earlier guesses were wrong. Your application is in fact built with some additional zope Products. I don't recognize icons by that description, I am guessing the Products were created specifically for this project.
When you point at an icon in the ZMI, you normally get a tooltip showing the meta_type of the Product. This identifies the product. You have a folder of type "Folder w/Customizer Support"; what about these person-at- desk icons? what tooltip do you get for those?
When you know the meta_type of the product, go to the zope root, click on Control Panel, then click on Product Management. There should be items there with similar names to what you see in the tooltips. You might even want to paste the whole list in a message here - we're familiar with the standard products, we could then guess which ones look like they were added for your application.
When you know which products are relevant to your needs, look at the little blue box icon next to them in the Product Management list. Most of the little blue boxes look "closed" (there is just a small little thing sticking up off the top of the box... don't ask me what that is supposed to represent!).
Do the icons for your products look like all the others, or are they clearly different - looking like an open box with the lid sticking off to the left?
Let us know which and we'll proceed from there.
When I click these objects (for example, employee) and check their properties, no property is defined.
OK, I was wrong about that, sorry. It's hard to guess how somebody built an application when you can't even see it :-)
But under these objects their are dtml methods, python scripts and other dtml files. Some external methods were added by earlier programmer under these folder/objects to export data from each of these objects (specialists). But I don't know how he determined attributes of these objects (like employee.id, employee.is_operator) etc.
These external methods in each specialist call a scipt like this to export data:
///////////////////////////////////////////////////////////////// def list_all_employees(self): self.REQUEST.RESPONSE.setHeader('Content-Type','text/csv') start_time=DateTime() container=self
ah I see, that's how "container" got into the External Method. I was wondering about that...
result=[] r="" q="" e_id="" e_name="" e_isop="" e_surname=""
employees=container.Catalog()
ok, this application DOES use ZCatalog. (At least, I am assuming that container.Catalog is an instance of ZCatalog). I wonder why the heck it's slow. That's not normal.
for employee in employees: e_name = employee.name (snip)
OK, you are not actually iterating over objects in the folder. You are iterating over catalog search results. Each search result contains information about an object - but it is not identical with the object. This means that in order for you to use an attribute of a result object, the catalog must be configured to keep this data about this object. This is done on the Metadata tab of the catalog instance.
So, go to one of these Catalogs, click on it, then click the Metadata tab. You should see that there is a Metadata entry for all the attributes that are accessed in the external method.
e_id = employee.id e_isop = "%s" % (employee.is_operator) e_surname = employee.surname q+=e_id+"#"+e_isop+"#"+e_name+"#"+e_surname+"\n"
return q
I wonder if there's a lot of stuff like this in the application. String concatenation is O(N**2) in python, and this script is potentially doing it a LOT depending on the size of the catalog results. It's much faster to build a list, and then join the list elements once at the end. List appends and string joins are O(N).
But as I said before, I can't actually tell if that's the cause for the slowness without profiling the app.
But problem is that he had to add these external methods under each of these object himself to export data.
certainly not! for that kind of job, acquisition is your friend. You could have one external method that calls several functions, one for each kind of data to export (employees, etc.), much like you have now. Put the external method at the top of the application folder tree.
Now, it sounds like for some reason you have multiple catalogs, though I'm not sure of this. Is there one Catalog for Employees, one for Students, etc.? If so, each function would look something like:
def extract_employees(self): catalog = self.path.to.where.the.catalog.lives.Catalog results = [] for emp in catalog(): e_id = employee.id ... etc ... results.append("%s#%s#%s#%s" % (e_id, e_isop, e_name, e_surname))
# now join the list elements with newlines return "\n".join(results)
Also how did he determine, whether there was an attribute "employee.is_operator" and names of other attributes?
He probably looked at the employee Product. We're getting there...
--
Paul Winkler http://www.slinkp.com Look! Up in the sky! It's UNDER-PEAR OF THE DEEP! (random hero from isometric.spaceninja.com)
_______________________________________________ Zope maillist - Zope@zope.org http://mail.zope.org/mailman/listinfo/zope ** No cross posts or HTML encoding! ** (Related lists - http://mail.zope.org/mailman/listinfo/zope-announce http://mail.zope.org/mailman/listinfo/zope-dev )
Sukhwinder Singh wrote:
Under Control_Panel/Products/School.
There are classes for each of the object I want to export. Like Students, Employees. Their icons are rectangles with lines in them. Tooltip show Z Class for them.
Click on some ZClass. Than click on Property Sheets tab. Most probably you will find one or more property sheets there. Click on them You will see the list of properties defined for this ZClass. After you have list of all properties of all your ZClasses writing the script which will iterate over your objects is trivial. And note: using stand-alone ZODB just to get access to ZClass properties is slightly overkill, imho
On ti, 01.07.2003 at 22:49 +1100, Sergey Volobuev wrote:
Sukhwinder Singh wrote:
Under Control_Panel/Products/School.
There are classes for each of the object I want to export. Like Students, Employees. Their icons are rectangles with lines in them. Tooltip show Z Class for them.
Click on some ZClass. Than click on Property Sheets tab. Most probably you will find one or more property sheets there. Click on them You will see the list of properties defined for this ZClass. After you have list of all properties of all your ZClasses writing the script which will iterate over your objects is trivial.
And note: using stand-alone ZODB just to get access to ZClass properties is slightly overkill, imho
I have no idea what this is relating to, but maybe this will help. You don't even have to know the properties of your classes, to get them all. ----- for jojo in context.propertysheets.items(): if jojo[0] != 'webdav': print jojo[1].propertyItems() return printed ----- -- paavo. "joskus voi tää meno käydä ahdistavaksi kun on täällä muodostunut tavaksi muuttaa jokaisen elämän arvo rahaksi"
Sergey Volobuev wrote
Click on some ZClass. Than click on Property Sheets tab. Most probably you will find one or more property sheets there. Click on them You will see the list of properties defined for this ZClass. After you have list of all properties of all your ZClasses writing the script which will iterate over your objects is trivial.
Yes I have been able to find all properties and their data types. Thank you very much. The script I am using to extract data from one object is like this: def export_students(self): container = self self.REQUEST.RESPONSE.setHeader('Content-Type','text/csv') students=container.Catalog() studentsData = "" firstTime = 1 for student in students: o = student.getObject() addr = (o.address) address = "" for ad in addr: address += ad studentsData = student.id + ", " + student.name + ", " + student.surname + ", " + address + ", " + student.area + ", " + o.licence_type + ", " + o.age_range_text + ", " + o.phone + ", " + o.mobile + ", " + o.comments + ", " + o.reason_for_leaving + ", " + o.comment_on_leaving + ", " + o.meeting_date_text + ", "+ student.instructor_id + ", " + o.operator_id + ", " + o.start_date +"\r\n" return studentsData The above script is working. But I am not able to understand why I have to use two variables to access properties. First uses student object returned by "for student in students" and the second uses o object returned by student.getObject(). If I use o where I have used student, error results and vice versa. What is the difference getObject and "for student in students"? Why can't I use just student to retrieve all properties or just o? --Sukhwinder Singh
On Thu, Jul 03, 2003 at 10:39:06AM -0700, Sukhwinder Singh wrote:
But I am not able to understand why I have to use two variables to access properties.
First uses student object returned by "for student in students" and the second uses o object returned by student.getObject().
As I said before, "students" in your example is a set of zcatalog results. Each result contains information about an object in the ZODB, but it is not that object. So for each result, you are de-referencing it to get the original object. Each catalog result object only knows about the metadata that the catalog is configured to index. Another solution would be to reconfigure the catalog to keep track of all the information you are interested in; this would make the extraction run faster, at the expense of taking more time to update this catalog. But since you are just extracting data from an application that will be retired, that is probably not worth the effort. -- Paul Winkler http://www.slinkp.com Look! Up in the sky! It's PREDOMINANT FORNICATOR! (random hero from isometric.spaceninja.com)
When I click any of these classes, message "There are currently no items in methods" is displayed. Under this folder (/Control_Panel/Products/School) there are 2-3 other type of objects. These are: 1.Zope factory 2. Zope Permission 3. DTML Methods. DTML methods are named like "Employee_add (Employee constructor) " These methods contain code like this: <dtml-with "Employee.createInObjectManager(REQUEST['id'], REQUEST)"> <dtml-comment> You can add code that modifies the new instance here. For example, if you have a property sheet that you want to update from form values, you can call it here: <dtml-call "propertysheets.Basic.manage_editProperties( REQUEST)"> </dtml-comment> </dtml-with> <dtml-if DestinationURL> <dtml-call "RESPONSE.redirect( DestinationURL+'/manage_workspace')"> <dtml-else> <dtml-call "RESPONSE.redirect( URL2+'/manage_workspace')"> </dtml-if> --Sukhwinder Singh
On Fri, 2003-06-27 at 08:58, Sukhwinder Singh wrote:
Now client wants to write this application using php/mysql.
Their loss, I suppose.
Zodb uses fileStorage to store data. I have tried zope's xml export but that doesn't make any sense to me.
Yeah... that's not what it's for. The challenge is not *just* that Zope stores objects, but that these objects exist to produce dynamic content. "Exporting" your content would mean either taking a "snapshot" of what the site looks like at a given moment or recreating the dynamism of the site in another medium. If you just want a snapshot of the site, that's simple... open a CLI on the host machine and type: $ wget -r http://localhost:8080 Easy as pie. If you're looking to capture the logic of the site, the best answer I can give is to use wget to grab the HTML and recreate the logic from scratch. You're re-doing this in PHP anyway, the logic is bound to be different and if you don't know DTML or ZPT well, it won't help you much to have the original code. But if you must, you *can* extract the code behind objects by browsing them in the Zope Management Interface. Depending on how big a system this is, you may find it handy to script Zope using Python. Best of luck, Dylan
participants (7)
-
Dennis Allison -
Dieter Maurer -
Dylan Reinhardt -
Paavo Parkkinen -
Paul Winkler -
Sergey Volobuev -
Sukhwinder Singh