Re: [Zope] Exporting zodb data (Urgent)
Their loss, I suppose.
They want to it because zope has become very very slow.
If you're looking to capture the logic of the site, the best answer I can give is to use wget to grab the HTML and recreate the logic from scratch. You're re-doing this in PHP anyway, the logic is bound to be different and if you don't know DTML or ZPT well, it won't help you much to have the original code.
I have code I have everything. Site is working in zope on my development system. I have tried everything I could find information about. They want to create exact copy of that already working system in php. Main problem is exporting original data. They can't loose it.
The challenge is not *just* that Zope stores objects, but that these objects exist to produce dynamic content. "Exporting" your content would mean either taking a "snapshot" of what the site looks like at a given moment or recreating the dynamism of the site in another medium.
Yes that is what I want, a snapshot. I am working on windows system. What is wget? What does it do? Will it work on Windows. Question is how do I know which objects store data and how many fields (attributes) are there and their data types etc. in each of those objects. And then how to extract data?
--Sukhwinder Singh ----- Original Message ----- From: "Dylan Reinhardt" <zope@dylanreinhardt.com> To: "Sukhwinder Singh" <ssruprai@hotmail.com> Cc: <zope@zope.org> Sent: Saturday, June 28, 2003 4:36 AM Subject: Re: [Zope] Exporting zodb data (Urgent)
On Fri, 2003-06-27 at 08:58, Sukhwinder Singh wrote:
Now client wants to write this application using php/mysql.
Their loss, I suppose.
Zodb uses fileStorage to store data. I have tried zope's xml export but that doesn't make any sense to me.
Yeah... that's not what it's for.
The challenge is not *just* that Zope stores objects, but that these objects exist to produce dynamic content. "Exporting" your content would mean either taking a "snapshot" of what the site looks like at a given moment or recreating the dynamism of the site in another medium.
If you just want a snapshot of the site, that's simple... open a CLI on the host machine and type:
$ wget -r http://localhost:8080
Easy as pie.
If you're looking to capture the logic of the site, the best answer I can give is to use wget to grab the HTML and recreate the logic from scratch. You're re-doing this in PHP anyway, the logic is bound to be different and if you don't know DTML or ZPT well, it won't help you much to have the original code.
But if you must, you *can* extract the code behind objects by browsing them in the Zope Management Interface. Depending on how big a system this is, you may find it handy to script Zope using Python.
Best of luck,
Dylan
On Sun, Jun 29, 2003 at 12:46:31AM +0530, Sukhwinder Singh wrote:
Their loss, I suppose.
They want to it because zope has become very very slow.
what makes them think php will be faster? zope performs pretty well with most jobs except serving large files (at least 10 MB). there are ways to speed up zope: - various kinds of cacheing to reduce load - using a profiler to find hot spots and optimize them
Yes that is what I want, a snapshot. I am working on windows system. What is wget? What does it do? Will it work on Windows.
it's a downloading & mirroring tool for unix. don't know if it runs on windows, there are probably similar GUI tools available for windows.
Question is how do I know which objects store data and how many fields (attributes) are there and their data types etc. in each of those objects. And then how to extract data?
there is no general-purpose way to do this for all zope objects. You haven't told us anything about what kind of objects they used in the zope application to store data. DTML methods or documents? ZPT? CMF content types? some 3rd-party downloaded product? Custom python products? Custom zclasses? If you don't know any of this, and the customer doesn't either, and you don't have sufficient zope experience to figure it out, then they've given you an unreasonable burden. An experienced zope / python developer could get this job done pretty quickly with a bit of python scripting. -- Paul Winkler http://www.slinkp.com Look! Up in the sky! It's MILDLY-AMERICAN SHROOM BAT! (random hero from isometric.spaceninja.com)
Hello, Thanks for replying. It is a school application which stored information about employees, students, trainings etc. What I know is they have used dtml methods, python scripts and folders to organize their app. But not custom products. I have checked those objects using /manage. And most of these are methods and other dtml files. I have also checked some of the dtml files.and python scripts. These files get data using something like: for employee in container.employees print employee.name These files/scripts/methods are in their respective folders. For example, a script which displays employee data is in employees folder and script which exports student data is in students folder. But when I try to use this method of using for each * in container.students or container.employees outside these folders (in the root folder) error is displayed like container is not defined etc. Is container something built in or their custom class? There is one more problem client is on the other part of the globe. My boss cannot lose this project now. So I must do it. While I am trying to figure out how to export data they are using the old system at this time. So they are updating the database. I want to provide them a way to export complete data by just on click. If that is not possible then I want to provide them an external method or python script which they install in root folder and when accessed exports data to mysql database. If there is no other solution than learning zope then tell me what are the things I should learn about only to export data. I have downloaded zope guide and developer guide and admin guide. What other resource I should check? Time is problem for me. --Sukhwinder Singh ----- Original Message ----- From: "Paul Winkler" <pw_lists@slinkp.com> To: "Sukhwinder Singh" <ssruprai@hotmail.com> Cc: <zope@zope.org> Sent: Sunday, June 29, 2003 1:22 AM Subject: Re: [Zope] Exporting zodb data (Urgent)
On Sun, Jun 29, 2003 at 12:46:31AM +0530, Sukhwinder Singh wrote:
Their loss, I suppose.
They want to it because zope has become very very slow.
what makes them think php will be faster? zope performs pretty well with most jobs except serving large files (at least 10 MB).
there are ways to speed up zope: - various kinds of cacheing to reduce load - using a profiler to find hot spots and optimize them
Yes that is what I want, a snapshot. I am working on windows system. What is wget? What does it do? Will it work on Windows.
it's a downloading & mirroring tool for unix. don't know if it runs on windows, there are probably similar GUI tools available for windows.
Question is how do I know which objects store data and how many fields (attributes) are there and their data types etc. in each of those objects. And then how to extract data?
there is no general-purpose way to do this for all zope objects. You haven't told us anything about what kind of objects they used in the zope application to store data. DTML methods or documents? ZPT? CMF content types? some 3rd-party downloaded product? Custom python products? Custom zclasses?
If you don't know any of this, and the customer doesn't either, and you don't have sufficient zope experience to figure it out, then they've given you an unreasonable burden. An experienced zope / python developer could get this job done pretty quickly with a bit of python scripting.
--
Paul Winkler http://www.slinkp.com Look! Up in the sky! It's MILDLY-AMERICAN SHROOM BAT! (random hero from isometric.spaceninja.com)
On Sun, Jun 29, 2003 at 02:20:45AM +0530, Sukhwinder Singh wrote:
Hello,
Thanks for replying.
It is a school application which stored information about employees, students, trainings etc. What I know is they have used dtml methods, python scripts and folders to organize their app. But not custom products. I have checked those objects using /manage. And most of these are methods and other dtml files. I have also checked some of the dtml files.and python scripts.
I'm still wondering why this app is slow. I would hazard a guess that the implementation is not adequate to the number of objects. E.g. there might be something like for foo in some_folder.objectItems(): do something with a property of foo ... which is fine when some_folder contains a small number of objects, but when it grows to thousands, this will be very inefficient. Even worse would be using zope_find() which is really meant to be a convenience for developers and administrators, not for application code. Such an application should be redesigned to use ZCatalog which is very fast. Of course this is only a guess, and optimization should NEVER be based on guesses, that is what profiling is for. I still don't understand how you are expected to do this job. To make an analogy, if I needed to extract a large data set from a MySQL database, I wouldn't give the task to someone who had never used SQL before. If I wanted to re-implement a Smalltalk application in Java, I wouldn't hire a java developer who has never used smalltalk. No wonder you're finding this difficult. But I guess it's too late, the boss has dictated PHP and you are stuck with the task. So I'll try to give you what you want, no matter how wrong I think it is :)
These files get data using something like:
for employee in container.employees print employee.name
well, i'm still only guessing what the real code is, since that doesn't look like anything that really works. Since you say there's nothing but folders, python scripts and DTML, I guess that maybe there is a DTML Document that represents each employee, and I further guess that they are using Properties to store data about each employee. Mind you, this is only a guess. You could verify this by looking at the Properties tab on some of the objects that represent employees.
These files/scripts/methods are in their respective folders. For example, a script which displays employee data is in employees folder and script which exports student data is in students folder.
But when I try to use this method of using for each * in container.students or container.employees outside these folders (in the root folder) error is displayed like container is not defined etc. Is container something built in or their custom class?
"container" is a bound name that's always available in Python Script (and in Page Templates which you don't seem to be using). It refers to the container (usually a Folder) in which the Script object lives. This is described in the Advanced Scripting chapter of the online Zope Book. The "container" name is NOT bound in an External Method, nor is it bound in DTML. In an External Method, you can use "self" instead (if self is the first parameter of the function definition bound to the external method). In DTML, it's typical to just ask for the name without a qualifying namespace like "container". DTML then automagically looks in many places for this name.
If there is no other solution than learning zope then tell me what are the things I should learn about only to export data. I have downloaded zope guide
Do you mean the Zope Book?
and developer guide
Probably not much help to you, it's for developing add-on Products.
and admin guide.
Dead project as far as I know, and probably not relevant anyway. You might find this chapter of the online Zope Book very useful: http://www.zope.org/Documentation/Books/ZopeBook/2_6Edition/ScriptingZope.st... I suspect the section on "The Zope API" will be particularly useful, once you understand the differences between Scripts and External Methods etc. That chapter builds on concepts that are described in earlier chapters: http://www.zope.org/Documentation/Books/ZopeBook/2_6Edition/UsingZope.stx (introduces Properties, among other things) http://www.zope.org/Documentation/Books/ZopeBook/2_6Edition/Acquisition.stx (introduces Acquisition, very important) http://www.zope.org/Documentation/Books/ZopeBook/2_6Edition/DTML.stx (basic DTML) http://www.zope.org/Documentation/Books/ZopeBook/2_6Edition/AppendixE.stx (describes how names are looked up in dtml-var expressions) -- Paul Winkler http://www.slinkp.com Look! Up in the sky! It's FJUK- XENON MEGA! (random hero from isometric.spaceninja.com)
I share Paul's conclusions. If this is anything like the normal school application which has grown, porting to PHP and maintaining the same functionality is a non-trivial task. Zope and PHP are very differnt. The best approach would be to simply improve the performance of the current system--you'll have to learn to Zope system, but if you end up doing the port you'll have to do that anyhow. My guess is that you have a system where a little analysis and a little thought can provide enormous performance gains. On Sun, 29 Jun 2003, Paul Winkler wrote:
On Sun, Jun 29, 2003 at 02:20:45AM +0530, Sukhwinder Singh wrote:
Hello,
Thanks for replying.
It is a school application which stored information about employees, students, trainings etc. What I know is they have used dtml methods, python scripts and folders to organize their app. But not custom products. I have checked those objects using /manage. And most of these are methods and other dtml files. I have also checked some of the dtml files.and python scripts.
I'm still wondering why this app is slow. I would hazard a guess that the implementation is not adequate to the number of objects. E.g. there might be something like
for foo in some_folder.objectItems(): do something with a property of foo
... which is fine when some_folder contains a small number of objects, but when it grows to thousands, this will be very inefficient. Even worse would be using zope_find() which is really meant to be a convenience for developers and administrators, not for application code.
Such an application should be redesigned to use ZCatalog which is very fast.
Of course this is only a guess, and optimization should NEVER be based on guesses, that is what profiling is for.
I still don't understand how you are expected to do this job. To make an analogy, if I needed to extract a large data set from a MySQL database, I wouldn't give the task to someone who had never used SQL before. If I wanted to re-implement a Smalltalk application in Java, I wouldn't hire a java developer who has never used smalltalk. No wonder you're finding this difficult.
But I guess it's too late, the boss has dictated PHP and you are stuck with the task. So I'll try to give you what you want, no matter how wrong I think it is :)
These files get data using something like:
for employee in container.employees print employee.name
well, i'm still only guessing what the real code is, since that doesn't look like anything that really works.
Since you say there's nothing but folders, python scripts and DTML, I guess that maybe there is a DTML Document that represents each employee, and I further guess that they are using Properties to store data about each employee. Mind you, this is only a guess. You could verify this by looking at the Properties tab on some of the objects that represent employees.
These files/scripts/methods are in their respective folders. For example, a script which displays employee data is in employees folder and script which exports student data is in students folder.
But when I try to use this method of using for each * in container.students or container.employees outside these folders (in the root folder) error is displayed like container is not defined etc. Is container something built in or their custom class?
"container" is a bound name that's always available in Python Script (and in Page Templates which you don't seem to be using). It refers to the container (usually a Folder) in which the Script object lives. This is described in the Advanced Scripting chapter of the online Zope Book.
The "container" name is NOT bound in an External Method, nor is it bound in DTML. In an External Method, you can use "self" instead (if self is the first parameter of the function definition bound to the external method). In DTML, it's typical to just ask for the name without a qualifying namespace like "container". DTML then automagically looks in many places for this name.
If there is no other solution than learning zope then tell me what are the things I should learn about only to export data. I have downloaded zope guide
Do you mean the Zope Book?
and developer guide
Probably not much help to you, it's for developing add-on Products.
and admin guide.
Dead project as far as I know, and probably not relevant anyway.
You might find this chapter of the online Zope Book very useful: http://www.zope.org/Documentation/Books/ZopeBook/2_6Edition/ScriptingZope.st... I suspect the section on "The Zope API" will be particularly useful, once you understand the differences between Scripts and External Methods etc.
That chapter builds on concepts that are described in earlier chapters:
http://www.zope.org/Documentation/Books/ZopeBook/2_6Edition/UsingZope.stx (introduces Properties, among other things)
http://www.zope.org/Documentation/Books/ZopeBook/2_6Edition/Acquisition.stx (introduces Acquisition, very important)
http://www.zope.org/Documentation/Books/ZopeBook/2_6Edition/DTML.stx (basic DTML)
http://www.zope.org/Documentation/Books/ZopeBook/2_6Edition/AppendixE.stx (describes how names are looked up in dtml-var expressions)
--
Paul Winkler http://www.slinkp.com Look! Up in the sky! It's FJUK- XENON MEGA! (random hero from isometric.spaceninja.com)
_______________________________________________ Zope maillist - Zope@zope.org http://mail.zope.org/mailman/listinfo/zope ** No cross posts or HTML encoding! ** (Related lists - http://mail.zope.org/mailman/listinfo/zope-announce http://mail.zope.org/mailman/listinfo/zope-dev )
Do you mean the Zope Book? Yes.
Thank you very much for helping me. Now I'll try to do it myself and explore the application from start to end completely. I think I cannot do much without learning zope. But I still have one question. If I learn about zope, will I be able to know what objects etc. store data by examining the application? Thanks again. Sukhwinder Singh
Sukhwinder Singh wrote at 2003-6-29 02:20 +0530:
These files get data using something like:
for employee in container.employees print employee.name
Go to the "Properties" tab of these objects. Do you see there the relevant properties?
... But when I try to use this method of using for each * in container.students or container.employees outside these folders (in the root folder) error is displayed like container is not defined etc. Is container something built in or their custom class?
You must use the "ObjectManager" API. Dieter
On Sat, 2003-06-28 at 12:16, Sukhwinder Singh wrote:
Their loss, I suppose.
They want to it because zope has become very very slow.
Let me guess... they haven't packed the database since the original developer left the company? Zope scales quite well, in my experience... I'd wager whatever problem you're having is probably some easily-fixed configuration or maintenance issue. It may be too late to argue that point, but I think you'd find a lot more help being offered if the client were interested in fixing the problem instead of blaming the software. Not your fault, probably...
Yes that is what I want, a snapshot. I am working on windows system. What is wget? What does it do? Will it work on Windows.
It is native to the Unix world, but available on Windoze as well: http://space.tin.it/computer/hherold/
Question is how do I know which objects store data and how many fields (attributes) are there and their data types etc. in each of those objects. And then how to extract data?
You'll have to inspect the objects... but that won't help you much unless you have a decent idea what you're looking at. But that may be irrelevant. A recursive wget should capture any data that is hyperlink accessible. Give it a try and see how close it comes to what you need. HTH, Dylan
Dylan Reinhardt wrote at 2003-6-28 18:48 -0700:
On Sat, 2003-06-28 at 12:16, Sukhwinder Singh wrote:
Their loss, I suppose.
They want to it because zope has become very very slow.
Let me guess... they haven't packed the database since the original developer left the company?
But this should not slow down the operation (while is may slow down startup). Dieter
Hi, Dieter Maurer wrote:
Dylan Reinhardt wrote at 2003-6-28 18:48 -0700:
On Sat, 2003-06-28 at 12:16, Sukhwinder Singh wrote:
Their loss, I suppose.
They want to it because zope has become very very slow.
Let me guess... they haven't packed the database since the original developer left the company?
But this should not slow down the operation (while is may slow down startup).
Sure, but another thing can happen: they packed millions of images in a simple folderobject, or have some other objects filling up and interating over them each time a page will be requested (such as news items or such) instead of using a ZCatalog. This can in fact slow down the operation. Regards Tino
On Sun, 2003-06-29 at 11:12, Dieter Maurer wrote:
Dylan Reinhardt wrote at 2003-6-28 18:48 -0700:
On Sat, 2003-06-28 at 12:16, Sukhwinder Singh wrote:
Their loss, I suppose.
They want to it because zope has become very very slow.
Let me guess... they haven't packed the database since the original developer left the company?
But this should not slow down the operation (while is may slow down startup).
Shouldn't... but in Windoze you could easily fill up enough of the drive to seriously degrade the entire system's performance. Most Windoze installs are still only one partition. But as the OP noted, packing the database made no difference. :-) Dylan
Let me guess... they haven't packed the database since the original developer left the company?
Zope scales quite well, in my experience... I'd wager whatever problem you're having is probably some easily-fixed configuration or maintenance issue. It may be too late to argue that point, but I think you'd find a lot more help being offered if the client were interested in fixing the problem instead of blaming the software. Not your fault, probably...
This application provides the option to pack the database and I have packed it myself too. But it is, still, too slow. Earlier programmer, who originally accepted this project to port this app to php/mysql, did extract data form zodb and has already written the same application in php. Speed and performance of the php-mysql is much much better. I am not blaming zope as I don't know much about it. But as I had to add more options to this application, I have to extract that data again. Earlier programmer, somehow extracted the data for himself only to write application in php. He did'nt write any script for client to export that data themself. But I need to provide client this option of exporting because they are using the system at this time. --Sukhwinder Singh
participants (6)
-
Dennis Allison -
Dieter Maurer -
Dylan Reinhardt -
Paul Winkler -
Sukhwinder Singh -
Tino Wildenhain