RE: URGENT: Can't start up zope
Ok, I've solved my own problem. I removed MailHost and modified this method in Product.py def _delObject(self,id): o = getattr(self, id) if hasattr(o, '_factories'): for factory in getattr(self, id)._factories(): factory._unregister() ProductFolder.inheritedAttribute('_delObject')(self, id) This allowed me to delete the strange object from the ProductManager in ContolPanel. Then when restarting with the MailHost product code put back everything was fine. I'd like to thank myself for the invaluable assistance I provided myself :) Now for my rant. How on earth did I get in this situation???? There are two serious problems here. 1) How on earth did an object get overridden by a completely different object. This might have something to do with object id's getting mixed up???? In any case its a really big bug and as I'm not sure when or how it happened I'm not sure how we can fix it. 2) It is a seriously problem if anything in the ODB can prevent zope starting. Since zope is only method of accessing the ODB then it can't be fixed (easily) unless Zope can be brought back up. It seems to me that much more care should be taken to find any unhandled exceptions in the zope init sequence. I disliked the idea of a putting all my data into some closed repository. I am completely left helpless if I want to rollback any changes in situations like this. There should be an intergration into a source control repository like CVS. Something that is file based so data can be manually edited. Also is there anyway to reduce the dependence of the main zope functionality on states kept inside the ODB? ie in this case is it necessary to keep persistant information on all the Products in the ODB? In short, Zope needs to be crash proof. How can we make this happen?
-----Original Message----- From: Jay, Dylan Sent: Saturday, June 12, 1999 13:11 To: 'zope@zope.org' Cc: 'support@digicool.com' Subject: Re: URGENT: Can't start up zope
-----Original Message----- From: Michel Pelletier Sent: Friday, June 11, 1999 2:42 PM To: 'djay@lucent.com'; zope@zope.org Cc: Support Subject: RE: URGENT: Can't start up zope.
-----Original Message----- From: Dylan Jay [mailto:djay@cse.unsw.edu.au] Sent: Friday, June 11, 1999 2:29 PM To: zope@zope.org Cc: support@digicool.com Subject: URGENT: Can't start up zope.
I was doing some external method work on our production server (silly boy) and something went very wrong. Every time I start zope it will crash. This is both as an NT service and using ZHTTPServer. ZHTTPServer gives a stack dump that mentions initializing products and factories and stuff (unfortunatly my office has a power outage currently so I can't the dump or the data.bbb until 5 hours or so). Using the bbb.py it reports no corruptions so I can only presume the database contains some configurations that screws up zope. To make matters worse our expensive backup solution doesn't seem to have backed up data.bbb.
Can you send us this traceback? Did a power outage occour while the server was running? Did the machine not get shut down properly and this this problem happened? Or did it just happen after a clean shutdown (ie, what was the state of the server before coming up the first time with this error).
No, It occured while testing an external method.
Heres the dump
C:\Program Files\Provision>..\python\python serve.py Traceback (innermost last): File "serve.py", line 100, in ? ZopeHTTPServer.main(args) File "C:\Program Files\Provision\ZopeHTTPServer\ZopeHTTPServer.py", line 498, in main start(module_file,host,port,threading,env) File "C:\Program Files\Provision\ZopeHTTPServer\ZopeHTTPServer.py", line 436, in start set_published_module(module_file,BoboRequestHandler,env) File "C:\Program Files\Provision\ZopeHTTPServer\ZopeHTTPServer.py", line 432, in set_published_module __import__(name) # to catch problem modules right away File "C:\Program Files\Provision\lib\python\Main.py", line 103, in ? Bobobase=OFS.Application.open_bobobase() File "C:\Program Files\Provision\lib\python\OFS\Application.py", line 371, in open_bobobase install_products(app) File "C:\Program Files\Provision\lib\python\OFS\Application.py", line 486, in install_products App.Product.initializeProduct(product, product_name, package_dir, app) File "C:\Program Files\Provision\lib\python\App\Product.py", line 428, in init ializeProduct if old is not None: products._delObject(name) File "C:\Program Files\Provision\lib\python\App\Product.py", line 145, in _del Object for factory in getattr(self, id)._factories(): factory._unregister() AttributeError: _factories
Now that I've had some time to debug it a little, what I discover is this. The above crash happens while trying to initialize the MailHost object. The reason is that getattr(self,id) in the call above is actually a DTML object not a Factory object. Somehow a piece of code got whacked in the wrong place. I can now get the site up by removing the MailHost product. However mail is an extermely important part of the site in that no new user can register without it. When a shut it down and add MailHost back in I get the same problem. What I need to do is somehow take out what is posing as the MailHost Product. I have no idea how to do this.
I will resend this mail to digicool.com with my database
-Michel
What I need to know are a) What the likely causes of a something like I'm describing? I realize I havn't given much to go on, but ANY information you can give me at this point would very very appreciated. b) What tools do I need to modify the data.bbb. I can convert it xml ok but it seems there's no way to convert it back. Is there any tool where I can just roll back the last few transactions? (This may not work actually as it was probably the external method that did the damage).
In case its relevent I guess I should include some detail of the external method I was working on. It was a method to determine if a user had permission to view an object. It looked at a users roles and the roles for the view permission. It would recurse with the parent if the object aquired roles.
Thanks in advance for any ideas anyone has.
Dylan. djay@lucent.com
"Jay, Dylan" wrote:
Now for my rant. How on earth did I get in this situation???? There are two serious problems here. 1) How on earth did an object get overridden by a completely different object.
An external methd could do it. External methods are dangerous in many ways. I'd like to hear more about the one you were debugging. I'd also like to see your database so I can try to figure out what happened. (Please *just* send your database to support@digicool.com, not the entire Zope list.)
This might have something to do with object id's getting mixed up????
I don't think so.
In any case its a really big bug and as I'm not sure when or how it happened I'm not sure how we can fix it.
So we'd like to figure it out with you.
2) It is a seriously problem if anything in the ODB can prevent zope starting. Since zope is only method of accessing the ODB then it can't be fixed (easily) unless Zope can be brought back up.
Zope isn't exactly the only way to access the ODB, but we certainly don't provide any other easy ways. One thing we should do is provide a script for doing undo from the command line so that fatal transactions can be undone.
It seems to me that much more care should be taken to find any unhandled exceptions in the zope init sequence. I disliked the idea of a putting all my data into some closed repository. I am completely left helpless if I want to rollback any changes in situations like this.
This is a key point. You should be able to roll back changes even if the database is down. I'll provide a tool for doing this next week.
There should be an intergration into a source control repository like CVS. Something that is file based so data can be manually edited.
I don't think that this has anything to do with CVS. It would be nice to be able to inspect and edit the database with a text editor. We are working on a database<-->XML tool that will allow exactly that.
Also is there anyway to reduce the dependence of the main zope functionality on states kept inside the ODB?
Yes, up to a point.
ie in this case is it necessary to keep persistant information on all the Products in the ODB?
Yes, but we can do a better job of reacting to problems. We are working on this. For example, there is a change in Product.py in the 2.0 source that is very similar to the fix you figured out yourself.
In short, Zope needs to be crash proof. How can we make this happen?
- We can do a better job at catching and handling errors in critical parts of Zope (like initialization). - We may want to consider disallowing external methods, or giving much sterner warnings about their use. - We can provide better tools for managing or manipulating the database when Zope does not want to come up. For example, we should provide a tool for undoing changes from the command line. Jim -- Jim Fulton mailto:jim@digicool.com Technical Director (888) 344-4332 Python Powered! Digital Creations http://www.digicool.com http://www.python.org Under US Code Title 47, Sec.227(b)(1)(C), Sec.227(a)(2)(B) This email address may not be added to any commercial mail list with out my permission. Violation of my privacy with advertising or SPAM will result in a suit for a MINIMUM of $500 damages/incident, $1500 for repeats.
"Jay, Dylan" wrote:
Also is there anyway to reduce the dependence of the main zope functionality on states kept inside the ODB? ie in this case is it necessary to keep
Jim made a very good reply, I'll add in a couple more blurbs as well. First, you're right, we need to continue doing everything we can to improve the reliability as well as provide tools for recovery. I can safely say Jim et al. are _keenly_ engaged on this. But like security, reliability isn't one silver bullet, it is a collection of options. Thus, we need you and others to keep brainstorming more ways to get there. As for the database itself, IMO there are two issues involved: storage of the objects and the objects themselves. In the first case, the current Zope database is responsible for managing records in a file. If something goes wrong, records might get mangled. Zope2 will let you have replacement options with other storage managers you might trust more, like a gdbm/bsddb file, a relational database, etc. Zope2 also has a more resilient database format, as well as an option to partition your object system into multiple storages (thus minimizing any damage)*. In the second case, at the end of the day the object data will be in a Python pickle. This means that the object data is still intimately tied to the application -- which is at the heart of object oriented programming. Of course the XML import/export of the database helps mitigate this. Thus my question: How much would allowing record storage and management in something you might trust (e.g. bsddb) alleviate the black-box feeling? Needless to say, the gang here at DC, particularly Jim, are mighty interested in what you think on the subject. * Perhaps the top-level folder in Zope2 should be in its own storage, so that changes to sub-subobjects won't be in danger of corrupting it? Just a thought? --Paul
At 05:33 13-6-99 , Paul Everitt wrote:
Thus my question: How much would allowing record storage and management in something you might trust (e.g. bsddb) alleviate the black-box feeling?
The fact that Zope 2 can store in a RDMS definitely alleviates that feeling for my colleagues, who are still somewhat Zope illiterate. But we will need tools to read the information stored in that DB, so it is accessible from other applications. We have been working with Informix Universal DB for some time here, and they are intrigued by a Web Application Platform that is object oriented, in as such that it stores information that it knows how to handle. I personally cannot bide with the informix way of connecting it's data to the web (no more MIval!). If the Zope data stored in a RDMS can be accessed by other apps, this would be benificial to promoting Zope to people that think in RDMS's as an end to all Web Applications, even though I personally don't see a direct use for such access. -- Martijn Pieters, Web Developer | Antraciet http://www.antraciet.nl | Tel: +31-35-7502100 Fax: +31-35-7502111 | mailto:mj@antraciet.nl http://www.antraciet.nl/~mj | PGP: http://wwwkeys.nl.pgp.net:11371/pks/lookup?op=get&search=0xA8A32149 ------------------------------------------
Martijn Pieters wrote:
If the Zope data stored in a RDMS can be accessed by other apps, this would be benificial to promoting Zope to people that think in RDMS's as an end to all Web Applications, even though I personally don't see a direct use for such access.
On a technical level, since the data in the RDBMS record is a pickled Python instance, it would be basically unusable to other applications. You'd gain a "standard" for record management, but not one for data management. On an abstract level, the statement "if Zope data can be accessed by other applications" seems to be a pretty clear violation of encapsulation, but that's just the object zealot party line. :^) --Paul
At 10:39 14-6-99 , Paul Everitt wrote:
On a technical level, since the data in the RDBMS record is a pickled Python instance, it would be basically unusable to other applications. You'd gain a "standard" for record management, but not one for data management.
On an abstract level, the statement "if Zope data can be accessed by other applications" seems to be a pretty clear violation of encapsulation, but that's just the object zealot party line. :^)
Yeah right, as if python enforces this =) I realized that the RDMS will hold pickles, that's why there might be the need for tools to unpickle them. Like Zope, haha. I explained this to my colleagues. I thik they still feel uncomfertable with the fact that the data is of use only to Zope. -- Martijn Pieters, Web Developer | Antraciet http://www.antraciet.nl | Tel: +31-35-7502100 Fax: +31-35-7502111 | mailto:mj@antraciet.nl http://www.antraciet.nl/~mj | PGP: http://wwwkeys.nl.pgp.net:11371/pks/lookup?op=get&search=0xA8A32149 ------------------------------------------
Paul, Please excuse this belated response to your question.
Thus my question: How much would allowing record storage and management in something you might trust (e.g. bsddb) alleviate the black-box feeling?
Your question was made in the context of the current and future forms of Zope persistent storage.
From my POV, most stores are black boxes, whether they are SQL servers, object databases, or some hybrid.
Although each type of database has its particular uses and relevance, the common thread for me has always been can I reliably conduct the following operations: 1. Populate the database. For object databases, the originating data may be some external non-object source, so it may be necessary to populate in stages. (i.e. First drop in the low-level, bottom-of-the-tree stuff, then work my way up the branches, wiring whatever linkages along the way. It's possible this must be done in multiple passes -- a staged build.) 2. Replicate the database. Given the database has been populated with information, I want to export it to a format of my choice(*): flat file, pickle, whatever; copy it to a new location a re-populate a fresh database from scratch. (*) "my choice" implies I'm responsible for implementing it. 3. Perform maintenance ("fix") the database. Despite best efforts, occasionally some corruption may occur, possibly as the result of a poorly designed user application. Items 1 & 2 imply the possibility of fixing these problems, independent of the application code. 4. Schema evolution. This tends to be my biggest hassle with object databases, especially in maintaining production systems. Pickles are pleasantly permissive, but I'm mostly comfortable with systems that I can easily populate from scratch. I've been remiss in keeping up with Jim's work, although I did review his description of BoboPOS3. [Slightly off-topic, but I'm also wondering if anybody has considered using the Zope database for any non-Zope projects.] Best regards, Jeff Bauer Rubicon, Inc.
participants (5)
-
Jay, Dylan -
Jeff Bauer -
Jim Fulton -
Martijn Pieters -
Paul Everitt