I've been struggling with this one non-stop since last night. I work at an ISP, we keep our customer records in zope. Somehow, i seem to have lost records for nearly 1/4 of our customers. I've tried deleting the index, packing the DB, exporting and then importing into a fresh Data.fs, using fsrecover.py i've also tried fsrefs.py, which tells me that i'm missing a bunch of objects. The size of my Data.fs is prettymuch unchanged, which might suggest that the data is still there, but how do i get to it? What can I do that I haven't tried already, or, what have i messed up that doing again will make all my worries go away. any help would be very much appreciated. thanks stefan
oh yea, and deleteing the objects in question is really not the preferred solution, because i would really rather not have to reconstruct the data from other means. thank you in advance for imparting your vast wisdom to (on?) me. stefan On Sunday, Mar 23, 2003, at 15:23 US/Eastern, Stefan A. Nagey wrote:
I've been struggling with this one non-stop since last night.
I work at an ISP, we keep our customer records in zope.
Somehow, i seem to have lost records for nearly 1/4 of our customers.
I've tried deleting the index, packing the DB, exporting and then importing into a fresh Data.fs, using fsrecover.py
i've also tried fsrefs.py, which tells me that i'm missing a bunch of objects.
The size of my Data.fs is prettymuch unchanged, which might suggest that the data is still there, but how do i get to it?
What can I do that I haven't tried already, or, what have i messed up that doing again will make all my worries go away.
any help would be very much appreciated.
thanks
stefan
_______________________________________________ Zope maillist - Zope@zope.org http://mail.zope.org/mailman/listinfo/zope ** No cross posts or HTML encoding! ** (Related lists - http://mail.zope.org/mailman/listinfo/zope-announce http://mail.zope.org/mailman/listinfo/zope-dev )
Stefan A. Nagey GSM: +1 (202) 415-0163
Hey Stefan, What version of Zope/Python you using? I have been trying to get rid of a POSKeyError for two weeks. BZ On Sunday, March 23, 2003, at 03:27 PM, Stefan A. Nagey wrote:
oh yea, and deleteing the objects in question is really not the preferred solution, because i would really rather not have to reconstruct the data from other means.
thank you in advance for imparting your vast wisdom to (on?) me.
stefan
On Sunday, Mar 23, 2003, at 15:23 US/Eastern, Stefan A. Nagey wrote:
I've been struggling with this one non-stop since last night.
I work at an ISP, we keep our customer records in zope.
Somehow, i seem to have lost records for nearly 1/4 of our customers.
I've tried deleting the index, packing the DB, exporting and then importing into a fresh Data.fs, using fsrecover.py
i've also tried fsrefs.py, which tells me that i'm missing a bunch of objects.
The size of my Data.fs is prettymuch unchanged, which might suggest that the data is still there, but how do i get to it?
What can I do that I haven't tried already, or, what have i messed up that doing again will make all my worries go away.
any help would be very much appreciated.
thanks
stefan
_______________________________________________ Zope maillist - Zope@zope.org http://mail.zope.org/mailman/listinfo/zope ** No cross posts or HTML encoding! ** (Related lists - http://mail.zope.org/mailman/listinfo/zope-announce http://mail.zope.org/mailman/listinfo/zope-dev )
Stefan A. Nagey GSM: +1 (202) 415-0163
_______________________________________________ Zope maillist - Zope@zope.org http://mail.zope.org/mailman/listinfo/zope ** No cross posts or HTML encoding! ** (Related lists - http://mail.zope.org/mailman/listinfo/zope-announce http://mail.zope.org/mailman/listinfo/zope-dev )
BZ- yea, i saw all the stuff that ppl have suggested to you, it hasn't seemed to help any. fsrefs.py seems to think that the objects are missing, but, my Data.fs hasn't changed in size, so i would tend to think that they're still hiding in there somewhere. running: Python 2.2.2, Zope 2.5.1 on linux 2.4.20 on an ext3 partition over a raid1 array (using software raid/md) stefan On Sunday, Mar 23, 2003, at 15:47 US/Eastern, BZ wrote:
Hey Stefan,
What version of Zope/Python you using? I have been trying to get rid of a POSKeyError for two weeks.
BZ
On Sunday, March 23, 2003, at 03:27 PM, Stefan A. Nagey wrote:
oh yea, and deleteing the objects in question is really not the preferred solution, because i would really rather not have to reconstruct the data from other means.
thank you in advance for imparting your vast wisdom to (on?) me.
stefan
On Sunday, Mar 23, 2003, at 15:23 US/Eastern, Stefan A. Nagey wrote:
I've been struggling with this one non-stop since last night.
I work at an ISP, we keep our customer records in zope.
Somehow, i seem to have lost records for nearly 1/4 of our customers.
I've tried deleting the index, packing the DB, exporting and then importing into a fresh Data.fs, using fsrecover.py
i've also tried fsrefs.py, which tells me that i'm missing a bunch of objects.
The size of my Data.fs is prettymuch unchanged, which might suggest that the data is still there, but how do i get to it?
What can I do that I haven't tried already, or, what have i messed up that doing again will make all my worries go away.
any help would be very much appreciated.
thanks
stefan
_______________________________________________ Zope maillist - Zope@zope.org http://mail.zope.org/mailman/listinfo/zope ** No cross posts or HTML encoding! ** (Related lists - http://mail.zope.org/mailman/listinfo/zope-announce http://mail.zope.org/mailman/listinfo/zope-dev )
Stefan A. Nagey GSM: +1 (202) 415-0163
_______________________________________________ Zope maillist - Zope@zope.org http://mail.zope.org/mailman/listinfo/zope ** No cross posts or HTML encoding! ** (Related lists - http://mail.zope.org/mailman/listinfo/zope-announce http://mail.zope.org/mailman/listinfo/zope-dev )
Stefan A. Nagey GSM: +1 (202) 415-0163
On Sun, Mar 23, 2003 at 03:50:50PM -0500, Stefan A. Nagey wrote:
BZ-
yea, i saw all the stuff that ppl have suggested to you, it hasn't seemed to help any.
fsrefs.py seems to think that the objects are missing, but, my Data.fs hasn't changed in size, so i would tend to think that they're still hiding in there somewhere.
running: Python 2.2.2, Zope 2.5.1
That's a version mismatch. Zope 2.5.1 is not officially approved with Python 2.2.2 by Zope Corp. It *should* work but has not been widely tested. The only recommended version is 2.1.3. I have no idea if this is related to your problem... --PW -- Paul Winkler http://www.slinkp.com Look! Up in the sky! It's ALLEGORICAL LATONIC CAPTAIN! (random hero from isometric.spaceninja.com)
I shouldn't think it is, because it's been running like this for quite some time, and no problems. the reason i use 2.2.2 is that httpsconnection before 2.2.2 doesn't work with microsoft IIS. Which is tough when your payment processor runs iis :-) stunnel drops too many connections to be of use. -s On Sunday, Mar 23, 2003, at 18:14 US/Eastern, Paul Winkler wrote:
On Sun, Mar 23, 2003 at 03:50:50PM -0500, Stefan A. Nagey wrote:
BZ-
yea, i saw all the stuff that ppl have suggested to you, it hasn't seemed to help any.
fsrefs.py seems to think that the objects are missing, but, my Data.fs hasn't changed in size, so i would tend to think that they're still hiding in there somewhere.
running: Python 2.2.2, Zope 2.5.1
That's a version mismatch. Zope 2.5.1 is not officially approved with Python 2.2.2 by Zope Corp. It *should* work but has not been widely tested. The only recommended version is 2.1.3.
I have no idea if this is related to your problem...
--PW
--
Paul Winkler http://www.slinkp.com Look! Up in the sky! It's ALLEGORICAL LATONIC CAPTAIN! (random hero from isometric.spaceninja.com)
_______________________________________________ Zope maillist - Zope@zope.org http://mail.zope.org/mailman/listinfo/zope ** No cross posts or HTML encoding! ** (Related lists - http://mail.zope.org/mailman/listinfo/zope-announce http://mail.zope.org/mailman/listinfo/zope-dev )
Stefan A. Nagey GSM: +1 (202) 415-0163
On Mon, 2003-03-24 at 00:49, Stefan A. Nagey wrote:
the reason i use 2.2.2 is that httpsconnection before 2.2.2 doesn't work with microsoft IIS. Which is tough when your payment processor runs iis :-)
You don't have to run Zope under 2.2 to get this functionality... you can just hand off to 2.2 with popen3 as necessary. Here's a quick howto: http://www.zope.org/Members/dylanr/secure_post HTH, Dylan
I can tell you what I have tried... 1) I have gone back to an old version, exported a good version of the object I THINK is breaking and imported it. Worked fine. Problem is, many things have happened since then and I can't use that old one. 2) Exported and imported the bad object (which is a /Members/ folder with tons of Members in it, and that is why I cannot throw it away). 3) Tried everything in Zope 2.6.1 (no help) 4) Used fsrecover.py in a bunch of ways. It seems to delete thing, but too many things. BZ On Sunday, March 23, 2003, at 03:50 PM, Stefan A. Nagey wrote:
BZ-
yea, i saw all the stuff that ppl have suggested to you, it hasn't seemed to help any.
fsrefs.py seems to think that the objects are missing, but, my Data.fs hasn't changed in size, so i would tend to think that they're still hiding in there somewhere.
running: Python 2.2.2, Zope 2.5.1 on linux 2.4.20 on an ext3 partition over a raid1 array (using software raid/md)
stefan
On Sunday, Mar 23, 2003, at 15:47 US/Eastern, BZ wrote:
Hey Stefan,
What version of Zope/Python you using? I have been trying to get rid of a POSKeyError for two weeks.
BZ
On Sunday, March 23, 2003, at 03:27 PM, Stefan A. Nagey wrote:
oh yea, and deleteing the objects in question is really not the preferred solution, because i would really rather not have to reconstruct the data from other means.
thank you in advance for imparting your vast wisdom to (on?) me.
stefan
On Sunday, Mar 23, 2003, at 15:23 US/Eastern, Stefan A. Nagey wrote:
I've been struggling with this one non-stop since last night.
I work at an ISP, we keep our customer records in zope.
Somehow, i seem to have lost records for nearly 1/4 of our customers.
I've tried deleting the index, packing the DB, exporting and then importing into a fresh Data.fs, using fsrecover.py
i've also tried fsrefs.py, which tells me that i'm missing a bunch of objects.
The size of my Data.fs is prettymuch unchanged, which might suggest that the data is still there, but how do i get to it?
What can I do that I haven't tried already, or, what have i messed up that doing again will make all my worries go away.
any help would be very much appreciated.
thanks
stefan
_______________________________________________ Zope maillist - Zope@zope.org http://mail.zope.org/mailman/listinfo/zope ** No cross posts or HTML encoding! ** (Related lists - http://mail.zope.org/mailman/listinfo/zope-announce http://mail.zope.org/mailman/listinfo/zope-dev )
Stefan A. Nagey GSM: +1 (202) 415-0163
_______________________________________________ Zope maillist - Zope@zope.org http://mail.zope.org/mailman/listinfo/zope ** No cross posts or HTML encoding! ** (Related lists - http://mail.zope.org/mailman/listinfo/zope-announce http://mail.zope.org/mailman/listinfo/zope-dev )
Stefan A. Nagey GSM: +1 (202) 415-0163
_______________________________________________ Zope maillist - Zope@zope.org http://mail.zope.org/mailman/listinfo/zope ** No cross posts or HTML encoding! ** (Related lists - http://mail.zope.org/mailman/listinfo/zope-announce http://mail.zope.org/mailman/listinfo/zope-dev )
[Stefan A. Nagey wrote (stefan@ispvip.biz) on 3/23/03 3:50 PM]
running: Python 2.2.2, Zope 2.5.1 on linux 2.4.20 on an ext3 partition over a raid1 array (using software raid/md)
FWIW, Python 2.2.x is not supported with Zope.2.5.1, you should run it with Python 2.1.3. <--> george donnelly - http://www.zettai.net/ - "We Love Newbies" :) Zope Hosting - Dynamic Website Design - Search Engine Promotion Yahoo, AIM: zettainet - MSN: zettainet@hotmail.com - ICQ: 51907738
Stefan A. Nagey wrote at 2003-3-23 15:23 -0500:
I've been struggling with this one non-stop since last night.
I work at an ISP, we keep our customer records in zope.
Somehow, i seem to have lost records for nearly 1/4 of our customers.
I've tried deleting the index, packing the DB, exporting and then importing into a fresh Data.fs, using fsrecover.py
i've also tried fsrefs.py, which tells me that i'm missing a bunch of objects.
The size of my Data.fs is prettymuch unchanged, which might suggest that the data is still there, but how do i get to it?
Provided you have enough disk space, use "fsdump" to get a textual representation of your "Data.fs". You will see object records containing "oid" values. These are hexadecimal representations of the values reported in "POSKeyError" (side node: why do "POSKeyError" not report the hexadecimal value?). To convert between hexadecimal and textual representation, you can use "U64" and "p64" from "ZODB.utils". "U64" converts a textual representation into a long and "p64" converts a long into a textual representation. Use a text editor to find the last reference to your missing object in the textual representation of your "Data.fs" (obtained by "fsdump"). That's the last time, the object has been modified. Look at the next transaction record. It has a "tid" field. Note the "offset" for this transaction. Truncate your "Data.fs" to the given offset (in a copy, of course). Start Zope with the truncated "Data.fs". Your object should be there. Export it (you will get the state as it was at the time of the transaction where you have cut "Data.fs"; you may try to use later transaction, to be more up-to-date). Import the exported object into a Data.fs where you want to work with. Dieter
Wow. Thanks Dieter. I can't wait to try this. Are these the steps? 1) Dump Data.fs (backup copy) to text file using fsdump 2) Convert POSKeyError to hex OID (object ID) using U64 3) Find last UID in the Data.fs dump using text editor 4) Find TID of next transaction's and offset number 5) Truncate Data.fs to that given offset (tool? how?) 6) Start Zope with truncated Data.fs 7) Find object with problem and export it 9) Import object into new Zope installation Questions: - How do you truncate a Data.fs to an offset? - How do I know where the object is that had the problem (will I see it in the Data.fs dump I did?) Thanks for all your help. BZ
Stefan A. Nagey wrote at 2003-3-23 15:23 -0500:
I've been struggling with this one non-stop since last night.
I work at an ISP, we keep our customer records in zope.
Somehow, i seem to have lost records for nearly 1/4 of our customers.
I've tried deleting the index, packing the DB, exporting and then importing into a fresh Data.fs, using fsrecover.py
i've also tried fsrefs.py, which tells me that i'm missing a bunch of objects.
The size of my Data.fs is prettymuch unchanged, which might suggest that the data is still there, but how do i get to it?
Provided you have enough disk space, use "fsdump" to get a textual representation of your "Data.fs".
You will see object records containing "oid" values. These are hexadecimal representations of the values reported in "POSKeyError" (side node: why do "POSKeyError" not report the hexadecimal value?). To convert between hexadecimal and textual representation, you can use "U64" and "p64" from "ZODB.utils". "U64" converts a textual representation into a long and "p64" converts a long into a textual representation.
Use a text editor to find the last reference to your missing object in the textual representation of your "Data.fs" (obtained by "fsdump"). That's the last time, the object has been modified.
Look at the next transaction record. It has a "tid" field. Note the "offset" for this transaction. Truncate your "Data.fs" to the given offset (in a copy, of course). Start Zope with the truncated "Data.fs". Your object should be there. Export it (you will get the state as it was at the time of the transaction where you have cut "Data.fs"; you may try to use later transaction, to be more up-to-date).
Import the exported object into a Data.fs where you want to work with.
Dieter
_______________________________________________ Zope maillist - Zope@zope.org http://mail.zope.org/mailman/listinfo/zope ** No cross posts or HTML encoding! ** (Related lists - http://mail.zope.org/mailman/listinfo/zope-announce http://mail.zope.org/mailman/listinfo/zope-dev )
BZ wrote at 2003-3-24 18:03 -0600:
... Questions: - How do you truncate a Data.fs to an offset?
There is a HowTo on Zope.org about repairing corrupted Data.fs. It tells you how to truncate a file. It's the "truncate" method of Pythons file objects.
- How do I know where the object is that had the problem (will I see it in the Data.fs dump I did?)
The POSKeyError tells you the oid of the missing object (as a string). The dump of your "Data.fs" lists all object records contained in the "Data.fs" with their oid in hexform. Object records are contained in transaction records (starting at "offset"). The last occurrence of the object is where the object has been written latest. When you truncate at the offset of the following transaction, you have the state as it was after the object was written. Dieter
Hi Stefan, Stefan A. Nagey wrote:
I work at an ISP, we keep our customer records in zope.
Somehow, i seem to have lost records for nearly 1/4 of our customers.
I would suggest going to your last nightly backup... cheers, Chris
participants (7)
-
BZ -
Chris Withers -
Dieter Maurer -
Dylan Reinhardt -
george donnelly -
Paul Winkler -
Stefan A. Nagey