Zope instance restarting itself
Hello, I have a Zope-2.6.1 instance running which behaves very weird. It restarts itself :-( First I discovered that the Data.fs truncated and restarted Zope. I then used fsrecovery.py on the Data.fs and it removed a bad transaction. Then I started Zope with the new Data.fs file. Now the instans restarts at the same time but without the error - infact withput any error. Any hints on how I can debug this? Greetings, Gitte Wange
Hi, On Wed, 2004-10-06 at 11:05, Gitte Wange wrote:
Hello,
I have a Zope-2.6.1 instance running which behaves very weird. It restarts itself :-( First I discovered that the Data.fs truncated and restarted Zope. I then used fsrecovery.py on the Data.fs and it removed a bad transaction. Then I started Zope with the new Data.fs file. Now the instans restarts at the same time but without the error - infact withput any error. Any hints on how I can debug this?
with zope2.6 modify the start script to have the -D at the end so zope does not detach from terminal and you see all output. Watch out for memory and filesystem and if there are any limits set (via ulimit) Regards Tino
Tino Wildenhain wrote:
Hi,
On Wed, 2004-10-06 at 11:05, Gitte Wange wrote:
Hello,
I have a Zope-2.6.1 instance running which behaves very weird. It restarts itself :-( First I discovered that the Data.fs truncated and restarted Zope. I then used fsrecovery.py on the Data.fs and it removed a bad transaction. Then I started Zope with the new Data.fs file. Now the instans restarts at the same time but without the error - infact withput any error. Any hints on how I can debug this?
with zope2.6 modify the start script to have the -D at the end so zope does not detach from terminal and you see all output.
Thanks - I missed that. So I can see that the Data.fs is STILL getting truncated because of corrupted data. Well I have already used fsrecovery.py (from Zope-2.6.2 because the one in 2.6.1 should be broken). Any other good voodoo tricks to be practised? :) Greetings, Gitte Wange
Hi, On Wed, 2004-10-06 at 12:39, Gitte Wange wrote:
Tino Wildenhain wrote:
...
with zope2.6 modify the start script to have the -D at the end so zope does not detach from terminal and you see all output.
Thanks - I missed that. So I can see that the Data.fs is STILL getting truncated because of corrupted data. Well I have already used fsrecovery.py (from Zope-2.6.2 because the one in 2.6.1 should be broken).
Any other good voodoo tricks to be practised? :)
Dance naked in the woods when full moon.... well ;) Data.fs isnt truncated for no reason. Check ulimit and check disk space. See your system logs if there are any bad disks or issues with memory (heavy apps regulary getting sig bus or something) Regards Tino
Tino Wildenhain wrote:
Hi,
On Wed, 2004-10-06 at 12:39, Gitte Wange wrote:
Tino Wildenhain wrote:
...
with zope2.6 modify the start script to have the -D at the end so zope does not detach from terminal and you see all output.
Thanks - I missed that. So I can see that the Data.fs is STILL getting truncated because of corrupted data. Well I have already used fsrecovery.py (from Zope-2.6.2 because the one in 2.6.1 should be broken).
Any other good voodoo tricks to be practised? :)
Dance naked in the woods when full moon....
I think people will help me if I DON'T do that ;-))
well ;) Data.fs isnt truncated for no reason. Check ulimit and check disk space. See your system logs if there are any bad disks or issues with memory (heavy apps regulary getting sig bus or something)
Diskspace is 3.3GB left - should be plenty for now ;) ulimit? I have never used it. Checking logs etc doens't give me a clue that anyting could be wrong. It's very weird since I have a lot of similar instances running on the same machine that runs perfectly. Oh well - I will try to start the instance on another machine to see if it's machine problems. Greetings, Gitte Wange
Gitte Wange wrote:
Tino Wildenhain wrote:
[SNIP]
well ;) Data.fs isnt truncated for no reason. Check ulimit and check disk space. See your system logs if there are any bad disks or issues with memory (heavy apps regulary getting sig bus or something)
Diskspace is 3.3GB left - should be plenty for now ;) ulimit? I have never used it. Checking logs etc doens't give me a clue that anyting could be wrong. It's very weird since I have a lot of similar instances running on the same machine that runs perfectly.
Oh well - I will try to start the instance on another machine to see if it's machine problems.
I have moved the recovered Data.fs to another machine (and another Zope) trying to start the bastard under Zope-2.7 leaving me with this error when I start Zope: ZODB.POSException.ConflictError: database conflict error (oid 0000000000000013, serial was 034acb34875e1bcc, now 0000000000000000) I'm not sure if it's a result of the recovery or if the Data.fs is just broken. I have tried gooling for hints, not finding any similar problem. Any ideas ? Greetings, Gitte Wange
Gitte Wange wrote:
Diskspace is 3.3GB left - should be plenty for now ;) ulimit? I have never used it.
Then find out more about it and check if it's resulting in your Zope dying ;-)
Checking logs etc doens't give me a clue that anyting could be wrong. It's very weird since I have a lot of similar instances running on the same machine that runs perfectly.
Check the size of your Data.fs file, if it's close to 2GB, then you need to make sure you filesystem and python support large files. Also check the amount of space free on wherever temporary files are stored on that server...
I have moved the recovered Data.fs to another machine (and another Zope) trying to start the bastard under Zope-2.7 leaving me with this error when I start Zope:
ZODB.POSException.ConflictError: database conflict error (oid 0000000000000013, serial was 034acb34875e1bcc, now 0000000000000000)
I'm not sure if it's a result of the recovery or if the Data.fs is just broken.
What products do you have installed? This means a product is trying to modify something that another product is also trying to modify on startup. It shouldn't really be possible to happen, but I have seen it when using PTS and multiple ZEO clients, but I don't think that's what you're doing here ;-) Chris -- Simplistix - Content Management, Zope & Python Consulting - http://www.simplistix.co.uk
Chris Withers wrote:
Gitte Wange wrote:
Diskspace is 3.3GB left - should be plenty for now ;) ulimit? I have never used it.
Then find out more about it and check if it's resulting in your Zope dying ;-)
You're right - it could have been an issue (or it can be in the future) - it's not the case this time (I'm almost sure of now I come to think of it)
Checking logs etc doens't give me a clue that anyting could be wrong. It's very weird since I have a lot of similar instances running on the same machine that runs perfectly.
Check the size of your Data.fs file, if it's close to 2GB, then you need to make sure you filesystem and python support large files. Also check the amount of space free on wherever temporary files are stored on that server...
And now I think we are getting close to the problem. Data.fs IS around 2GB (2147483592 bytes to be exact). I will try to find out how FreeBSD likes that big files.
I have moved the recovered Data.fs to another machine (and another Zope) trying to start the bastard under Zope-2.7 leaving me with this error when I start Zope:
ZODB.POSException.ConflictError: database conflict error (oid 0000000000000013, serial was 034acb34875e1bcc, now 0000000000000000)
I'm not sure if it's a result of the recovery or if the Data.fs is just broken.
What products do you have installed? This means a product is trying to modify something that another product is also trying to modify on startup. It shouldn't really be possible to happen, but I have seen it when using PTS and multiple ZEO clients, but I don't think that's what you're doing here ;-)
Well the product it fails with is CMFArticle. It shouldn't do anything at startup but if the Data.fs is hitting a filesize limit that would make the instance crash. Thanks for booting my brain Chris :) I hope the problem is "just" that the Data.fs is too big. Greetings, Gitte Wange
Gitte Wange wrote:
Chris Withers wrote:
Gitte Wange wrote:
Checking logs etc doens't give me a clue that anyting could be wrong. It's very weird since I have a lot of similar instances running on the same machine that runs perfectly.
Check the size of your Data.fs file, if it's close to 2GB, then you need to make sure you filesystem and python support large files. Also check the amount of space free on wherever temporary files are stored on that server...
And now I think we are getting close to the problem. Data.fs IS around 2GB (2147483592 bytes to be exact). I will try to find out how FreeBSD likes that big files.
I compiled python-2.3.3 for my FreeBSD-5.2.1 myself - any flags I should use for python to have LFS or is it "just there"? Greetings, Gitte Wange
Gitte Wange wrote:
And now I think we are getting close to the problem. Data.fs IS around 2GB (2147483592 bytes to be exact).
Yes, that's a 2GB limit.
I will try to find out how FreeBSD likes that big files.
"FreeBSD" probably doesn't care, however, whatever filesystem you're using might have a 2GB limit on files.
I compiled python-2.3.3 for my FreeBSD-5.2.1 myself - any flags I should use for python to have LFS or is it "just there"?
I think it's there by default with 2.3.3, but do ./configure --help or whatever the python equivalent is before compilation and check there aren't any relevent options that could be supplied... cheers, Chris PS: You could also look at using mounted storages to split the Data.fs so it doesn't get so big... -- Simplistix - Content Management, Zope & Python Consulting - http://www.simplistix.co.uk
Gitte Wange wrote: [...]
And now I think we are getting close to the problem. Data.fs IS around 2GB (2147483592 bytes to be exact). I will try to find out how FreeBSD likes that big files.
I compiled python-2.3.3 for my FreeBSD-5.2.1 myself - any flags I should use for python to have LFS or is it "just there"?
Greetings, Gitte Wange
You can test if your Python runs with lfs with test_largefile.py. This script is in the test-subdirectory of your Python installation, e.g. /usr/lib/python/test HTH, Wolfram
Hi Gitte, On Thu, 2004-10-07 at 20:02, Gitte Wange wrote:
Gitte Wange wrote: ...
And now I think we are getting close to the problem. Data.fs IS around 2GB (2147483592 bytes to be exact). I will try to find out how FreeBSD likes that big files.
I compiled python-2.3.3 for my FreeBSD-5.2.1 myself - any flags I should use for python to have LFS or is it "just there"?
At least there is a howto or just use ./configure --help as it was suggested. A quick check for LFS is:
f=open("/tmp/foo","w") f.tell() 0L
When you see the L after Zero, chances are LFS is working :-) Regards Tino
Gitte Wange wrote at 2004-10-7 11:26 +0200:
... I have moved the recovered Data.fs to another machine (and another Zope) trying to start the bastard under Zope-2.7 leaving me with this error when I start Zope:
ZODB.POSException.ConflictError: database conflict error (oid 0000000000000013, serial was 034acb34875e1bcc, now 0000000000000000)
This means, that this transaction wanted to create a new object ("now: 000...") but when it tried to write it, an object with this oid was already present. Apparently, the mechanism to assign new "oid"s is broken. A stale "Data.fs.index" file might be the reason... -- Dieter
Dieter Maurer wrote:
Gitte Wange wrote at 2004-10-7 11:26 +0200:
... I have moved the recovered Data.fs to another machine (and another Zope) trying to start the bastard under Zope-2.7 leaving me with this error when I start Zope:
ZODB.POSException.ConflictError: database conflict error (oid 0000000000000013, serial was 034acb34875e1bcc, now 0000000000000000)
This means, that this transaction wanted to create a new object ("now: 000...") but when it tried to write it, an object with this oid was already present.
Apparently, the mechanism to assign new "oid"s is broken.
A stale "Data.fs.index" file might be the reason...
Hmmm ... this means? :-) Would it help to remove the Data.fs.index file? Greetings, Gitte Wange
Gitte Wange wrote:
Hmmm ... this means? :-) Would it help to remove the Data.fs.index file?
Yes. -- Simplistix - Content Management, Zope & Python Consulting - http://www.simplistix.co.uk
participants (5)
-
Chris Withers -
Dieter Maurer -
Gitte Wange -
Tino Wildenhain -
Wolfram Kraus