[ZODB-Dev] [ zodb-Bugs-547020 ] Weird ZEO error: Aiieee! error code 25
noreply@sourceforge.net
noreply@sourceforge.net
Tue, 09 Jul 2002 07:17:15 -0700
Bugs item #547020, was opened at 2002-04-22 10:47
You can respond by visiting:
https://sourceforge.net/tracker/?func=detail&atid=115628&aid=547020&group_id=15628
Category: None
Group: None
Status: Open
Resolution: None
Priority: 5
Submitted By: Chris Withers (fresh)
Assigned to: Jeremy Hylton (jhylton)
Summary: Weird ZEO error: Aiieee! error code 25
Initial Comment:
On a Python ZEO client, we get:
> File c:\zope\2-4-2_base\lib\python\ZEO\ClientStorage.py, line 426, in tpc_vote
> (Object: (x.x.com', xxxx))
> File c:\zope\2-4-2_base\lib\python\ZEO\zrpc.py, line 228, in __call__
> TypeError: exceptions must be strings, classes, or instances
The matching entry on the Storage Server is:
> 2002-04-22T10:33:43 ERROR(200) zdaemon zdaemon: Mon Apr 22 11:33:43 2002: Aiieee! 2107
exited with error code: 25
After that, the storage server tries to fork, and we got the following entry pattern in the logs:
> ------
> 2002-04-22T10:33:43 INFO(0) zdaemon zdaemon: Mon Apr 22 11:33:43 2002: Houston, we
have
> forked
> ------
> 2002-04-22T10:33:43 INFO(0) zdaemon zdaemon: Mon Apr 22 11:33:43 2002: Hi, I just forked
> off a kid: 2155
> ------
> 2002-04-22T10:33:43 INFO(0) zdaemon zdaemon: Mon Apr 22 11:33:43 2002: Houston, we
have
forked
..however, the storage server doesn't ever accept connections afterwards until it is manually
stopped and restarted.
During this restart process, we often get the following entries in the storage server logs:
> ------
> 2002-04-21T09:22:22 PROBLEM(100) ZODB FS FS21 warn: /x/Data.fs > truncated, possibly due
to damaged records at 2147482867
>
> ------
> 2002-04-21T09:22:22 PROBLEM(100) ZODB FS FS21 warn: Writing truncated data from >
/x/Data.fs to /x/Data.fs.tr14
>
> ------
...but regardless of whether we get those messages or not, the storage server takes an age to
start and uses 100% CPU the whole time.
This storage server has been completely stable for about a month, and this problem has started
happening reccurrently in the last few days.
What's the best way to go about find what's going on?
----------------------------------------------------------------------
>Comment By: Chris Withers (fresh)
Date: 2002-07-09 14:17
Message:
Logged In: YES
user_id=24723
This has been biting me again.
Any chance this meaningless error message
> File c:\zope\2-4-2_base\lib\python\ZEO\ClientStorage.py, line 426, in tpc_vote
> (Object: (x.x.com', xxxx))
> File c:\zope\2-4-2_base\lib\python\ZEO\zrpc.py, line 228, in __call__ > TypeError: exceptions must be strings,
classes, or instances
...could be made more meaningful?
The same thing happens on Zope 2.5.1 and ZEO 1.0 final...
cheers,
Chris
----------------------------------------------------------------------
Comment By: Chris Withers (fresh)
Date: 2002-04-23 09:23
Message:
Logged In: YES
user_id=24723
Ah, okay, well, the error is misleading. At the hint of a 'file too big' problem I actually looked at the size of
the Data.fs: 2048 Mb, ie: 2GB.
So, this is just simply the 2GB Limit. Can anything be done to make the error message more helpful?
I need to ask some questions related to permenantly solving this problem (packing was a good band-aid fix)
but zodb-dev@zope.org will be more appropriate for that...
----------------------------------------------------------------------
Comment By: Jeremy Hylton (jhylton)
Date: 2002-04-22 19:19
Message:
Logged In: YES
user_id=31392
I suspect it is something funky with your environment. On
my machine (Mandrake 7.2?) RLIMIT_FSIZE > 10**18.
----------------------------------------------------------------------
Comment By: Jeremy Hylton (jhylton)
Date: 2002-04-22 19:01
Message:
Logged In: YES
user_id=31392
The zdaemon error message is probably a little misleading.
I believe that 25 is the signal sent to the child causing
it to exit. If that's right, then you're getting SIGXFSZ,
which means that it tried to extend a file past the rlimit
(RLIMIT_FSIZE).
Does that sound plausible? How big is the file? Do you
have a custom RLIMIT_FSIZE or know what the default is for
your OS?
It makes sense that this happens during tpc_vote(), because
that's when all the data is copied from the tempfile to the
Data.fs. If a failure occurs at this point, I'll bet that
on restart FileStorage ignores the index and recomputes it,
which would explain why it is slow.
Not sure how we can detect this problem more gracefully.
Until today, I had never even heard of SIGXFSZ.
----------------------------------------------------------------------
You can respond by visiting:
https://sourceforge.net/tracker/?func=detail&atid=115628&aid=547020&group_id=15628