[Zope] RE : Re: RE : Re: Frequent Zope crashes (Zope 2.9.8)

[Paul Brettschneider]

Hi,

--- Tres Seaver <tseaver at palladion.com> a écrit :

> -----BEGIN PGP SIGNED MESSAGE-----
> Hash: SHA1
> 
> Paul Brettschneider wrote:
> > Hi,
> > 
> > --- Tres Seaver <tseaver at palladion.com> a écrit :
> >> Paul Brettschneider wrote:
> >>> Hello,
> >>>
> >>> my Zope 2.9.8 instance crashes up to 6 times per
> >> hour.
> >>> This is very unfortunate since the constant
> >> restarting
> >>> brings performance to its knees.
> >>>
> >>> It runs under Linux in 64 bit mode on an AMD64 .
> >>> I managed to catch two backtraces with gdb
> >>> (see end of the mail). Both backtraces show a
> >> crash
> >>> in cc_oid_unreferenced(ccobject *self, PyObject
> >> *oid)
> >>> in persistent/cPickleCache.c:
> >>> Either in line 576: v =
> PyDict_GetItem(self->data,
> >>> oid);
> >>> or in line 607: Py_DECREF((ccobject
> >>> *)((cPersistentObject *)v)->cache);
> >>>
> >>> v and v->cache seem to point to heap:
> >>> (gdb) print v
> >>> $1 = (PyObject *) 0x5f8920
> >>> (gdb) print ((cPersistentObject *)v)->cache
> >>> $2 = (PerCache *) 0x613620
> >>>
> >>> Always called from Per_dealloc(cPersistentObject
> >>> *self) in persistent/cPersistence.c
> >>> in line 578:
> >>> cPersistenceCAPI->percachedel(self->cache,
> >> self->oid);
> >>> Is this a known issue?
> >>>
> >>> Thank you for any help,
> >> Can you reproduce using the following
> from-scratch
> >> build?
> > 
> > No, the crash only happens with this zope instance
> and
> > only under heavy load. I will try to remove all
> custom
> > products before filing a bug report.
> 
> Hmm, I was hopeful that there might have been a
> build glitch (some 32-
> vs. 64 bit thing).  Isolating such a problem will be
> easier if we can
> reproduce the error on a system whose Zope you built
> yourself from source.

This instance is compiled from source. I will try
again with an updated gcc.

> I wouldn't rip out third-party products, esepecially
> those which have no
> C extensions:  they can hardly be provoking the
> segfault.  Hmm, I wonder
> if you might be using a third-party library which
> *does* (e.g., an RDBMS
> or LDAP client library, PIL, etc.).  Verifying the
> mechanism used to
> build them would be important, too.

I removed a custom product which I suspected being the
culprit. The crashes still happen, but now only about
five times a day (probably due to much less traffic at
the moment), making obtaining gdb backtraces hard. I
managed to do it and it still segfaults in the same
place:

#0  0x0000000000436777 in PyDict_Contains ()
#1  0x00000000004369ad in PyDict_GetItem ()
#2  0x00002b3c60f78f37 in cc_oid_unreferenced
(self=0x2aaaaab034c8, oid=0x2aaaadae6cf0) at
persistent/cPickleCache.c:576
#3  0x00002b3c60d70d28 in Per_dealloc
(self=0x2aaaadae4a28) at persistent/cPersistence.c:578

The currently installed products are:
drwxr-xr-x  8 zope users 4096 2007-10-31 11:16
CMFActionIcons
drwxr-xr-x  9 zope users 4096 2007-10-31 11:16
CMFCalendar
drwxr-xr-x  9 zope users 4096 2007-10-31 11:16 CMFCore
drwxr-xr-x 12 zope users 4096 2007-10-31 11:16
CMFDefault
drwxr-xr-x  6 zope users 4096 2007-10-31 11:16
CMFSetup
drwxr-xr-x  7 zope users 4096 2007-10-31 11:16
CMFTopic
drwxr-xr-x  5 zope users 4096 2007-10-31 11:16 CMFUid
drwxr-sr-x  4 zope users 4096 2007-11-06 11:46
CookieCrumbler
drwxr-xr-x  7 zope users 4096 2007-10-31 11:20
DCWorkflow
drwxr-sr-x  2 zope users 4096 2008-01-28 21:26
DeadlockDebugger
drwx--Sr-x  4 zope users 4096 2007-11-24 11:05 Epoz
drwxr-xr-x  6 zope users 4096 2007-11-24 11:05
ExternalEditor
drwxr-xr-x  7 zope users 4096 2007-11-24 11:05
Localizer
drwxr-sr-x  3 root users 4096 2007-11-06 11:37
TranslationService
drwxr-sr-x  2 root users 4096 2007-11-06 11:37 ZNagios
drwxr-sr-x  3 zope users 4096 2008-02-17 14:57
ZPsycopgDA

If I am informed correctly, only ZPsycopgDA contains
C-code. The psycopg module is not built from source
but installed via the Debian package repository.

The imports in all external methods are:
from base64 import encodestring
from DateTime import *
from DateTime import DateTime
from DocumentTemplate import HTML
from email.Header import Header
from htmlentitydefs import entitydefs
from HTMLParser import
HTMLParser,HTMLParseError,piclose, charref, entityref
from math import floor
from os import popen
from PIL import Image
from popen2 import popen2
from re import match
from string import *
from string import atoi
from string import atoi, atof
from string import atoi, atof, split, find
from string import atoi, atof, split, join
from string import find
from string import join
from string import lower,find
from string import replace
from string import split
from string import split, atoi
from string import strip
from StringIO import StringIO
from StructuredText import HTML
from tempfile import NamedTemporaryFile
from whrandom import choice
from whrandom import randint
from ZODB.POSException import POSKeyError
import cStringIO
import difflib
import httplib
import os
import PIL.Image
import PIL.Image, PIL.ImageDraw, PIL.ImageFont
import re
import regex
import re,string
import rfc822, string
import smtplib
import string
import StringIO

Nothing really suspicious.

Thanks,
Paul


      _____________________________________________________________________________ 
Ne gardez plus qu'une seule adresse mail ! Copiez vos mails vers Yahoo! Mail http://mail.yahoo.fr