[Zope-dev] Very severe memory leak

[Shane Hathaway]

On 08/25/2003 05:12 PM, Leonardo Rochael Almeida wrote:
> But maybe this means that the leak is not related to the DateTime
> refcounts. It's just that the fast and continual increase in DateTime
> objects is really bugging me. BTW, what is the usual DateTime refcount
> in heavy ZCatalog sites you guys run?

We try to keep it near 0. :-)  DateTimes are full of nonsense.  Zope's 
DateTime troubles drove the creation of Python 2.3's new datetime 
module.  From what I can tell, the new datetimes are excellent.

> Let me restate something important, because I forget about it myself
> sometimes when I'm thinking about this problem: *upgrading to 2.6.1 made
> the situation definetly worse*. The site became lot faster, but the
> memory situation became intolerable, so I believe there is something
> introduced between 2.5.1 and 2.6.1 that is causing our problems. Is
> there a fix in 2.6.2 for some leak introduced *in* the 2.6 series?

I think there are, actually, but AFAIK they were obscure and rare.

> Another thing, we have already estabilished that clearing the caches
> resets the DateTime refcount, so the DateTime must be anchored to the
> caches somehow. So, if the DateTime refcounts are constantly in the
> 50k-100k range, how come the cache object count for all threads is below
> 5k?

If the DateTimes are in catalog metadata or indexes, this would make sense.

> If they're indirectly attached, it means that a lot of DateTimes are
> anchored in just a few objects, so no matter how much I reduce the
> target cache size, it might never be enough to throw out the
> DateTimes...

They are probably in the ZCatalog.  Remove all metadata fields you can. 
  If you have index dates, use DateTimeIndexes rather than the standard 
indexes.

I find it helpful to simply visit the information ZCatalog is storing 
for just one cataloged object.  In the catalog contents view, click one 
object.  Look at each field and consider how ZCatalog might misuse that 
field for other objects.

> hmm, any chance of this remote console showing up somewhere? :-)

I've attached what I threw together.  It Works For Me. ;-)  I put it in 
Zope's Extensions directory then made an external method to call the 
"listenlocal" function.  It sets up a TCP listener bound only to 
localhost, so you have to telnet to port 8765 locally.  No readline 
support.  No method is provided for shutting down the listener (except 
restarting Zope).  To exit the session, push ctrl-D then enter.

> I might use it to navigate the thread object cache.

It's unfortunate you have to go so deep, but that sounds like a good plan.

Shane
-------------- next part --------------

import socket
from code import InteractiveConsole
from types import IntType, StringType
from thread import start_new_thread, get_ident
import sys
import time


class ThreadedObjectProxy:
    """Proxy to different objects based which thread invoked it.
    """
    def __init__(self, default):
        self._default = default
        self._alts = {}

    def setAlternative(self, o):
        self._alts[get_ident()] = o

    def delAlternative(self):
        try: del self._alts[get_ident()]
        except KeyError: pass

    def __getattr__(self, name):
        ob = self._alts.get(get_ident(), self._default)
        return getattr(ob, name)


class RemoteConsole(InteractiveConsole):

    def __init__(self, sock, file, filename=None, locals=None):
        if filename is None:
            filename = str(file)
        self.sock = sock
        self.file = file
        InteractiveConsole.__init__(self, locals=locals, filename=filename)

    def raw_input(self, prompt=''):
        if prompt:
            self.file.write(prompt)
        s = self.file.readline().rstrip()
        if s == '\x04':  # Ctrl-D
            raise EOFError
        return s

    def interactAndClose(self):
        sys.stdout.setAlternative(self.file)
        sys.stderr.setAlternative(self.file)
        sys.stdin.setAlternative(self.file)
        try:
            try:
                self.interact()
            except EOFError:
                pass
        finally:
            sys.stdout.delAlternative()
            sys.stderr.delAlternative()
            sys.stdin.delAlternative()
            self.file.close()
            self.sock.close()


def setupStreams():
    if not hasattr(sys.stdout, 'setAlternative'):
        sys.stdout = ThreadedObjectProxy(sys.stdout)
    if not hasattr(sys.stderr, 'setAlternative'):
        sys.stderr = ThreadedObjectProxy(sys.stderr)
    if not hasattr(sys.stdin, 'setAlternative'):
        sys.stdin = ThreadedObjectProxy(sys.stdin)


def accept_connections(s):
    while 1:
        cs, addr = s.accept()
        f = cs.makefile('w+', 0)
        i = RemoteConsole(cs, f)
        start_new_thread(i.interactAndClose, ())


def listen(addr, locals=None):
    setupStreams()
    if isinstance(addr, StringType):
        t = socket.AF_UNIX
    elif isinstance(addr, IntType):
        t = socket.AF_INET
        addr = ('', addr)
    else:
        t = socket.AF_INET
    s = socket.socket(t, socket.SOCK_STREAM)
    s.setsockopt(
        socket.SOL_SOCKET, socket.SO_REUSEADDR,
        s.getsockopt (socket.SOL_SOCKET, socket.SO_REUSEADDR) | 1
        )
    s.bind(addr)
    s.listen(1)
    start_new_thread(accept_connections, (s,))


def listenlocal(port=8765):
    listen(('localhost', port))


if __name__ == '__main__':
    listen(8765)
    while 1:
        time.sleep(1)