[Zope-Checkins] CVS: Zope/lib/python/Products/CallProfiler - .cvsignore:1.1.2.1 CallProfiler.py:1.1.2.1 README.txt:1.1.2.1 TODO.txt:1.1.2.1 init.py:1.1.2.1 make_dist.sh:1.1.2.1 profiler.py:1.1.2.1 refresh.txt:1.1.2.1 run_tests:1.1.2.1 version.txt:1.1.2.1

Fri, 17 May 2002 01:26:55 -0400

Update of /cvs-repository/Zope/lib/python/Products/CallProfiler
In directory cvs.zope.org:/tmp/cvs-serv26935/CallProfiler

Added Files:
      Tag: anthony-CallProfiler-branch
	.cvsignore CallProfiler.py README.txt TODO.txt __init__.py 
	make_dist.sh profiler.py refresh.txt run_tests version.txt 
Log Message:
Initial checkin of CallProfiler branch. This is the internal ekit version.

Added since the 1.4 release:

  profiler_cache_hook - hooks into the ZCache stuff for hit/miss counts
  sorting on the summary page
  Add / replace a profileable module "on the fly"

=== Added File Zope/lib/python/Products/CallProfiler/.cvsignore ===
*.pyc
*.pyo
*.gz

=== Added File Zope/lib/python/Products/CallProfiler/CallProfiler.py === (467/567 lines abridged)
# Copyright (c) 2002 ekit.com Inc (http://www.ekit-inc.com/)
# 
# Permission is hereby granted, free of charge, to any person obtaining a copy
# of this software and associated documentation files (the "Software"), to deal
# in the Software without restriction, including without limitation the rights
# to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
# copies of the Software, and to permit persons to whom the Software is
# furnished to do so, subject to the following conditions:
# 
#   The above copyright notice and this permission notice shall be included in
#   all copies or substantial portions of the Software.
# 
# THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
# IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
# FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
# AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
# LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
# OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
# SOFTWARE.

# $Id: CallProfiler.py,v 1.1.2.1 2002/05/17 05:26:53 anthony Exp $

from zLOG import LOG, ERROR, INFO
from Globals import InitializeClass, HTMLFile
from OFS.SimpleItem import Item
from Acquisition import Implicit
from Persistence import Persistent
from AccessControl import ClassSecurityInfo
from AccessControl import ModuleSecurityInfo
modulesecurity = ModuleSecurityInfo()

import cStringIO
import time, operator
from thread import get_ident

# get the profiler store
from profiler import profiler

def profiler_call_hook(self, *args, **kw):
    '''A call hook
    '''
    mt = self.meta_type
    sid = self.getId()
    profiler.startCall(mt, sid)
    try:
        return self.profiler_call_original(*args, **kw)
    finally:
        profiler.endCall()

def profiler_publish_hook(request, *args, **kw):

[-=- -=- -=- 467 lines omitted -=- -=- -=-]

# Revision 1.10  2002/02/07 23:12:48  rjones
# Fixes to the data gathering and display
#
# Revision 1.9  2002/02/07 05:09:11  rjones
# Better call stack handling
#
# Revision 1.8  2002/02/06 00:33:55  rjones
# Lots of data handling improvements:
#   . moved the data handling part off into a separate module
#   . that module has some basic unit tests
#
# Revision 1.7  2002/02/05 22:11:02  rjones
# Fixes
#
# Revision 1.6  2002/02/05 04:50:13  rjones
# Fixes, aggregation, oh my! :)
#
# Revision 1.5  2002/02/01 05:42:17  rjones
# fixes
#
# Revision 1.4  2002/01/31 23:11:36  rjones
# copyright and CVS comment cleanups
#
# Revision 1.3  2002/01/31 06:19:17  rjones
# Now adds itself to the Control Panel, and isn't available for adding elsewhere.
#
# Revision 1.2  2002/01/31 05:03:27  rjones
# adding CallProfiler to HEAD
#
# Revision 1.1.2.6  2002/01/31 04:16:33  rjones
# Some more cleanups
#
# Revision 1.1.2.5  2002/01/31 04:09:42  rjones
# More cleanups in the profiler code so we can get at the data more easily
#
# Revision 1.1.2.4  2002/01/31 00:50:08  rjones
# Profiler now patches the Zope modules as required:
#  . ZPublisher/Publish.py publish function
#  . others as defined in the profileable_modules dict in CallProfiler.py
#
# Revision 1.1.2.3  2002/01/30 07:36:00  rjones
# split off the processing code from the display code - should be easy enough
# to do the render in ZPT or *shudder* DTML.
#
# Revision 1.1.2.2  2002/01/30 05:41:33  rjones
# cosmetic changes
#
# Revision 1.1.2.1  2002/01/30 04:48:38  rjones
# CallProfiler initial version
#

=== Added File Zope/lib/python/Products/CallProfiler/README.txt ===
Call Profiler Documentation (the minimalist approach)
===========================

Purpose:
   To monitor the chain of DTML, ZSQL, ZPT, PythonMethod, PythonScript, ...
calls in a Zope request and gather timing information, for the purpose of
identifying hot-spots for potential optimisation.

Usage:
   Once the product is installed in your Products directory and Zope has
been restarted, visit the Call Profiler link in your Control Panel.

   In the configuration tab, check the types of documents you wish to
obtain the timing information for. You may also clear any previously
gathered timing information using the "clear" buttons.

   Once you have selected the documents to monitor and have clicked
"Monitor selected calls", load up the pages you wish to profile. Once
they're loaded (or even as they're loading :) you should visit the results
tab. There you will find a list of the requests made by browsers with some
timing information:
   . clicking the url will load the page requested
   . clicking on the time of the request will bring up a blow-by-blow
     breakdown of the documents called to form the request.

   You may also see the requests aggregated by URL on the "Requests By URL"
tab - giving the minimum, average and maximum times for the responses. You
may:
   . click the url to load the page requested
   . click the session times to see the detailed breakdown of the request

   The detailed view is set to highlight calls that broach 3% (yellow), 5%
(orange) and 10% (red) of the total request time.

   If a given document call has sub-calls:
   . the '...' times indicate the time spent in the call (between sub-calls)
   . the (end) time gives two timings:
     1. the first is the time spent in the call not including sub-calls
     2. the second time is the total time of the call including sub-calls

Note:
   All profiling information is lost when either:
    1. Zope is restarted, or
    2. the Call Profiler product's code is re-loaded.

   Also, don't leave the profiler on for too long - it uses a boundless
memory-based store for the timing values. It will use up all your memory
eventually (though we haven't done any testing to determine how long that
might be).

License
=======
Copyright (c) ekit.com Inc (http://www.ekit-inc.com/)

Permission is hereby granted, free of charge, to any person obtaining a copy
of this software and associated documentation files (the "Software"), to deal
in the Software without restriction, including without limitation the rights
to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
copies of the Software, and to permit persons to whom the Software is
furnished to do so, subject to the following conditions:

  The above copyright notice and this permission notice shall be included in
  all copies or substantial portions of the Software.

THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
SOFTWARE.

This product includes software developed by Digital Creations for use in
the Z Object Publishing Environment (http://www.zope.org/).
(specifically, we use the control panel installation code from LeakFinder in
 the __init__.py module)

=== Added File Zope/lib/python/Products/CallProfiler/TODO.txt ===
TODO

. determine the pecentages for highlighting dynamically
  - if there's < 5 calls, red is >50%
  - if there's < 10 calls, red is >20%?
  - else red is >20%?

. stuff the results away in the ODB when a request is finished

. allow auto-culling or auto-aggregation of results when it's left on for a
  long time

. include a stringification of PARENTS and getPhysicalPath to indicate where
  particular methods are

DONE

. sorting of columns in the results

. clean up the data storage so I can reduce the computation time

. better call graph display

. url lopping when > 100 chars

. detect caching hit/miss

=== Added File Zope/lib/python/Products/CallProfiler/__init__.py ===
# Copyright (c) ekit.com Inc (http://www.ekit-inc.com/)
# Permission is hereby granted, free of charge, to any person obtaining a copy
# of this software and associated documentation files (the "Software"), to deal
# in the Software without restriction, including without limitation the rights
# to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
# copies of the Software, and to permit persons to whom the Software is
# furnished to do so, subject to the following conditions:
# 
#   The above copyright notice and this permission notice shall be included in
#   all copies or substantial portions of the Software.
# 
# THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
# IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
# FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
# AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
# LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
# OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
# SOFTWARE.

# This product includes software developed by Digital Creations for use in
# the Z Object Publishing Environment (http://www.zope.org/).
# (specifically, we use the control panel installation code from LeakFinder)

__version__ = '1.6'

def installControlPanel(context, panelClass):
    from App.ApplicationManager import ApplicationManager
    from Acquisition import aq_base
    app = context._ProductContext__app
    cp = app.Control_Panel
    id = panelClass.id
    if 0: # Enable to clean up the control panel.
        try: del cp._objects
        except: pass
    cp.id # Unghostify.
    if hasattr(cp, id):
        return cp._getOb(id)
    if cp.__dict__.has_key('_objects'):
        # _objects has been overridden.  We have to persist.
        existing = getattr(aq_base(cp), id, None)
        if existing is None or existing.__class__ != panelClass:
            cp._setObject(id, panelClass())
    else:
        # Don't persist what we don't have to.
        objects = ApplicationManager._objects
        objects = filter(lambda o, id=id: o['id'] != id, objects)
        ApplicationManager._objects = objects + (
            {'id':id, 'meta_type':panelClass.meta_type},)
        try: delattr(cp, id)
        except: pass
        setattr(ApplicationManager, id, panelClass())
    return cp._getOb(id)

def initialize(context):
    import CallProfiler

    installControlPanel(context, CallProfiler.CallProfiler)

    import Globals, OFS
    CallProfiler.CallProfiler.icon = 'misc_/CallProfiler/profiler.gif'
    icon = Globals.ImageFile('www/profiler.gif', globals())
    icon.__roles__=None
    if not hasattr(OFS.misc_.misc_, 'CallProfiler'):
        setattr(OFS.misc_.misc_, 'CallProfiler',
	    OFS.misc_.Misc_('CallProfiler', {}))
    getattr(OFS.misc_.misc_, 'CallProfiler')['profiler.gif']=icon

    context.registerClass(
	CallProfiler, meta_type = 'Call Profiler',
	constructors = (CallProfiler.manage_addForm, CallProfiler.manage_add)
    )

=== Added File Zope/lib/python/Products/CallProfiler/make_dist.sh ===
#!/bin/sh

# figure the dist name
name=`cat version.txt`
echo "Building $name.tar.gz"

# make the dist dir
rm -rf dist
mkdir dist
mkdir dist/CallProfiler
mkdir dist/CallProfiler/dtml
mkdir dist/CallProfiler/www
mkdir dist/CallProfiler/test

# copy in the dist files
cp *.py *.txt run_tests dist/CallProfiler
cp dtml/*.dtml dist/CallProfiler/dtml/
cp www/*.* dist/CallProfiler/www/
cp test/*.py dist/CallProfiler/test/

# package
(cd dist; tar cf ${name}.tar CallProfiler; gzip ${name}.tar)
mv dist/${name}.tar.gz .

# cleanup
rm -rf dist

=== Added File Zope/lib/python/Products/CallProfiler/profiler.py === (472/572 lines abridged)
# Copyright (c) 2002 ekit.com Inc (http://www.ekit-inc.com/)
# 
# Permission is hereby granted, free of charge, to any person obtaining a copy
# of this software and associated documentation files (the "Software"), to deal
# in the Software without restriction, including without limitation the rights
# to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
# copies of the Software, and to permit persons to whom the Software is
# furnished to do so, subject to the following conditions:
# 
#   The above copyright notice and this permission notice shall be included in
#   all copies or substantial portions of the Software.
# 
# THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
# IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
# FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
# AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
# LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
# OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
# SOFTWARE.

# $Id: profiler.py,v 1.1.2.1 2002/05/17 05:26:53 anthony Exp $

import time, operator
from thread import get_ident

class Profiler:
    def __init__(self):
        self.reset()

    def reset(self):
        '''Reset the dicts
        '''
        # these vars hold the timing values
        self.thread = {}        # maps thread IDs to TID
        self.transaction = {}   # holds the full transaction info by TID

    def hasTID(self, tid):
        return self.transaction.has_key(tid)

    def startRequest(self, request):
        '''Register the start of a request (called by the Publisher)

           The StartDict entry will be transferred to the TimingDict once
           we have a valid timing mark (so it's a request we care about).
        '''
        tid = '%s:%s'%(time.time(), id(request))
        self.thread[get_ident()] = tid
        self.transaction[tid] = Transaction(tid, request.URL+request.PATH_INFO)

    def endRequest(self):

[-=- -=- -=- 472 lines omitted -=- -=- -=-]

                l.append('%-4s %-2s %-12s %-3s %-5s %s'%(
                    '.'*depth, event['ave_time_elapsed'], event['object'],
                    event['ave_time_total'], perc, proc))
            else:
                l.append('%-4s     %-12s %-3s %-5s'%(
                    '.'*depth, '(processing)', event['ave_time_total'],
                    perc))

        return '\n'.join(l)

    def __getitem__(self, name):
        '''Make this object look like one of the event info dicts
        '''
        if name == 'events': return self.events
        raise KeyError, name
#
# $Log: profiler.py,v $
# Revision 1.1.2.1  2002/05/17 05:26:53  anthony
# Initial checkin of CallProfiler branch. This is the internal ekit version.
#
# Added since the 1.4 release:
#
#   profiler_cache_hook - hooks into the ZCache stuff for hit/miss counts
#   sorting on the summary page
#   Add / replace a profileable module "on the fly"
#
# Revision 1.7  2002/02/14 05:54:21  rjones
# Handles caching, has sortable display columns
#
# Revision 1.6  2002/02/13 00:04:30  rjones
# *** empty log message ***
#
# Revision 1.5  2002/02/12 23:40:42  rjones
# added cache hit/miss detection
#
# Revision 1.4  2002/02/08 00:06:33  rjones
# added call counter
#
# Revision 1.3  2002/02/07 23:12:49  rjones
# Fixes to the data gathering and display
#
# Revision 1.2  2002/02/07 05:09:11  rjones
# Better call stack handling
#
# Revision 1.1  2002/02/06 00:33:55  rjones
# Lots of data handling improvements:
#   . moved the data handling part off into a separate module
#   . that module has some basic unit tests
#
#

=== Added File Zope/lib/python/Products/CallProfiler/refresh.txt ===

=== Added File Zope/lib/python/Products/CallProfiler/run_tests ===
#! /usr/bin/env python2

import test
test.go()

=== Added File Zope/lib/python/Products/CallProfiler/version.txt ===
CallProfiler-1.6

[Zope-Checkins] CVS: Zope/lib/python/Products/CallProfiler - .cvsignore:1.1.2.1 CallProfiler.py:1.1.2.1 README.txt:1.1.2.1 TODO.txt:1.1.2.1 __init__.py:1.1.2.1 make_dist.sh:1.1.2.1 profiler.py:1.1.2.1 refresh.txt:1.1.2.1 run_tests:1.1.2.1 version.txt:1.1.2.1

[Zope-Checkins] CVS: Zope/lib/python/Products/CallProfiler - .cvsignore:1.1.2.1 CallProfiler.py:1.1.2.1 README.txt:1.1.2.1 TODO.txt:1.1.2.1 init.py:1.1.2.1 make_dist.sh:1.1.2.1 profiler.py:1.1.2.1 refresh.txt:1.1.2.1 run_tests:1.1.2.1 version.txt:1.1.2.1