[Zope] Zope 2.7 FTP Bandwidth Limiter?
Chris McDonough
chrism at plope.com
Fri Feb 27 13:34:02 EST 2004
As you said, I'm afraid that doesn't give much information.
The profiler does capture Medusa calls (it captures all Python calls, it
doesn't discriminate), but it's not even clear that the issue is really
in Medusa. It might be and likely is way up in Zope somewhere.
Note that I don't know what it means to sort by internal time, so I'm
really not sure how to interpret the profile data. It might be better
to sort it descending by total time or cumulative time. Also, it might
be more informative to have more than 100 lines of profiling data (500
lines or so would be better). Basically, you're just looking for stuff
that takes an inordinate amount of time.
The next step if profiling doesn't help is to attach to the running
process using gdb while downloading to a sense of where it's looping. I
don't remember how to do this off the top of my head but there have been
a few maillist threads which discuss it. It also might be possible to
run Zope under strace to what's happening when the download is going on.
- C
On Fri, 2004-02-27 at 11:59, Edward Pollard wrote:
> The profiler seems to offer absolutely no information of use. I suspect
> it does not include Medusa performance.
>
> The only thing that popped up was a call to FileStorage.py, but that
> call did not seem to monopolize the CPU.
>
> Attached are three logs
> 1) The FTP session
> 2) The profiler data for that time period.
> 3) the TOP data for that period
>
>
> Due to the bugs in 2.6.2, I've had to disable FTP access to the server,
> which has my clients infuriated. Any help at all with this would be
> most grateful. The only other option is to got back to 2.6.0, but that
> means we can't compress the ZODB without truncating it to 4 bytes.
>
> Ed
>
> 230 Login successful.
> Remote system type is UNIX.
> Using binary mode to transfer files.
> ftp> put big_file.exe
> local: big_file.exe remote: big_file.exe
> 227 Entering Passive Mode (127,0,0,1,202,169)
> 150 Opening Binary connection for big_file.exe
> 226 Transfer complete.
> 18817367 bytes sent in 0.288 secs (6.4e+04 Kbytes/sec)
> ftp> get large_bin.exe
> local: large_bin.exe remote: large_bin.exe
> 227 Entering Passive Mode (127,0,0,1,202,156)
> 150 Opening Binary mode data connection for file 'large_bin.exe'
> ########################################################################
> ########################################################################
> ########################################################################
> ########################################################################
> ########################################################################
> ########################################################################
> receive aborted
> waiting for remote to finish abort
> 426 Connection closed; transfer aborted
> 226 ABOR command successful.
> 442368 bytes received in 35.2 secs (12 Kbytes/sec)
> ftp> exit
> 221 Goodbye.
>
>
> Here is the Profiler data that covers the timeframe of this activity
> (where tottime > 0)
>
> Ordered by: internal time
> List reduced from 446 to 100 due to restriction <100>
>
> ncalls tottime percall cumtime percall filename:lineno(function)
> 3 0.280 0.093 0.280 0.093
> ApplicationManager.py:183(refcount)
> 21 0.040 0.002 0.050 0.002
> Connection.py:598(_set_ghost_state)
> 812 0.040 0.000 0.040 0.000 DT_HTML.py:23(search)
> 21 0.040 0.002 0.040 0.002 FileStorage.py:651(_load)
> 1007 0.020 0.000 0.020 0.000
> HTTPRequest.py:1211(__getitem__)
> 16 0.020 0.001 0.310 0.019 DT_In.py:618(renderwob)
> 114 0.020 0.000 0.040 0.000 HTTPRequest.py:1231(keys)
> 473/197 0.020 0.000 0.020 0.000
> DT_Util.py:341(parse_params)
> 22 0.020 0.001 0.030 0.001
> PersistentExtra.py:23(bobobase_modification_time)
> 96/39 0.020 0.000 0.050 0.001
> DT_String.py:244(parse_close)
> 117 0.020 0.000 0.040 0.000 DT_Var.py:170(__init__)
> 1638 0.020 0.000 0.020 0.000 DT_Var.py:183(<lambda>)
> 725 0.020 0.000 0.030 0.000 DT_String.py:95(_parseTag)
> 181 0.010 0.000 0.010 0.000 DT_Util.py:229(name_param)
> 24 0.010 0.000 0.010 0.000
> DateTime.py:352(_julianday)
> 2 0.010 0.005 0.010 0.005
> ApplicationManager.py:285(version_txt)
> 134 0.010 0.000 0.010 0.000 DT_String.py:186(skip_eol)
> 19/17 0.010 0.001 0.010 0.001
> Traversable.py:95(getPhysicalPath)
> 249 0.010 0.000 0.010 0.000
> Connection.py:180(_persistent_load)
> 77/75 0.010 0.000 0.120 0.002 DT_Util.py:175(eval)
> 725 0.010 0.000 0.010 0.000 DT_HTML.py:129(start)
> 1158 0.010 0.000 0.010 0.000 HTTPRequest.py:1090(get)
> 7 0.010 0.001 0.010 0.001
> HTTPRequest.py:148(setServerURL)
> 8 0.010 0.001 0.010 0.001 posixpath.py:171(exists)
> 6 0.010 0.002 0.010 0.002
> sre_compile.py:229(_mk_bitmap)
> 725 0.010 0.000 0.010 0.000 DT_HTML.py:152(parseTag)
> 11 0.010 0.001 0.020 0.002
> VirtualHostMonster.py:130(__call__)
> 8 0.010 0.001 0.020 0.003 DT_String.py:514(read_raw)
> 59/20 0.010 0.000 0.180 0.009
> DT_String.py:195(parse_block)
> 2 0.010 0.005 0.200 0.100
> ApplicationManager.py:204(refdict)
> 78 0.010 0.000 0.010 0.000 User.py:171(allowed)
> 11 0.010 0.001 0.010 0.001
> HTTPResponse.py:266(setBody)
>
>
> TOP data, where %CPU > 0:
>
> PID USER PRI NI SIZE RSS SHARE STAT %CPU %MEM TIME CPU
> COMMAND
> 15732 apache 25 0 80292 78M 3624 R 99.8 7.7 1:25 0
> python2.3
> 15807 root 15 0 1252 1252 1016 S 0.2 0.1 0:00 0 ftp
> 1 root 15 0 476 448 424 S 0.0 0.0 0:04 0 init
>
>
> On Feb 26, 2004, at 10:40 AM, Chris McDonough wrote:
>
> > The 100% CPU spike is likely the cause of the slow performance. I
> > think
> > a reasonable strategy might be to turn on profiling in Zope 2.7 (see
> > zope.conf profile-publisher? key, and Control_Panel -> Debug) to see
> > what falls out of that. I'm sure there's a lot of information about
> > this if you google for "zope profiling".
> >
> > On Thu, 2004-02-26 at 08:26, Bobb wrote:
> >> Which OS / processor / memory are you running Edward ? Which network
> >> type ?
> >> Ethernet/speed, etc.
> >> Does the machine have an non-zope ftp server on it?
> >> If you use that do you get the same results?
> >> I find the 100% cpu spike even stranger than slow performance... (but
> >> that's
> >> just me ;) )
> >>
> >> ----- Original Message -----
> >> From: "Edward Pollard" <pollej at uleth.ca>
> >> To: "Bobb" <rawbobb at hotmail.com>
> >> Cc: <zope at zope.org>
> >> Sent: Wednesday, February 25, 2004 3:21 PM
> >> Subject: Re: [Zope] Zope 2.7 FTP Bandwidth Limiter?
> >>
> >>
> >> Indeed we still only get 10k/sec via ftp to localhost.
> >>
> >> It is worth noting that FTP upload is full-speed. Also noteworthy is
> >> that the CPU on this box is going to 100% when I download via FTP (45%
> >> user/55% system).
> >>
> >> This problem is exhibited, pretty much identically, in Zope 2.6.4 and
> >> 2.7.
> >>
> >> Zope 2.6.2 seems to have critical bugs in its FTP implementation,
> >> which
> >> means I needed to be off of it yesterday.
> >>
> >> • Dieter does not have this problem in 2.7, which may suggest that the
> >> problem is not Zope.
> >>
> >> • We are running different versions of Python (2.1 for Zope 2.6.4,
> >> 2.3.3 for Zope 2.7) for each of these servers, which may suggests it
> >> isn't a Python-related problem, unless there is something we did wrong
> >> twice. (And our 2.6.2 is running fine on Python 2.1, although it is a
> >> separate install of it than the one 2.6.4 uses).
> >>
> >> • Everything else runs at a proper transfer rate on this machine - and
> >> executing the 2.6.4 and 2.7 installs on the machine where we run 2.6.2
> >> as a process successfully yields the same results - which may suggest
> >> that the hardware is fine.
> >>
> >> Conclusion: I got nothing.
> >>
> >> Does anyone have any thoughts at all on this subject?
> >>
> >> Ed
> >>
> >>
> >> On Feb 22, 2004, at 11:51 AM, Bobb wrote:
> >>
> >>> I'm curious what about ftp'ing locally 127.0.0.1:ftpport ?
> >>> still 10k/sec?
> >>>
> >>> ----- Original Message -----
> >>> From: "Edward Pollard" <pollej at uleth.ca>
> >>> To: <zope at zope.org>
> >>> Sent: Saturday, February 21, 2004 4:20 PM
> >>> Subject: [Zope] Zope 2.7 FTP Bandwidth Limiter?
> >>>
> >>>
> >>>> We're experimenting with Zope 2.7, and are currently unable to get
> >>>> FTP
> >>>> transfers to go any faster that 10k/sec. This seems to be some sort
> >>>> of
> >>>> arbitrary limit, as we have plenty of bandwidth and horsepower
> >>>> available. Indeed, we can pretty much make as many FTP sessions as
> >>>> we
> >>>> want simultaneously, they just all go at 10k/sec.
> >>>>
> >>>> Was there some sort of limiter or throttle introduced somewhere
> >>>> along
> >>>> the lines that I am unaware of?
> >>>>
> >>>> ---
> >>>> Edward J. Pollard, B.Sc
> >>>> Webmaster, University of Lethbridge
> >>>>
> >>>>
> >>>> _______________________________________________
> >>>> Zope maillist - Zope at zope.org
> >>>> http://mail.zope.org/mailman/listinfo/zope
> >>>> ** No cross posts or HTML encoding! **
> >>>> (Related lists -
> >>>> http://mail.zope.org/mailman/listinfo/zope-announce
> >>>> http://mail.zope.org/mailman/listinfo/zope-dev )
> >>>>
> >>
> >>
> >> _______________________________________________
> >> Zope maillist - Zope at zope.org
> >> http://mail.zope.org/mailman/listinfo/zope
> >> ** No cross posts or HTML encoding! **
> >> (Related lists -
> >> http://mail.zope.org/mailman/listinfo/zope-announce
> >> http://mail.zope.org/mailman/listinfo/zope-dev )
> >>
> >>
> >> _______________________________________________
> >> Zope maillist - Zope at zope.org
> >> http://mail.zope.org/mailman/listinfo/zope
> >> ** No cross posts or HTML encoding! **
> >> (Related lists -
> >> http://mail.zope.org/mailman/listinfo/zope-announce
> >> http://mail.zope.org/mailman/listinfo/zope-dev )
> >
>
>
> _______________________________________________
> Zope maillist - Zope at zope.org
> http://mail.zope.org/mailman/listinfo/zope
> ** No cross posts or HTML encoding! **
> (Related lists -
> http://mail.zope.org/mailman/listinfo/zope-announce
> http://mail.zope.org/mailman/listinfo/zope-dev )
More information about the Zope
mailing list