[Zope] Zope 2.7 FTP Bandwidth Limiter?

Edward Pollard pollej at uleth.ca
Fri Feb 27 11:59:14 EST 2004 

The profiler seems to offer absolutely no information of use. I suspect  
it does not include Medusa performance.

The only thing that popped up was a call to FileStorage.py, but that  
call did not seem to monopolize the CPU.

Attached are three logs
1) The FTP session
2) The profiler data for that time period.
3) the TOP data for that period


Due to the bugs in 2.6.2, I've had to disable FTP access to the server,  
which has my clients infuriated. Any help at all with this would be  
most grateful. The only other option is to got back to 2.6.0, but that  
means we can't compress the ZODB without truncating it to 4 bytes.

Ed

230 Login successful.
Remote system type is UNIX.
Using binary mode to transfer files.
ftp> put big_file.exe
local: big_file.exe remote: big_file.exe
227 Entering Passive Mode (127,0,0,1,202,169)
150 Opening Binary connection for big_file.exe
226 Transfer complete.
18817367 bytes sent in 0.288 secs (6.4e+04 Kbytes/sec)
ftp> get large_bin.exe
local: large_bin.exe remote: large_bin.exe
227 Entering Passive Mode (127,0,0,1,202,156)
150 Opening Binary mode data connection for file 'large_bin.exe'
######################################################################## 
######################################################################## 
######################################################################## 
######################################################################## 
######################################################################## 
########################################################################
receive aborted
waiting for remote to finish abort
426 Connection closed; transfer aborted
226 ABOR command successful.
442368 bytes received in 35.2 secs (12 Kbytes/sec)
ftp> exit
221 Goodbye.


Here is the Profiler data that covers the timeframe of this activity  
(where tottime > 0)

Ordered by: internal time
    List reduced from 446 to 100 due to restriction <100>

    ncalls  tottime  percall  cumtime  percall filename:lineno(function)
         3    0.280    0.093    0.280    0.093  
ApplicationManager.py:183(refcount)
        21    0.040    0.002    0.050    0.002  
Connection.py:598(_set_ghost_state)
       812    0.040    0.000    0.040    0.000 DT_HTML.py:23(search)
        21    0.040    0.002    0.040    0.002 FileStorage.py:651(_load)
      1007    0.020    0.000    0.020    0.000  
HTTPRequest.py:1211(__getitem__)
        16    0.020    0.001    0.310    0.019 DT_In.py:618(renderwob)
       114    0.020    0.000    0.040    0.000 HTTPRequest.py:1231(keys)
   473/197    0.020    0.000    0.020    0.000  
DT_Util.py:341(parse_params)
        22    0.020    0.001    0.030    0.001  
PersistentExtra.py:23(bobobase_modification_time)
     96/39    0.020    0.000    0.050    0.001  
DT_String.py:244(parse_close)
       117    0.020    0.000    0.040    0.000 DT_Var.py:170(__init__)
      1638    0.020    0.000    0.020    0.000 DT_Var.py:183(<lambda>)
       725    0.020    0.000    0.030    0.000 DT_String.py:95(_parseTag)
       181    0.010    0.000    0.010    0.000 DT_Util.py:229(name_param)
        24    0.010    0.000    0.010    0.000  
DateTime.py:352(_julianday)
         2    0.010    0.005    0.010    0.005  
ApplicationManager.py:285(version_txt)
       134    0.010    0.000    0.010    0.000 DT_String.py:186(skip_eol)
     19/17    0.010    0.001    0.010    0.001  
Traversable.py:95(getPhysicalPath)
       249    0.010    0.000    0.010    0.000  
Connection.py:180(_persistent_load)
     77/75    0.010    0.000    0.120    0.002 DT_Util.py:175(eval)
       725    0.010    0.000    0.010    0.000 DT_HTML.py:129(start)
      1158    0.010    0.000    0.010    0.000 HTTPRequest.py:1090(get)
         7    0.010    0.001    0.010    0.001  
HTTPRequest.py:148(setServerURL)
         8    0.010    0.001    0.010    0.001 posixpath.py:171(exists)
         6    0.010    0.002    0.010    0.002  
sre_compile.py:229(_mk_bitmap)
       725    0.010    0.000    0.010    0.000 DT_HTML.py:152(parseTag)
        11    0.010    0.001    0.020    0.002  
VirtualHostMonster.py:130(__call__)
         8    0.010    0.001    0.020    0.003 DT_String.py:514(read_raw)
     59/20    0.010    0.000    0.180    0.009  
DT_String.py:195(parse_block)
         2    0.010    0.005    0.200    0.100  
ApplicationManager.py:204(refdict)
        78    0.010    0.000    0.010    0.000 User.py:171(allowed)
        11    0.010    0.001    0.010    0.001  
HTTPResponse.py:266(setBody)


TOP data, where %CPU > 0:

   PID USER     PRI  NI  SIZE  RSS SHARE STAT %CPU %MEM   TIME CPU  
COMMAND
15732 apache    25   0 80292  78M  3624 R    99.8  7.7   1:25   0  
python2.3
15807 root      15   0  1252 1252  1016 S     0.2  0.1   0:00   0 ftp
     1 root      15   0   476  448   424 S     0.0  0.0   0:04   0 init


On Feb 26, 2004, at 10:40 AM, Chris McDonough wrote:

> The 100% CPU spike is likely the cause of the slow performance.  I  
> think
> a reasonable strategy might be to turn on profiling in Zope 2.7 (see
> zope.conf profile-publisher? key, and Control_Panel -> Debug) to see
> what falls out of that.  I'm sure there's a lot of information about
> this if you google for "zope profiling".
>
> On Thu, 2004-02-26 at 08:26, Bobb wrote:
>> Which OS / processor / memory are you running Edward ? Which network  
>> type ?
>> Ethernet/speed, etc.
>> Does the machine have an non-zope ftp server on it?
>> If you use that do you get the same results?
>> I find the 100% cpu spike even stranger than slow performance... (but  
>> that's
>> just me ;) )
>>
>> ----- Original Message -----
>> From: "Edward Pollard" <pollej at uleth.ca>
>> To: "Bobb" <rawbobb at hotmail.com>
>> Cc: <zope at zope.org>
>> Sent: Wednesday, February 25, 2004 3:21 PM
>> Subject: Re: [Zope] Zope 2.7 FTP Bandwidth Limiter?
>>
>>
>> Indeed we still only get 10k/sec via ftp to localhost.
>>
>> It is worth noting that FTP upload is full-speed. Also noteworthy is
>> that the CPU on this box is going to 100% when I download via FTP (45%
>> user/55% system).
>>
>> This problem is exhibited, pretty much identically, in Zope 2.6.4 and
>> 2.7.
>>
>> Zope 2.6.2 seems to have critical bugs in its FTP implementation,  
>> which
>> means I needed to be off of it yesterday.
>>
>> • Dieter does not have this problem in 2.7, which may suggest that the
>> problem is not Zope.
>>
>> • We are running different versions of Python (2.1 for Zope 2.6.4,
>> 2.3.3 for Zope 2.7) for each of these servers, which may suggests it
>> isn't a Python-related problem, unless there is something we did wrong
>> twice. (And our 2.6.2 is running fine on Python 2.1, although it is a
>> separate install of it than the one 2.6.4 uses).
>>
>> • Everything else runs at a proper transfer rate on this machine - and
>> executing the 2.6.4 and 2.7 installs on the machine where we run 2.6.2
>> as a process successfully yields the same results - which may suggest
>> that the hardware is fine.
>>
>> Conclusion: I got nothing.
>>
>> Does anyone have any thoughts at all on this subject?
>>
>> Ed
>>
>>
>> On Feb 22, 2004, at 11:51 AM, Bobb wrote:
>>
>>> I'm curious what about ftp'ing locally 127.0.0.1:ftpport ?
>>> still 10k/sec?
>>>
>>> ----- Original Message -----
>>> From: "Edward Pollard" <pollej at uleth.ca>
>>> To: <zope at zope.org>
>>> Sent: Saturday, February 21, 2004 4:20 PM
>>> Subject: [Zope] Zope 2.7 FTP Bandwidth Limiter?
>>>
>>>
>>>> We're experimenting with Zope 2.7, and are currently unable to get  
>>>> FTP
>>>> transfers to go any faster that 10k/sec. This seems to be some sort  
>>>> of
>>>> arbitrary limit, as we have plenty of bandwidth and horsepower
>>>> available. Indeed, we can pretty much make as many FTP sessions as  
>>>> we
>>>> want simultaneously, they just all go at 10k/sec.
>>>>
>>>> Was there some sort of limiter or throttle introduced somewhere  
>>>> along
>>>> the lines that I am unaware of?
>>>>
>>>> ---
>>>> Edward J. Pollard, B.Sc
>>>> Webmaster, University of Lethbridge
>>>>
>>>>
>>>> _______________________________________________
>>>> Zope maillist  -  Zope at zope.org
>>>> http://mail.zope.org/mailman/listinfo/zope
>>>> **   No cross posts or HTML encoding!  **
>>>> (Related lists -
>>>>  http://mail.zope.org/mailman/listinfo/zope-announce
>>>>  http://mail.zope.org/mailman/listinfo/zope-dev )
>>>>
>>
>>
>> _______________________________________________
>> Zope maillist  -  Zope at zope.org
>> http://mail.zope.org/mailman/listinfo/zope
>> **   No cross posts or HTML encoding!  **
>> (Related lists -
>>  http://mail.zope.org/mailman/listinfo/zope-announce
>>  http://mail.zope.org/mailman/listinfo/zope-dev )
>>
>>
>> _______________________________________________
>> Zope maillist  -  Zope at zope.org
>> http://mail.zope.org/mailman/listinfo/zope
>> **   No cross posts or HTML encoding!  **
>> (Related lists -
>>  http://mail.zope.org/mailman/listinfo/zope-announce
>>  http://mail.zope.org/mailman/listinfo/zope-dev )
>

More information about the Zope mailing list