FWIW, I think I tracked this down to a too-small buffer size in the FTP server code. The latest code from CVS (Zope-2_7-branch) allows for much faster download speeds (~ 100X), at least for me. On Fri, 2004-02-27 at 11:59, Edward Pollard wrote:
The profiler seems to offer absolutely no information of use. I suspect it does not include Medusa performance.
The only thing that popped up was a call to FileStorage.py, but that call did not seem to monopolize the CPU.
Attached are three logs 1) The FTP session 2) The profiler data for that time period. 3) the TOP data for that period
Due to the bugs in 2.6.2, I've had to disable FTP access to the server, which has my clients infuriated. Any help at all with this would be most grateful. The only other option is to got back to 2.6.0, but that means we can't compress the ZODB without truncating it to 4 bytes.
Ed
230 Login successful. Remote system type is UNIX. Using binary mode to transfer files. ftp> put big_file.exe local: big_file.exe remote: big_file.exe 227 Entering Passive Mode (127,0,0,1,202,169) 150 Opening Binary connection for big_file.exe 226 Transfer complete. 18817367 bytes sent in 0.288 secs (6.4e+04 Kbytes/sec) ftp> get large_bin.exe local: large_bin.exe remote: large_bin.exe 227 Entering Passive Mode (127,0,0,1,202,156) 150 Opening Binary mode data connection for file 'large_bin.exe' ######################################################################## ######################################################################## ######################################################################## ######################################################################## ######################################################################## ######################################################################## receive aborted waiting for remote to finish abort 426 Connection closed; transfer aborted 226 ABOR command successful. 442368 bytes received in 35.2 secs (12 Kbytes/sec) ftp> exit 221 Goodbye.
Here is the Profiler data that covers the timeframe of this activity (where tottime > 0)
Ordered by: internal time List reduced from 446 to 100 due to restriction <100>
ncalls tottime percall cumtime percall filename:lineno(function) 3 0.280 0.093 0.280 0.093 ApplicationManager.py:183(refcount) 21 0.040 0.002 0.050 0.002 Connection.py:598(_set_ghost_state) 812 0.040 0.000 0.040 0.000 DT_HTML.py:23(search) 21 0.040 0.002 0.040 0.002 FileStorage.py:651(_load) 1007 0.020 0.000 0.020 0.000 HTTPRequest.py:1211(__getitem__) 16 0.020 0.001 0.310 0.019 DT_In.py:618(renderwob) 114 0.020 0.000 0.040 0.000 HTTPRequest.py:1231(keys) 473/197 0.020 0.000 0.020 0.000 DT_Util.py:341(parse_params) 22 0.020 0.001 0.030 0.001 PersistentExtra.py:23(bobobase_modification_time) 96/39 0.020 0.000 0.050 0.001 DT_String.py:244(parse_close) 117 0.020 0.000 0.040 0.000 DT_Var.py:170(__init__) 1638 0.020 0.000 0.020 0.000 DT_Var.py:183(<lambda>) 725 0.020 0.000 0.030 0.000 DT_String.py:95(_parseTag) 181 0.010 0.000 0.010 0.000 DT_Util.py:229(name_param) 24 0.010 0.000 0.010 0.000 DateTime.py:352(_julianday) 2 0.010 0.005 0.010 0.005 ApplicationManager.py:285(version_txt) 134 0.010 0.000 0.010 0.000 DT_String.py:186(skip_eol) 19/17 0.010 0.001 0.010 0.001 Traversable.py:95(getPhysicalPath) 249 0.010 0.000 0.010 0.000 Connection.py:180(_persistent_load) 77/75 0.010 0.000 0.120 0.002 DT_Util.py:175(eval) 725 0.010 0.000 0.010 0.000 DT_HTML.py:129(start) 1158 0.010 0.000 0.010 0.000 HTTPRequest.py:1090(get) 7 0.010 0.001 0.010 0.001 HTTPRequest.py:148(setServerURL) 8 0.010 0.001 0.010 0.001 posixpath.py:171(exists) 6 0.010 0.002 0.010 0.002 sre_compile.py:229(_mk_bitmap) 725 0.010 0.000 0.010 0.000 DT_HTML.py:152(parseTag) 11 0.010 0.001 0.020 0.002 VirtualHostMonster.py:130(__call__) 8 0.010 0.001 0.020 0.003 DT_String.py:514(read_raw) 59/20 0.010 0.000 0.180 0.009 DT_String.py:195(parse_block) 2 0.010 0.005 0.200 0.100 ApplicationManager.py:204(refdict) 78 0.010 0.000 0.010 0.000 User.py:171(allowed) 11 0.010 0.001 0.010 0.001 HTTPResponse.py:266(setBody)
TOP data, where %CPU > 0:
PID USER PRI NI SIZE RSS SHARE STAT %CPU %MEM TIME CPU COMMAND 15732 apache 25 0 80292 78M 3624 R 99.8 7.7 1:25 0 python2.3 15807 root 15 0 1252 1252 1016 S 0.2 0.1 0:00 0 ftp 1 root 15 0 476 448 424 S 0.0 0.0 0:04 0 init
On Feb 26, 2004, at 10:40 AM, Chris McDonough wrote:
The 100% CPU spike is likely the cause of the slow performance. I think a reasonable strategy might be to turn on profiling in Zope 2.7 (see zope.conf profile-publisher? key, and Control_Panel -> Debug) to see what falls out of that. I'm sure there's a lot of information about this if you google for "zope profiling".
On Thu, 2004-02-26 at 08:26, Bobb wrote:
Which OS / processor / memory are you running Edward ? Which network type ? Ethernet/speed, etc. Does the machine have an non-zope ftp server on it? If you use that do you get the same results? I find the 100% cpu spike even stranger than slow performance... (but that's just me ;) )
----- Original Message ----- From: "Edward Pollard" <pollej@uleth.ca> To: "Bobb" <rawbobb@hotmail.com> Cc: <zope@zope.org> Sent: Wednesday, February 25, 2004 3:21 PM Subject: Re: [Zope] Zope 2.7 FTP Bandwidth Limiter?
Indeed we still only get 10k/sec via ftp to localhost.
It is worth noting that FTP upload is full-speed. Also noteworthy is that the CPU on this box is going to 100% when I download via FTP (45% user/55% system).
This problem is exhibited, pretty much identically, in Zope 2.6.4 and 2.7.
Zope 2.6.2 seems to have critical bugs in its FTP implementation, which means I needed to be off of it yesterday.
• Dieter does not have this problem in 2.7, which may suggest that the problem is not Zope.
• We are running different versions of Python (2.1 for Zope 2.6.4, 2.3.3 for Zope 2.7) for each of these servers, which may suggests it isn't a Python-related problem, unless there is something we did wrong twice. (And our 2.6.2 is running fine on Python 2.1, although it is a separate install of it than the one 2.6.4 uses).
• Everything else runs at a proper transfer rate on this machine - and executing the 2.6.4 and 2.7 installs on the machine where we run 2.6.2 as a process successfully yields the same results - which may suggest that the hardware is fine.
Conclusion: I got nothing.
Does anyone have any thoughts at all on this subject?
Ed
On Feb 22, 2004, at 11:51 AM, Bobb wrote:
I'm curious what about ftp'ing locally 127.0.0.1:ftpport ? still 10k/sec?
----- Original Message ----- From: "Edward Pollard" <pollej@uleth.ca> To: <zope@zope.org> Sent: Saturday, February 21, 2004 4:20 PM Subject: [Zope] Zope 2.7 FTP Bandwidth Limiter?
We're experimenting with Zope 2.7, and are currently unable to get FTP transfers to go any faster that 10k/sec. This seems to be some sort of arbitrary limit, as we have plenty of bandwidth and horsepower available. Indeed, we can pretty much make as many FTP sessions as we want simultaneously, they just all go at 10k/sec.
Was there some sort of limiter or throttle introduced somewhere along the lines that I am unaware of?
--- Edward J. Pollard, B.Sc Webmaster, University of Lethbridge
_______________________________________________ Zope maillist - Zope@zope.org http://mail.zope.org/mailman/listinfo/zope ** No cross posts or HTML encoding! ** (Related lists - http://mail.zope.org/mailman/listinfo/zope-announce http://mail.zope.org/mailman/listinfo/zope-dev )
_______________________________________________ Zope maillist - Zope@zope.org http://mail.zope.org/mailman/listinfo/zope ** No cross posts or HTML encoding! ** (Related lists - http://mail.zope.org/mailman/listinfo/zope-announce http://mail.zope.org/mailman/listinfo/zope-dev )
_______________________________________________ Zope maillist - Zope@zope.org http://mail.zope.org/mailman/listinfo/zope ** No cross posts or HTML encoding! ** (Related lists - http://mail.zope.org/mailman/listinfo/zope-announce http://mail.zope.org/mailman/listinfo/zope-dev )
_______________________________________________ Zope maillist - Zope@zope.org http://mail.zope.org/mailman/listinfo/zope ** No cross posts or HTML encoding! ** (Related lists - http://mail.zope.org/mailman/listinfo/zope-announce http://mail.zope.org/mailman/listinfo/zope-dev )