[ZCM] [ZC] 1342/ 6 Comment "ZServer/FTPServer.py
exceptions.MemoryError"
Collector: Zope Bugs, Features,
and Patches ...
zope-coders-admin at zope.org
Thu May 20 11:08:29 EDT 2004
Issue #1342 Update (Comment) "ZServer/FTPServer.py exceptions.MemoryError"
Status Pending, ZServer/bug low
To followup, visit:
http://collector.zope.org/Zope/1342
==============================================================
= Comment - Entry #6 by shiao on May 20, 2004 11:08 am
The following is the top summary before the tests.
load averages: 0.10, 0.14, 0.10 10:54:52
81 processes: 1 running, 79 idle, 1 zombie
CPU states: 0.2% user, 0.0% nice, 0.0% system, 0.0% interrupt, 99.8% idle
Memory: Real: 117M/162M act/tot Free: 83M Swap: 0K/1024M used/tot
After running zope:
load averages: 0.42, 0.22, 0.13 10:55:47
82 processes: 1 running, 80 idle, 1 zombie
CPU states: 2.2% user, 0.0% nice, 1.1% system, 0.0% interrupt, 96.7% idle
Memory: Real: 132M/178M act/tot Free: 67M Swap: 0K/1024M used/tot
Now the test is to upload 73 images via ftp to a newly created folder with aprox. 23M of data.
$ du -s
223720 .
$ ftp localhost 10021
...
ftp> mkdir upload
257 MKD command successful.
ftp> cd upload
250 CWD command successful.
ftp> passiv on
Passive mode on.
ftp> prompt
Interactive mode off
ftp> mput *.jpg
227 Entering Passive Mode (127,0,0,1,191,46)
150 Opening Binary connection for img_1801.jpg
100% |*****************************************************************************************************************************| 342 KB 00:00
226 Transfer complete.
350785 bytes sent in 1.03 seconds (331.46 KB/s)
local: img_1802.jpg remote: img_1802.jpg
227 Entering Passive Mode (127,0,0,1,5,20)
150 Opening Binary connection for img_1802.jpg
Yes! While uploading the files top showed how Free Memory went down to 16M:
load averages: 0.29, 0.32, 0.21 11:04:18
86 processes: 1 running, 84 idle, 1 zombie
CPU states: 0.0% user, 0.0% nice, 0.3% system, 0.0% interrupt, 99.7% idle
Memory: Real: 183M/229M act/tot Free: 16M Swap: 0K/1024M used/tot
Stoping the zope instance Free memory went back to 81M.
The questions now are, how is this happening? Are all the files uploaded stored in memory? When is the memory deallocated? Is it possible to flush it to disk? Is there any test I can do?
Thanks!
________________________________________
= Comment - Entry #5 by mcdonc on May 20, 2004 12:18 am
It looks like your machine isn't able to allocate Python/Zope any memory (you're running out of RAM somehow).
________________________________________
= Comment - Entry #4 by shiao on May 20, 2004 12:02 am
I guess that 0x1000000 should say 0xF4240.
________________________________________
= Comment - Entry #3 by shiao on May 19, 2004 11:57 pm
OpenBSD will compile with the following options
-DNDEBUG -O2 -DTHREAD_STACK_SIZE=0x20000 -fPIC
I compile a new version with DTHREAD_STACK_SIZE=0x1000000
python version is
Python 2.3.3 (#1, May 19 2004, 23:31:25)
[GCC 2.95.3 20010125 (prerelease, propolice)] on openbsd3
And ulimit has the following settings.
$ ulimit -a
time(cpu-seconds) unlimited
file(blocks) unlimited
coredump(blocks) unlimited
data(kbytes) 65536
stack(kbytes) 4096
lockedmem(kbytes) 78305
memory(kbytes) 233452
nofiles(descriptors) 64
processes 64
runzope and trying to ftp some files it reports the following:
500 'EPSV': command not understood.
227 Entering Passive Mode (127,0,0,1,57,24)
451 Server Error: thread.error, can't start new thread
Changing ulimit -a
$ ulimit -a
time(cpu-seconds) unlimited
file(blocks) unlimited
coredump(blocks) unlimited
data(kbytes) 100000
stack(kbytes) 9128
lockedmem(kbytes) 78305
memory(kbytes) 233452
nofiles(descriptors) 128
processes 64
I am able to upload some files, but then event.log shows:
2004-05-19T23:51:26 ERROR(200) ZServer uncaptured python exception, closing channel <ZServer.medusa.ftp_server.recv_channel connected 127.0.0.1:46297 at 0x3cd67eec> (exceptions.MemoryError:out of memory [/usr/local/lib/python2.3/asyncore.py|read|69] [/usr/local/lib/python2.3/asyncore.py|handle_read_event|390] [/usr/home/shiao/prog/Zope-2.7.0/lib/python/ZServer/medusa/ftp_server.py|handle_read|937] [/usr/home/shiao/prog/Zope-2.7.0/lib/python/ZServer/FTPServer.py|write|543]
In this case, 25 were uploaded ok.
Note that python does not core dump, and ZServer runs just fine for new connections. I will try to upload one file at a time (manually) to see if it works. Maybe the dealy between files makes a differences (?)
________________________________________
= Comment - Entry #2 by camil7 on May 19, 2004 3:19 pm
Chances are this has something to do with the "too small default stack size" issue on BSD .
Please read the Zope/doc/PLATFORMS/BSD.txt which
should be shipped with the Zope documentation
(at least in the sources).
________________________________________
= Request - Entry #1 by shiao on May 18, 2004 11:47 pm
OS: OpenBSD 3.5
Zope: 2.7.0
Python 2.3.3 (#1, Mar 23 2004, 16:24:51)
[GCC 2.95.3 20010125 (prerelease, propolice)] on openbsd3
The following exception occurs when uploading files (.jpg) via ftp passive to a newly created instance. This does not occur using the same test on a Gentoo 2.6.5/Python 2.3.3/Zope 2.7.0. I believe it should be something about python. Is there a way to confirm this?
2004-05-18T23:35:50 ERROR(200) ZServer uncaptured python exception, closing channel <ZServer.medusa.ftp_server.recv_channel connected at 0x3cefc1ac> (exceptions.MemoryError:out of memory [/usr/local/lib/python2.3/asyncore.py|read|69] [/usr/local/lib/python2.3/asyncore.py|handle_read_event|390] [/usr/home/shiao/prog/Zope-2.7.0/lib/python/ZServer/medusa/ftp_server.py|handle_read|937] [/usr/home/shiao/prog/Zope-2.7.0/lib/python/ZServer/FTPServer.py|write|543])
==============================================================
More information about the Zope-Collector-Monitor
mailing list