120MB file import failures
Hello, I apologize if this is a FAQ, I could not find the answer. I would like to put a 120MB file into ZODB and it is constantly failing. TTW insert file fails but works on smaller fails. WebDAV fails but works on smaller files. Plus FTP fails having activate it as a last resort. My environment is FreeBSD 6 with current Zope, Python 2.3.x and a /tmp directory of 256MB... I noticed once that this could cause this failure. Any suggestions? I do have the file on the OS's filesystem but I see that the object "import" only applies to ZODB-ready files. Appreciated, Michael
Am Donnerstag, den 24.11.2005, 11:49 -0800 schrieb Michael Dexter:
Hello,
I apologize if this is a FAQ, I could not find the answer.
I would like to put a 120MB file into ZODB and it is constantly failing.
TTW insert file fails but works on smaller fails.
WebDAV fails but works on smaller files.
Plus FTP fails having activate it as a last resort.
My environment is FreeBSD 6 with current Zope, Python 2.3.x and a /tmp directory of 256MB... I noticed once that this could cause this failure.
Any suggestions?
I do have the file on the OS's filesystem but I see that the object "import" only applies to ZODB-ready files.
You could always: zopectl debug filedata=open("fileystempath/to/your/file") app.path_in_zope.manage_addFile("fileid",filedata) import transaction transaction.get().commit() Or use the uploader in the attachment (lets see if it goes thru the list) like this: ./bin/zopectl run fileimport.py bigfile /path/in/zope/ Greats Tino
Michael Dexter wrote:
I apologize if this is a FAQ, I could not find the answer.
I would like to put a 120MB file into ZODB and it is constantly failing.
Have a look at Tramlines from Infrae... cheers, Chris -- Simplistix - Content Management, Zope & Python Consulting - http://www.simplistix.co.uk
Michael Dexter wrote at 2005-11-24 11:49 -0800:
I apologize if this is a FAQ, I could not find the answer.
I would like to put a 120MB file into ZODB and it is constantly failing.
But you should have learned that you cannot simply say "is failing" but must describe precisely (error information, traceback) *how* it is failing... -- Dieter
Hello, I found the source of my problem uploading a large file to ZODB over DAV and an a workaround that shouldn't have worked but worked. For those who are interested: For completeness:
I would like to put a 120MB file into ZODB and it is constantly failing.
But you should have learned that you cannot simply say "is failing" but must describe precisely (error information, traceback) *how* it is failing...
Correct Dieter, I was vague but it indeed would progress and then fail. Goliath, a mac web dav client, would report "Unknown connection error" and the Finder would report "... cannot be completed... one or more required items not found (error code -43) I apologize that I am not familiar with Zope's traceback. Thank you to Tino for providing a script to copy files into ZODB. This solves a few other issues. So. I knew to watch my /tmp directory and it was not being used. It turns out that Python (presumably, not Zope) was using /var/tmp (over RAM, given that large-file-threshold in zope.conf was lower than 120MB). I linked /var/tmp to /tmp and oddly it started using /usr/??? but at least it worked because there was more than 120MB available there. I am not quite sure where it ended up putting it. (seemingly not /var in the zope instance) The solution appears to be that python can use the TMPDIR variable to specify where tmp should default to (reportedly it will try various locations) but, I am not clear how I can add this to Zope's rc script to declare it. I tried it as simply setting the environment variable "set TMPDIR=/tmp" but that gave warnings about the startup logic. 1. Should TMPDIR work? 2. Where should I declare it when using rc scripts? 3. Any chance zope.conf can use it? References: http://mail.zope.org/pipermail/zope-dev/2005-March/024534.html http://mail.zope.org/pipermail/zope-dev/2005-March/024532.html Appreciated, Michael.
On 6 Dec 2005, at 01:05, Michael Dexter wrote:
The solution appears to be that python can use the TMPDIR variable to specify where tmp should default to (reportedly it will try various locations) but, I am not clear how I can add this to Zope's rc script to declare it. I tried it as simply setting the environment variable "set TMPDIR=/tmp" but that gave warnings about the startup logic.
1. Should TMPDIR work? 2. Where should I declare it when using rc scripts? 3. Any chance zope.conf can use it?
zope.conf has a directive "environment" for exactly that purpose. jens
Michael Dexter wrote:
1. Should TMPDIR work?
Yes.
2. Where should I declare it when using rc scripts?
I'd probably put in the bin/runzope in your Zope instance.
3. Any chance zope.conf can use it?
no, unless I'm misunderstanding you... cheers, Chris -- Simplistix - Content Management, Zope & Python Consulting - http://www.simplistix.co.uk
Michael Dexter wrote at 2005-12-5 17:05 -0800:
... The solution appears to be that python can use the TMPDIR variable to specify where tmp should default to (reportedly it will try various locations) but, I am not clear how I can add this to Zope's rc script to declare it. I tried it as simply setting the environment variable "set TMPDIR=/tmp" but that gave warnings about the startup logic.
... 3. Any chance zope.conf can use it?
It might be too late to be effective. Set it in your environment (e.g. "zoperun") before you start Zope. -- Dieter
participants (5)
-
Chris Withers -
Dieter Maurer -
Jens Vagelpohl -
Michael Dexter -
Tino Wildenhain