Zope FTP - OverflowError: requested number of bytes is more than a Python string can hold - Zope

Zope FTP - OverflowError: requested number of bytes is more than a Python string can hold

Jan Schulze

4 Dec 2009 4 Dec '09

11:47 a.m.

Hi all, I am are using ZServer/Medusa for FTP-Uploads to our ZODB. This works for smaller files, but the above error appears in instance.log, when using a ~5 GB file. There's probably nothing I can do about it, as it's an internal python marshalling limit. Or would there be a way to make uploading big files work? This is the complete error message from instance.log: 2009-12-04T12:34:49 ERROR Zope.SiteErrorLog http://0.0.0.0/Zope2/sara/data/RudSoil_vs_lssu_160807.rma/PUT Traceback (innermost last): Module ZPublisher.Publish, line 119, in publish Module ZPublisher.mapply, line 88, in mapply Module ZPublisher.Publish, line 42, in call_object Module webdav.NullResource, line 167, in PUT Module Products.Archetypes.WebDAVSupport, line 116, in PUT Module Products.Archetypes.utils, line 160, in mapply Module Products.Marshall.marshaller, line 99, in demarshall OverflowError: requested number of bytes is more than a Python string can hold ------ 2009-12-04T12:34:57 ERROR ZServer uncaptured python exception, closing channel <ZServer.FTPServer.zope_ftp_channel connected 134.2.9.145:42267 at 0x1e5c6a70> (socket.error:(9, 'Bad file descriptor') [/raid/opt/Python-2.4.6/lib/python2.4/asynchat.py|initiate_send|219] [/raid/opt/Python-2.4.6/lib/python2.4/asyncore.py|send|332] [/raid/opt/Python-2.4.6/lib/python2.4/socket.py|_dummy|136]) Best Regards, Jan

Show replies by date

Andreas Jung

4 Dec 4 Dec

11:53 a.m.

New subject: [Zope] Zope FTP - OverflowError: requested number of bytes is more than a Python string can hold

You're trying to upload a 5GB file into Zope? Andreas Am 04.12.09 12:47, schrieb Jan Schulze:

...

Hi all,

I am are using ZServer/Medusa for FTP-Uploads to our ZODB. This works for smaller files, but the above error appears in instance.log, when using a ~5 GB file.

There's probably nothing I can do about it, as it's an internal python marshalling limit. Or would there be a way to make uploading big files work?

This is the complete error message from instance.log:

2009-12-04T12:34:49 ERROR Zope.SiteErrorLog http://0.0.0.0/Zope2/sara/data/RudSoil_vs_lssu_160807.rma/PUT Traceback (innermost last): Module ZPublisher.Publish, line 119, in publish Module ZPublisher.mapply, line 88, in mapply Module ZPublisher.Publish, line 42, in call_object Module webdav.NullResource, line 167, in PUT Module Products.Archetypes.WebDAVSupport, line 116, in PUT Module Products.Archetypes.utils, line 160, in mapply Module Products.Marshall.marshaller, line 99, in demarshall OverflowError: requested number of bytes is more than a Python string can hold ------ 2009-12-04T12:34:57 ERROR ZServer uncaptured python exception, closing channel <ZServer.FTPServer.zope_ftp_channel connected 134.2.9.145:42267 at 0x1e5c6a70> (socket.error:(9, 'Bad file descriptor') [/raid/opt/Python-2.4.6/lib/python2.4/asynchat.py|initiate_send|219] [/raid/opt/Python-2.4.6/lib/python2.4/asyncore.py|send|332] [/raid/opt/Python-2.4.6/lib/python2.4/socket.py|_dummy|136])

Best Regards, Jan _______________________________________________ Zope maillist - Zope@zope.org https://mail.zope.org/mailman/listinfo/zope ** No cross posts or HTML encoding! ** (Related lists - https://mail.zope.org/mailman/listinfo/zope-announce https://mail.zope.org/mailman/listinfo/zope-dev )

-- ZOPYX Ltd. & Co KG \ zopyx group Charlottenstr. 37/1 \ The full-service network for your D-72070 Tübingen \ Python, Zope and Plone projects www.zopyx.com, info@zopyx.com \ www.zopyxgroup.com ------------------------------------------------------------------------ E-Publishing, Python, Zope & Plone development, Consulting

Andreas Jung

11:59 a.m.

New subject: [Zope] Zope FTP - OverflowError: requested number of bytes is more than a Python string can hold

And here is a nice comment within marshaller.py: 93 def demarshall(self, obj, data, **kw):$ 94 if kw.has_key('file'):$ 95 if not data:$ 96 # XXX Yuck! Shouldn't read the whole file, never.$ 97 # OTOH, if you care about large files, you should be$ 98 # using the PrimaryFieldMarshaller or something$ 99 # similar.$ 100 data = kw['file'].read()$ 101 del kw['file']$ So when this is your own content-types: it's broken and should likely use PrimaryFieldMarshaller, otherwise: Plone is guilty one. In general I would upload large files to Zope/Plone only w/ blob support or iw.fss support (Plone) Andreas Am 04.12.09 12:53, schrieb Andreas Jung:

...

You're trying to upload a 5GB file into Zope?

Andreas

Am 04.12.09 12:47, schrieb Jan Schulze:

...
Hi all,

I am are using ZServer/Medusa for FTP-Uploads to our ZODB. This works for smaller files, but the above error appears in instance.log, when using a ~5 GB file.

There's probably nothing I can do about it, as it's an internal python marshalling limit. Or would there be a way to make uploading big files work?

This is the complete error message from instance.log:

2009-12-04T12:34:49 ERROR Zope.SiteErrorLog http://0.0.0.0/Zope2/sara/data/RudSoil_vs_lssu_160807.rma/PUT Traceback (innermost last): Module ZPublisher.Publish, line 119, in publish Module ZPublisher.mapply, line 88, in mapply Module ZPublisher.Publish, line 42, in call_object Module webdav.NullResource, line 167, in PUT Module Products.Archetypes.WebDAVSupport, line 116, in PUT Module Products.Archetypes.utils, line 160, in mapply Module Products.Marshall.marshaller, line 99, in demarshall OverflowError: requested number of bytes is more than a Python string can hold ------ 2009-12-04T12:34:57 ERROR ZServer uncaptured python exception, closing channel <ZServer.FTPServer.zope_ftp_channel connected 134.2.9.145:42267 at 0x1e5c6a70> (socket.error:(9, 'Bad file descriptor') [/raid/opt/Python-2.4.6/lib/python2.4/asynchat.py|initiate_send|219] [/raid/opt/Python-2.4.6/lib/python2.4/asyncore.py|send|332] [/raid/opt/Python-2.4.6/lib/python2.4/socket.py|_dummy|136])

Best Regards, Jan _______________________________________________ Zope maillist - Zope@zope.org https://mail.zope.org/mailman/listinfo/zope ** No cross posts or HTML encoding! ** (Related lists - https://mail.zope.org/mailman/listinfo/zope-announce https://mail.zope.org/mailman/listinfo/zope-dev )

_______________________________________________ Zope maillist - Zope@zope.org https://mail.zope.org/mailman/listinfo/zope ** No cross posts or HTML encoding! ** (Related lists - https://mail.zope.org/mailman/listinfo/zope-announce https://mail.zope.org/mailman/listinfo/zope-dev )

Tres Seaver

3:40 p.m.

New subject: [Zope] Zope FTP - OverflowError: requested number of bytes is more than a Python string can hold

-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1 Jan Schulze wrote:

...

I am are using ZServer/Medusa for FTP-Uploads to our ZODB. This works for smaller files, but the above error appears in instance.log, when using a ~5 GB file.

There's probably nothing I can do about it, as it's an internal python marshalling limit. Or would there be a way to make uploading big files work?

This is the complete error message from instance.log:

2009-12-04T12:34:49 ERROR Zope.SiteErrorLog http://0.0.0.0/Zope2/sara/data/RudSoil_vs_lssu_160807.rma/PUT Traceback (innermost last): Module ZPublisher.Publish, line 119, in publish Module ZPublisher.mapply, line 88, in mapply Module ZPublisher.Publish, line 42, in call_object Module webdav.NullResource, line 167, in PUT Module Products.Archetypes.WebDAVSupport, line 116, in PUT Module Products.Archetypes.utils, line 160, in mapply Module Products.Marshall.marshaller, line 99, in demarshall OverflowError: requested number of bytes is more than a Python string can hold ------ 2009-12-04T12:34:57 ERROR ZServer uncaptured python exception, closing channel <ZServer.FTPServer.zope_ftp_channel connected 134.2.9.145:42267 at 0x1e5c6a70> (socket.error:(9, 'Bad file descriptor') [/raid/opt/Python-2.4.6/lib/python2.4/asynchat.py|initiate_send|219] [/raid/opt/Python-2.4.6/lib/python2.4/asyncore.py|send|332] [/raid/opt/Python-2.4.6/lib/python2.4/socket.py|_dummy|136])

You need to be using a blob-aware field type here. This is not a Python / medusa problem, but an Archetypes / Marshall problem: check with the Plone folks to see if there is a supported solution for the versino of Plone you are running, or look at Chris McDonough's blob package: http://plope.com/software/blob/ Tres. - -- =================================================================== Tres Seaver +1 540-429-0999 tseaver@palladion.com Palladion Software "Excellence by Design" http://palladion.com -----BEGIN PGP SIGNATURE----- Version: GnuPG v1.4.9 (GNU/Linux) Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org iEYEARECAAYFAksZLXAACgkQ+gerLs4ltQ7AzQCghVeRiD3F1ETunBeQtwiZz636 f8sAnjmJNc46qxoSY2xP+NUJZEvgy9F6 =4jae -----END PGP SIGNATURE-----

5966

Age (days ago)

5966

Last active (days ago)

List overview

3 comments

3 participants

participants (3)

Andreas Jung
Jan Schulze
Tres Seaver