Re: [Zope] uploading large files w/ LocalFS
Yes, big upload/download with zserver is very bad, and https is not a good idea too ... Gilles ----- Original Message ----- From: "Jamie Heilman" <jamie@audible.transient.net> To: "Mike Doanh Tran" <mtran@shufflemasterrd.com> Cc: <zope@zope.org> Sent: Tuesday, May 27, 2003 6:26 PM Subject: Re: [Zope] uploading large files w/ LocalFS
Mike Doanh Tran wrote:
I am trying to allow remote developers to upload large development tarballs onto our web server using LocalFS. The problem is when upload files are larger than 20MB the server times out. Does anyone have a better solution of allowing users to upload large files onto a Linux based webserver? Is https the right protocol for large file transfer?
Are you uploading the files through ZServer? If so, don't.
-- Jamie Heilman http://audible.transient.net/~jamie/ "We must be born with an intuition of mortality. Before we know the words for it, before we know there are words, out we come bloodied and
squalling
with the knowledge that for all the compasses in the world, there's only one direction, and time is its only measure." -Rosencrantz
_______________________________________________ Zope maillist - Zope@zope.org http://mail.zope.org/mailman/listinfo/zope ** No cross posts or HTML encoding! ** (Related lists - http://mail.zope.org/mailman/listinfo/zope-announce http://mail.zope.org/mailman/listinfo/zope-dev )
On Wed, May 28, 2003 at 11:32:19AM -0500, Gilles wrote:
Yes, big upload/download with zserver is very bad,
This could use a little more discussion I think. * Why is it bad? I can think of one reason: because each upload will keep one thread and one ZODB connection busy as long as it's going, and zope has a fixed number of threads and ZODB connections (set at startup time). * What are the alternatives? It's easy to say "use Apache", but this is rather glib if you're working on a content management system for which upload of large files is a requirement. And using an external browser means you lose so many nice things like zope's security management. I've thought it would be nice to have something like "ExtBrowserFile" which would work as follows: * select "ExtBrowserFile" from the "Add..." menu (zmi or cmf). * Form has a typical file upload widget * Target of form is a (configurable) external upload script running on e.g. apache * Zope stores only a small object which contains properties such as title, image height & width (not sure how to get these), and most importantly, a valid URL at which the image or file can be viewed. That solves a lot of the problems, but not security. You can of course secure the litle meta-object in Zope but what about the actual data? -- Paul Winkler http://www.slinkp.com Look! Up in the sky! It's THE ADAMANTIUM! (random hero from isometric.spaceninja.com)
On Wed, 2003-05-28 at 02:26, Paul Winkler wrote:
On Wed, May 28, 2003 at 11:32:19AM -0500, Gilles wrote:
Yes, big upload/download with zserver is very bad,
This could use a little more discussion I think. * Why is it bad? I can think of one reason: because each upload will keep one thread and one ZODB connection busy as long as it's going, and zope has a fixed number of threads and ZODB connections (set at startup time).
Good points, to be sure. As I gather, there is a more fundamental reason: HTTP upload just isn't that solid to begin with.
* What are the alternatives? It's easy to say "use Apache",
And fun too! :-)
but this is rather glib if you're working on a content management system for which upload of large files is a requirement.
Indeed. My answer to that is "provide/use SSH". Clearly there needs to be some way of uploading large files and FTP just isn't it any more.
And using an external browser means you lose so many nice things like zope's security management.
Though SSH won't use the Zope security model (that I know of), key-based authentication isn't that onerous to manage and/or deploy. $.02, Dylan
On Wed, May 28, 2003 at 08:23:53AM -0700, Dylan Reinhardt wrote:
And using an external browser means oops, typo... s/browser/server/ you lose so many nice things like zope's security management.
Though SSH won't use the Zope security model (that I know of), key-based authentication isn't that onerous to manage and/or deploy.
I meant for managing access to the downloadable resources. If some content is in zope and some is available via apache, and some of the apache content needs to be restricted to some users, it's a lot of extra work to make this transparent to your end users. I don't really know how I'd deal with that. -- Paul Winkler http://www.slinkp.com Look! Up in the sky! It's INTRA-GOBLET ABOVE TIME! (random hero from isometric.spaceninja.com)
On Wed, 2003-05-28 at 04:39, Paul Winkler wrote:
On Wed, May 28, 2003 at 08:23:53AM -0700, Dylan Reinhardt wrote:
And using an external browser means oops, typo... s/browser/server/ you lose so many nice things like zope's security management.
Though SSH won't use the Zope security model (that I know of), key-based authentication isn't that onerous to manage and/or deploy.
I meant for managing access to the downloadable resources.
Ah... the subject line threw me off a bit. :-)
If some content is in zope and some is available via apache, and some of the apache content needs to be restricted to some users, it's a lot of extra work to make this transparent to your end users.
Yeah, that's a tough one.
I don't really know how I'd deal with that.
I punted when I came across this problem in one of my apps... just ended up serving all private/restricted content from the ZMI. Not very elegant, but my requirements weren't large enough scale to attempt anything more complex. I've been toying with Twisted a bit lately and wonder if it might be used to create a helper app for Zope that functions as a separate web server that can exchange/lookup some kind of authentication token with Zope. Large files could then be served on a separate port and restricted to those with a cookie and/or URL munge approved by Zope. This hasn't gotten anywhere near my radar screen... but it seems like something that I might attempt if/when this need crops up next time. Dylan
On Wed, May 28, 2003 at 09:56:00AM -0700, Dylan Reinhardt wrote:
I've been toying with Twisted a bit lately and wonder if it might be used to create a helper app for Zope that functions as a separate web server that can exchange/lookup some kind of authentication token with Zope.
hmm... something like a TwistedCredUserFolder? ;-)
Large files could then be served on a separate port and restricted to those with a cookie and/or URL munge approved by Zope.
that could work. Either that or run zope *behind* twisted and implement some kind of authentication pass-through (single sign-on). I dunno whether that's feasible. On a somewhat related note, I've also been annoyed lately that zope supports FTP but not SFTP. I've looked at the SFTP specification and it looks like a couple days' work to do in Twisted, maybe less if you know what you're doing :-) So I'm toying with the idea of: 1) adding sftp support for twisted ... probably this means leveraging Twisted Conch (ssh). 2) writing a twisted server that makes calls to a Zope.app() instance so data uploaded to this twisted server can be handled with zope's ftp PUT support. I guess that this would require some care with synchronizing transactions (via get_transaction.commit() and _p_jar.sync()) I haven't really thought this through. The alternative would be hacking SFTP support into Zope itself and I don't know enough yet to know if that would be easier or harder.
This hasn't gotten anywhere near my radar screen...
ditto. -- Paul Winkler http://www.slinkp.com Look! Up in the sky! It's SUSPICIOUS GROUPIE SARDAUKAR! (random hero from isometric.spaceninja.com)
participants (3)
-
Dylan Reinhardt -
Gilles -
Paul Winkler