Using urllib2 to access Zope URLs
I'd like to write a script that I can run from the command line -- NOT a Python script object in Zope or even an external method -- that I can run to upload files into Zope. Here's some code that I'm using right now that, according to the documentation and examples I've seen for the urllib2 library, OUGHT to work: -------------------- import urllib2 pwdmgr = urllib2.HTTPPasswordMgrWithDefaultRealm() pwdmgr.add_password(None, 'http://localhost:8080/", 'myusername', 'mypassword') auth = urllib2.HTTPBasicAuthHandler(pwdmgr) opener = urllib2.build_opener(auth) urllib2.install_opener(opener) req = urllib2.Request('http://localhost:8080/manage_main') f = urllib2.urlopen(req) print f.read() f.close() -------------------- The line with the urlopen command always raises an error, however: urllib2.HTTPError: HTTP Error 401: Unauthorized The same script has no trouble obtaining the results of queries to unprotected sites, but seems completely incapable of getting past the Zope authorization. FYI: Tests with the wget utility show that it is possible to get past the authorization, so the problem lies somewhere with this script. What is it that I'm missing? --Damon
On Tue, 2003-05-20 at 15:31, Dylan Reinhardt wrote:
On Tue, 2003-05-20 at 12:53, Damon Butler wrote:
Tests with the wget utility show that it is possible to get past the authorization
Oh? Has anyone else seen this?
Whaddya mean? The following command returns the proper HTML: wget --http-user=username --http-passwd=password http://localhost:8080/manage_main This tells me that it is possible to supply the username and password programatically from outside the Zope framework and (presumably) get some real work done. But the authorization doesn't carry through under python.urllib2. Why is that? --Damon
Sorry... I thought you were saying you had found a way to *circumvent* the authentication. Good to hear that wasn't what you meant. I'm not sure why it doesn't work for urllib2... I've only used that for making POST requests, I've never used it with HTTP authentication. Perhaps someone on comp.lang.python would know if nobody here speaks up... it's more of a general Python question. Dylan On Tue, 2003-05-20 at 13:25, Damon Butler wrote:
On Tue, 2003-05-20 at 15:31, Dylan Reinhardt wrote:
On Tue, 2003-05-20 at 12:53, Damon Butler wrote:
Tests with the wget utility show that it is possible to get past the authorization
Oh? Has anyone else seen this?
Whaddya mean? The following command returns the proper HTML:
wget --http-user=username --http-passwd=password http://localhost:8080/manage_main
This tells me that it is possible to supply the username and password programatically from outside the Zope framework and (presumably) get some real work done. But the authorization doesn't carry through under python.urllib2. Why is that?
--Damon
Damon Butler wrote at 2003-5-20 14:53 -0500:
I'd like to write a script that I can run from the command line -- NOT a Python script object in Zope or even an external method -- that I can run to upload files into Zope. Here's some code that I'm using right now that, according to the documentation and examples I've seen for the urllib2 library, OUGHT to work:
I use "ZPublisher.Client". It is much easier then "urllib2". Dieter
On Wed, 2003-05-21 at 13:47, Dieter Maurer wrote:
Damon Butler wrote at 2003-5-20 14:53 -0500:
I'd like to write a script that I can run from the command line -- NOT a Python script object in Zope or even an external method -- that I can run to upload files into Zope. Here's some code that I'm using right now that, according to the documentation and examples I've seen for the urllib2 library, OUGHT to work:
I use "ZPublisher.Client". It is much easier then "urllib2".
Thanks for the tip. ZPublisher.Client looks very promising, but it doesn't work for me. It looks as if the program is choking on something. Very small files (text files or jpegs are what I tested) will successfully upload, but the script always chokes with an error anyway. Trying to upload "large" files (anything over a few dozen kb, say) cause the script to just hang and spin its wheels. I have to kill the process. The file does not get uploaded and no Traceback is created. Here's a sample command line and the resultant Traceback from an attempt to upload a small text file: ==================== % python Client.py -u me:mypasswd http://localhost:8080/manage_addFile id=smallTextFile.txt file:file=/home/damon/tmp/smallTextFile.txt Traceback (most recent call last): File "./Client.py", line 563, in ? main() File "./Client.py", line 556, in main headers, body = apply(f,(),kw) File "./Client.py", line 109, in __call__ if hasattr(v,'read'): return self._mp_call(kw) File "./Client.py", line 231, in _mp_call self.handleError('', ec, em, headers, response) File "./Client.py", line 170, in handleError raise t, RemoteException(t,v,f,l,self.url,query,ec,em,response) bci.ServerError: 302 (File: Unknown Line: Unknown) 302 Moved Temporarily for http://localhost:8080/manage_addFile ==================== Any idea what's happening? --Damon
Damon Butler wrote at 2003-5-27 15:15 -0500:
... Here's a sample command line and the resultant Traceback from an attempt to upload a small text file:
==================== % python Client.py -u me:mypasswd http://localhost:8080/manage_addFile id=smallTextFile.txt file:file=/home/damon/tmp/smallTextFile.txt
Traceback (most recent call last): File "./Client.py", line 563, in ? main() File "./Client.py", line 556, in main headers, body = apply(f,(),kw) File "./Client.py", line 109, in __call__ if hasattr(v,'read'): return self._mp_call(kw) File "./Client.py", line 231, in _mp_call self.handleError('', ec, em, headers, response) File "./Client.py", line 170, in handleError raise t, RemoteException(t,v,f,l,self.url,query,ec,em,response) bci.ServerError: 302 (File: Unknown Line: Unknown) 302 Moved Temporarily for http://localhost:8080/manage_addFile ====================
Any idea what's happening?
You could ignore this "error". The HTTP spec says, that response codes 2xx are good and response codes 3xx mean, the request is not complete. Apparently, "Client" interprets the HTTP spec very tightly. Any response code other than 2xx results in an exception. For most automisation tasks, response code 302 can be treated as "ok", however. .... hangs for large files .... I analysed such a problem recently. It turned out that the script did not hang but that the transfer took very long. In my case, the transfer of a 60 MB file took 11 minutes (locally on one host). During this time, the IOWait of the processor was between 50 and 90 %. I was really astonished about this low bandwidth. I do not know what subcomponent was responsible but I am sure it was not ZPublisher.Client (it just builds a huge string and transfers it via the socket in a single "send"; this is very memory intensive but should not cause high IOWait over minutes (reading the file took 20 s)). Dieter
On Tue, 2003-05-27 at 15:54, Dieter Maurer wrote:
Traceback (most recent call last): File "./Client.py", line 563, in ? main() File "./Client.py", line 556, in main headers, body = apply(f,(),kw) File "./Client.py", line 109, in __call__ if hasattr(v,'read'): return self._mp_call(kw) File "./Client.py", line 231, in _mp_call self.handleError('', ec, em, headers, response) File "./Client.py", line 170, in handleError raise t, RemoteException(t,v,f,l,self.url,query,ec,em,response) bci.ServerError: 302 (File: Unknown Line: Unknown) 302 Moved Temporarily for http://localhost:8080/manage_addFile ====================
You could ignore this "error".
The HTTP spec says, that response codes 2xx are good and response codes 3xx mean, the request is not complete. Apparently, "Client" interprets the HTTP spec very tightly. Any response code other than 2xx results in an exception.
For most automisation tasks, response code 302 can be treated as "ok", however.
That's good to know. Thanks!
.... hangs for large files ....
I analysed such a problem recently. It turned out that the script did not hang but that the transfer took very long. In my case, the transfer of a 60 MB file took 11 minutes (locally on one host). During this time, the IOWait of the processor was between 50 and 90 %. I was really astonished about this low bandwidth. I do not know what subcomponent was responsible but I am sure it was not ZPublisher.Client (it just builds a huge string and transfers it via the socket in a single "send"; this is very memory intensive but should not cause high IOWait over minutes (reading the file took 20 s)).
Well, that's no good. There is a happy ending, however. I managed to find a solution to my problem with a recipe from ZopeLabs: http://www.zopelabs.com/cookbook/1029932854 It uses urllib instead of urllib2, and with a bit of refactoring, I was able to make a script that uploaded batches to arbitrary urls from the command line. Just what I wanted. Thanks for your help! --Damon
participants (3)
-
Damon Butler -
Dieter Maurer -
Dylan Reinhardt