[Tutor] urllib2 with a proxy

[Kent Johnson]

Try this (untested and I don't know what I'm talking about but maybe it 
will work):

import urllib2
passmgr = urllib2.HTTPPasswordMgrWithDefaultRealm()
passmgr.add_password(None, 'http://proxy:8080', 'username', 'password')
authinfo = urllib2.ProxyBasicAuthHandler(passmgr)
proxy_support = urllib2.ProxyHandler({"http" : "http://proxy:8080"})

opener = urllib2.build_opener(proxy_support, authinfo)
urllib2.install_opener(opener)
f = urllib2.urlopen('http://www.google.com')
buf = f.read()
print buf
f.close()


The host for the password manager is probably the host you are 
authenticating against (http://proxy:8080). The realm is the 
authentication realm on the proxy; this is part of the auth headers 
returned from the proxy.

Note: in your code below, I think you need a ProxyBasicAuthHandler, not 
an HTTPBasicAuthHandler; you have misspelled 'http://proxy:8080'.

HTH
Kent

kbond wrote:
> Kent,
> 
> I did test your suggestion but unfortunatly it is still not working.
> Somehow I do not understand  why it  is so  complex  to  go trough  my 
> proxy.  I do not have  any trouble if I use  firefox or mozilla.
> I am completely stuck by this tiny detail.
> If someone on this list has clear and simple explanation about what is 
> realm and host in "add_password('realm', 'host', 'username', 
> 'password')" that will also help me a lot.
> 
> Thank you for your help any pointer or code sample are more than welcome.
> 
> *code** *
> import urllib2
> # set up authentication info
> authinfo = urllib2.HTTPBasicAuthHandler()
> #add_password('realm', 'host', 'username', 'password')
> authinfo.add_password(None, "http://google.com", 'username', 'pwd')
> proxy_support = urllib2.ProxyHandler({"http" : "http//:proxy:8080"})
> # build a new opener that adds authentication and caching FTP handlers
> opener = urllib2.build_opener(proxy_support,
> authinfo,urllib2.CacheFTPHandler())
> # install it
> urllib2.install_opener(opener)
> f = urllib2.urlopen('http://google.com')
> buf = f.read()
> print buf
> f.close()
> 
> 
> *error*
> 
> Traceback (most recent call last):
> File "testProxy.py", line 12, in ?
>   f = urllib2.urlopen('http://google.com')
> File "E:\users\install\Python\lib\urllib2.py", line 129, in urlopen
>   return _opener.open(url, data)
> File "E:\users\install\Python\lib\urllib2.py", line 326, in open
>   '_open', req)
> File "E:\users\install\Python\lib\urllib2.py", line 306, in _call_chain
>   result = func(*args)
> File "E:\users\install\Python\lib\urllib2.py", line 491, in <lambda>
>   lambda r, proxy=url, type=type, meth=self.proxy_open: \
> File "E:\users\install\Python\lib\urllib2.py", line 498, in proxy_open
>   if '@' in host:
> TypeError: iterable argument required
> 
> 
> 
> kbond wrote:
> 
>> Thank you for your help kent.
>> Already the second time that you are helping me.
>> In fact I face this proxy trouble right after your first answer I was 
>> about to try your suggestion to solve the clientForm trouble when this 
>> proxy came into the danse.
>> When I started my project I was far to imagine that talking to the 
>> world wide web was that difficult by script.
>> Unfortunatly I will not be able to test before monday because I do not 
>> use a proxy at home.
>> Do you have an idea of what are realm and  host in the add_password() 
>> method?
>> I am making the assomption that host stand for the host I want to 
>> contact but I do not have any idea of what realm is?
>> Do you know how I can get it? Why firefox don't need this information 
>> to go out?
>>
>> You will find below the code I will try on monday.
>> Thank you for your help
>>
>> *Code*
>> import urllib2
>> # set up authentication info
>> authinfo = urllib2.HTTPBasicAuthHandler()
>> #add_password('realm', 'host', 'username', 'password')
>> authinfo.add_password(None, "http://google.com", 'userName', 'pwd')
>> proxy_support = urllib2.ProxyHandler({"http" : "http//:proxy:8080"})
>> # build a new opener that adds authentication and caching FTP handlers
>> opener = urllib2.build_opener(proxy_support, 
>> authinfo,urllib2.CacheFTPHandler())
>> # install it
>> urllib2.install_opener(opener)
>> f = urllib2.urlopen('http://google.com')
>> buf = f.read()
>> print buf
>> f.close()
>>
>> Kent Johnson wrote:
>>
>>> I'm not familiar with this library but looking at your code and the 
>>> docs I can make some guesses where the problem might be:
>>> - The second argument to ProxyHandler should be a complete URL, e.g. 
>>> 'http://proxy:8080'. This is the cause of the exception you are seeing.
>>> - Try using a HTTPBasicAuthHandler instead of a HTTPDigestAuthHandler 
>>> unless you are sure your proxy uses digest authentication
>>> - You should pass an instance of CacheFTPHandler to build_opener, not 
>>> the class itself. In other words, pass CacheFTPHandler() with 
>>> parentheses.
>>>
>>> HTH
>>> Kent
>>>
>>> At 10:56 AM 11/6/2004 -0500, kbond wrote:
>>>
>>>> hello,
>>>>
>>>> I am trying to get access an internet page  using urllib2. I am able 
>>>> to do so form home where I do not have proxy but for some reason I 
>>>> am unable to do it form my company.
>>>>
>>>> I found many ressource on the internet explaining how to do it but I 
>>>> was unable to apply it to my contexte. The interesting thing is that 
>>>> I did manage to configure Firefox (my fovourite browser) so I guess  
>>>> I should have all the information to do it by script.
>>>> I think I am doing something is the usage of "add_password". Could 
>>>> you please explain me what realm and host are? and how can I get them?
>>>>
>>>> You will find bellow my firefox configuration.
>>>> It would be great If someone can help me to put all this together.
>>>> Thank you for your help
>>>>
>>>> *Code*
>>>> import urllib2
>>>> # set up authentication info
>>>> authinfo = urllib2.HTTPDigestAuthHandler()
>>>> #proxy_auth_handler.add_password('realm', 'host', 'username', 
>>>> 'password')
>>>> authinfo.add_password(None, "http://google.com", 'userName', 'pwd')  
>>>> * *
>>>> proxy_support = urllib2.ProxyHandler({"http" : "proxy:8080"})
>>>> # build a new opener that adds authentication and caching FTP handlers
>>>> opener = urllib2.build_opener(proxy_support, 
>>>> authinfo,urllib2.CacheFTPHandler)
>>>> # install it
>>>> urllib2.install_opener(opener)
>>>> f = urllib2.urlopen('http://google.com')
>>>> buf = f.read()
>>>> print buf
>>>> f.close()
>>>>
>>>> *Error Message*
>>>>
>>>> Traceback (most recent call last):
>>>>  File "test.py", line 18, in ?
>>>>    f = urllib2.urlopen('http://google.com')
>>>>  File "E:\users\install\Python\lib\urllib2.py", line 129, in urlopen
>>>>    return _opener.open(url, data)
>>>>  File "E:\users\install\Python\lib\urllib2.py", line 326, in open
>>>>    '_open', req)
>>>>  File "E:\users\install\Python\lib\urllib2.py", line 306, in 
>>>> _call_chain
>>>>    result = func(*args)
>>>>  File "E:\users\install\Python\lib\urllib2.py", line 491, in <lambda>
>>>>    lambda r, proxy=url, type=type, meth=self.proxy_open: \
>>>>  File "E:\users\install\Python\lib\urllib2.py", line 498, in proxy_open
>>>>    if '@' in host:
>>>> TypeError: iterable argument required
>>>>
>>>> *Firefox configuration*
>>>>
>>>>
>>>> _______________________________________________
>>>> Tutor maillist  -  Tutor at python.org
>>>> http://mail.python.org/mailman/listinfo/tutor
>>>
>>>
>>>
>>>
>>>
>>> _______________________________________________
>>> Tutor maillist  -  Tutor at python.org
>>> http://mail.python.org/mailman/listinfo/tutor
>>>
>>>
>>
>> _______________________________________________
>> Tutor maillist  -  Tutor at python.org
>> http://mail.python.org/mailman/listinfo/tutor
>>
>>
> _______________________________________________
> Tutor maillist  -  Tutor at python.org
> http://mail.python.org/mailman/listinfo/tutor
>