On 2008-03-02 21:11, Brett Cannon wrote:
>> > Well, look at the docs for urllib. There is a list of restrictions
>> > (e.g., does not support the use of proxies which require
>> > authentication). From what I can tell, those items on the list that
>> > are an actual restriction do not carry over to urllib2.
>>
>> I'm not sure I follow you: urllib *does* support proxies that
>> require authentication (see the .open_http() method).
>>
>
> According to http://docs.python.org/dev/library/urllib.html#module-urllib:
>
> "This module does not support the use of proxies which require authentication".
Looks like the documentation is not up-to-date:
...
if proxy_auth: h.putheader('Proxy-Authorization', 'Basic %s' % proxy_auth)
...
>> > Another thing,
>> > how do you add a custom line to the header for the request in urllib
>> > (e.g., Referer)? The docs for URLOpener don't seem to provide a way.
>> > urllib2, on the other hand, has a very specific way to add headers.
>>
>> That's easy:
>>
>> class URLReader(urllib.URLopener):
>>
>> # Crawler name
>> agentname = 'mxHTMLTools-Crawler'
>>
>> def __init__(*args):
>>
>> """ Add a user-agent header to the HTTP requests.
>> """
>> self = args[0]
>> apply(urllib.URLopener.__init__, args)
>> # Override the default settings for self.addheaders:
>> assert len(self.addheaders) == 1
>> self.addheaders = [
>> ('user-agent', '%s/%s' % (self.agentname, HTMLTools.__version__)),
>> ]
>> ...
>>
>
> But none of that is documented. So if the classes do stay then they
> really need to have their documentation flushed out (along with making
> sure they have the proper unit tests for those exposed APIs, of
> course).
Agreed.
>> > But as I said in my last email, I am happy to include URLOpener if
>> > some other people are willing to back the idea up.
>>
>> Fair enough.
>
> I will send a separate email to the SIG since people have probably
> stopped following most of this thread. =)
Please also CC the stdlib SIG again... that CC got lost somewhere
in the thread :-)
--
Marc-Andre Lemburg
eGenix.com
Professional Python Services directly from the Source (#1, Mar 03 2008)
>>> Python/Zope Consulting and Support ... http://www.egenix.com/
>>> mxODBC.Zope.Database.Adapter ... http://zope.egenix.com/
>>> mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/
________________________________________________________________________
:::: Try mxODBC.Zope.DA for Windows,Linux,Solaris,MacOSX for free ! ::::
eGenix.com Software, Skills and Services GmbH Pastor-Loeh-Str.48
D-40764 Langenfeld, Germany. CEO Dipl.-Math. Marc-Andre Lemburg
Registered at Amtsgericht Duesseldorf: HRB 46611