Deficiency in urllib/socket for https?

[John J. Lee]

Gary Feldman <gafStopSpamData at ziplink.stopallspam.net> writes:

> I think I've found a deficiency in the design of urllib related to https.
> 
> In order to complete an https connection, it appears that URLOpener and
> hence FancyURLOpener require the key and cert files.  Or at least, it's not
> clear from the description of socket.ssl what it does if they're omitted.  

Nor from urllib -- see below.  In fact, it seems that verification is
just skipped if they're not there.


> However, urlopen has no way to specify such things.  Nor should it - for
> typical uses, a person simply trying to retrieve data from an ssl site
> really doesn't want to know or care about keys and certificate directories.
> One just wants to provide an https url and have it work.   Ideally, there
> should be defaults for the certificate files.

Hmm, looking at both urllib and urllib2, I see urllib2 doesn't use any
key or certificate files at all.  So, two points: this is a deficiency
in urllib2 that should be fixed, and, if you're not bothered about key
verification, I'd guess just not providing key / cert files will work.

Hmm, urllib documentation seems wrong here:

 Additional keyword parameters, collected in x509, are used for
 authentication with the https: scheme. The keywords key_file and
 cert_file are supported; both are needed to actually retrieve a
 resource at an https: URL.

The fact that https works in urllib2 (which does not provide key /
cert files) seems to demonstrate that they're *not* required, and that
verification is skipped if they're not supplied.

If you *are* bothered about verification, use the x509 arg to
FancyURLOpener (which is documented, see above).  The urlopen function
is just a convenience -- just cut-n-paste the trivial code from
urllib.py and adapt it to your needs if you need something more
complicated.


> This implies that somewhere in the function hierarchy, I suspect in
> socket.ssl, there needs to be some clever defaults.  I don't know if they
> folks maintaining the Python distribution really want to be in the business
> of maintaining key and certificate directories (probably not), but there at
> least ought to be a way to specify default directories (oh, no, another
> environment variable?).  Thinking idealistically, it would be great if it
> could share the default certs on the system (i.e. on UNIX, find a Netscape
> or Mozilla install directory and use those, and on MS Windows, do whatever
> it takes to use the Windows mechanism).

That sounds great if you have the time to write the code.  Nobody else
is likely to.


John