Deficiency in urllib/socket for https?
John J. Lee
jjl at pobox.com
Fri Aug 22 16:47:59 CEST 2003
Gary Feldman <gafStopSpamData at ziplink.stopallspam.net> writes:
> I think I've found a deficiency in the design of urllib related to https.
> In order to complete an https connection, it appears that URLOpener and
> hence FancyURLOpener require the key and cert files. Or at least, it's not
> clear from the description of socket.ssl what it does if they're omitted.
Nor from urllib -- see below. In fact, it seems that verification is
just skipped if they're not there.
> However, urlopen has no way to specify such things. Nor should it - for
> typical uses, a person simply trying to retrieve data from an ssl site
> really doesn't want to know or care about keys and certificate directories.
> One just wants to provide an https url and have it work. Ideally, there
> should be defaults for the certificate files.
Hmm, looking at both urllib and urllib2, I see urllib2 doesn't use any
key or certificate files at all. So, two points: this is a deficiency
in urllib2 that should be fixed, and, if you're not bothered about key
verification, I'd guess just not providing key / cert files will work.
Hmm, urllib documentation seems wrong here:
Additional keyword parameters, collected in x509, are used for
authentication with the https: scheme. The keywords key_file and
cert_file are supported; both are needed to actually retrieve a
resource at an https: URL.
The fact that https works in urllib2 (which does not provide key /
cert files) seems to demonstrate that they're *not* required, and that
verification is skipped if they're not supplied.
If you *are* bothered about verification, use the x509 arg to
FancyURLOpener (which is documented, see above). The urlopen function
is just a convenience -- just cut-n-paste the trivial code from
urllib.py and adapt it to your needs if you need something more
> This implies that somewhere in the function hierarchy, I suspect in
> socket.ssl, there needs to be some clever defaults. I don't know if they
> folks maintaining the Python distribution really want to be in the business
> of maintaining key and certificate directories (probably not), but there at
> least ought to be a way to specify default directories (oh, no, another
> environment variable?). Thinking idealistically, it would be great if it
> could share the default certs on the system (i.e. on UNIX, find a Netscape
> or Mozilla install directory and use those, and on MS Windows, do whatever
> it takes to use the Windows mechanism).
That sounds great if you have the time to write the code. Nobody else
is likely to.
More information about the Python-list