[Python-Dev] python 2.7 + https + urlopen = ?

Python Urlopen urllib3 at yahoo.com
Mon Aug 13 03:28:47 CEST 2012

I am a python 2.7.x user and am hoping to reach with this email some python developers who would be sympathetic to this scenario (And I understand that you might not, which is perfectly fine -- I've already requested one developer not to reply ) : 

How would you feel, if you issued :

import urllib

and the command got you data from some other URL without telling you! You use firefox, and the site is different than the data you got! Same with chrome. Safari. Even IE !
Cheated? (Well I was mad -- after IE worked).

Then, you dig a little and say, hey there are bugs in networks/code, lets try the other tools that are available on python 2.x, who uses urlopen from urllib in 2012. There are tons, right?

urllib2, urllib, urllib3, requests, twisted.getPage, ...

None of them worked! Wow. Then you wonder, whats going on. You poke one of the server administrator, and he sends you the logs, and you see the problem. The keyword being "SNI". Now you start googling. First read about SNI perhaps. Here is a 2 line summary:

SNI is a server side "feature" that extends SSL and TLS protocols to let you talk to a https server which is on an IP that serves multiple certificates for multiple https servers. SNI was first used in 2004 and OpenSSL started support in 2006. In 2007, it was backported to OpenSSL 0.9.x. In 2009 there was a bug filed with python-devs for fixing this in 2.6. The feature enhancement (or "bug fix") eventually happened -- for 3.2+. (http://en.wikipedia.org/wiki/Server_Name_Indication)

Then you google more and you land up on this page: http://bugs.python.org/issue5639

which shows you that 2.6 has a patch. Then you wonder, why wasn't it included in 2.7 -- and you read -- AP : "No, Python 2 only receives bug fixes.". You instantly hate the guy. Sorry AP, nothing personal, but please do not reply to this post. I think I know what your reply will be.  

After a lot of pain, I got myself out of this trouble, and my code now works correctly on 2.7.x (thanks to Jean-Paul Calderone's pyopenssl). But do "you" think this is a "feature" and not a "bug"? -- And do you think debating on this, killing time on the debate, and letting all python 2.x users suffer sooner or later is right --. Something as basic as urlopen! 

Thanks for your time and I wish good luck to most python users.

More information about the Python-Dev mailing list