[Python-Dev] Py3K thought: use external library for client-side HTTP

[A.M. Kuchling]

Thought: We should drop all of httplib, urllib, urllib2, and ftplib,
and instead adopt some third-party library for HTTP/FTP/whatever,
write a Python wrapper, and use it instead.  (The only such library I
know of is libcurl, but doubtless there are other candidates; see
http://curl.haxx.se/libcurl/competitors.html for a list.)

Rationale:

* HTTP client-side support is pretty complicated.  HTTP itself
  has many corners (httplib.py alone is 1420 lines long, and urllib2
  is 1300 lines).

* There are many possible permutations of proxying, SSL on/off,
  and authentication.  We probably haven't tested every permutation,
  and probably lack the volunteer effort to test them all.
  If you search for 'http' in the bug tracker, you find about 16 or so 
  bugs submitted for httplib/urllib/urllib2, most of them for one
  permutation or another.

  With a third-party library, the work of maintaining RFC compliance falls
  to someone else.

* A third-party library might support more features than we have time
  to implement.

A downside: these libraries would be in C, and might be the source of
security bugs.  Python code may be buggy, but probably won't fall prey
to buffer overflow.  We'd also have to keep in sync with the library.

Similar arguments could be made for a server-side solution, but here I
have no idea what we might choose.  A server-side HTTP implementation
+ a WSGI gateway might be all that Python 3000 needs.

Good idea?  Dumb idea?

--amk