ANN: urllib3 0.2 - HTTP library with thread-safe connection pooling and file posting support

Andrey Petrov shazow at
Sun Nov 30 23:29:11 CET 2008

Project website:

Also available on pypi:

(Licensed under MIT)


 * Re-use the same socket connection for multiple requests
 * File posting (``encode_multipart_formdata``)
 * Built-in redirection and retries (optional)
 * Thread-safe

What's wrong with urllib and urllib2?

There are two critical features missing from the Python standard
Connection re-using/pooling and file posting. It's not terribly hard
implement these yourself, but it's much easier to use a module that
did the work for you.

The Python standard libraries ``urllib`` and ``urllib2`` have little
to do
with each other. They were designed to be independent and standalone,
solving a different scope of problems, and ``urllib3`` follows in a

Why do I want to reuse connections?

Performance. When you normally do a urllib call, a separate socket
connection is created with each request. By reusing existing sockets
(supported since HTTP 1.1), the requests will take up less resources
on the
server's end, and also provide a faster response time at the client's
With some simple benchmarks (see `test/
), downloading 15 URLs from is about twice as fast when
HTTPConnectionPool (which uses 1 connection) than using plain urllib
uses 15 connections).

This library is perfect for:

 * Talking to an API
 * Crawling a website
 * Any situation where being able to post files, handle redirection,
   retrying is useful. It's relatively lightweight, so it can be used


Go to the `Examples wiki <
for more nice syntax-highlighted examples.

But, long story short::

  from urllib3 import HTTPConnectionPool

  API_URL = ''

  http_pool = HTTPConnectionPool.from_url(API_URL)

  fields = {'v': '1.0', 'q': 'urllib3'}
  r = http_pool.get_url(API_URL, fields)

  print r.status,

Enjoy! Feedback is very welcome, please send it to shazow at gmail.

- Andrey

I apologize in advance for the potentially controversial name, but
after much consideration it turned out to be the most descriptive one
I could think of.

More information about the Python-announce-list mailing list