[CentralOH] Goroutines in Python

[Eric Floehr]

All,

Go is a pretty cool language, and Joe did a great job introducing it last
night. By all means, play with it, try it, etc. It may work better than
Python for your needs.

One of the things that intrigued me the most was goroutines. I've been
playing with Python 3 on and off, and one of the biggest new features of
Python 3.4 is the introduction of asynchronous IO and light-weight
coroutines (goroutines are mostly like coroutines).

If you want to play around with Python's version of these Go topics, there
are a number of great resources, including Brian Costlow's async talk at
the last PyOhio (links in the gist below).

I dug around for some examples of fetching URLs using the new asyncio, as I
currently use heavy-weight threads to pull about 17,000 unique URLs every
day, and am looking to improve that (there are plans for 10x the number of
URLs).  So Joe's URL fetching example was of particular interest.

So I took a quick stab at recreating it. I cheated a little by using a
third-party library that has native support for asyncio (aiohttp), but with
a little more work, requests can be adapted for asyncio. If you don't have
a binary aiohttp library available for your system, it requires gcc and
python-dev (source) to build.

I also added optional throttling via a semaphore. You can set how many HTTP
requests you want to issue at any one time. You'll have to uncomment some
code for that... as it runs, it fires them all off at once.

Anyway, here is the gist with the Go version and the Python version, along
with links to some good docs and videos on ayncio in Python.

https://gist.github.com/efloehr/d3626762835aa2de83b8

Cheers,
Eric
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/centraloh/attachments/20150224/2961aab6/attachment.html>