[Twisted-Python] [ANN] Turtl 0.1.1 throttler released
Hi, Turtl is an HTTP proxy whose purpose is to throttle connections to specific hostnames to avoid breaking terms of usage of those API providers (like del.icio.us, technorati and so on). At the core of turtl is a throttling deferred that works in a similar way as DeferredSemaphore() except that it will enforce also a rate (N calls every M seconds) at which deferreds added to it are fired. In the past few weeks it's been improved a couple obscure bugs have been ironed out. It's been running as a proxy for a couple of years and recently we started using it as a crawler rate limiter. Source code lives on bitbucket: https://bitbucket.org/adroll/turtl/overview Here's a small example of its usage: import time from twisted.internet import reactor, defer from twisted.protocols.policies import WrappingFactory from twisted.web import client, server, resource from turtl import engine throttle = engine.ThrottlingDeferred(concurrency=1, calls=2, interval=1) class FakeResource(resource.Resource): isLeaf = True def render(self, request): return "hello" def setupServer(): site = server.Site(FakeResource()) wrapper = WrappingFactory(site) port = reactor.listenTCP(0, wrapper, interface="127.0.0.1") portno = port.getHost().port return portno def stop(_): return reactor.stop() def makeUrl(port): return "http://localhost:%s/" % (port) def prinl(page): print time.time(), page port = setupServer() url = makeUrl(port) defer.DeferredList([throttle.run(client.getPage, url).addBoth(prinl) for i in xrange(1000)]).addBoth(stop) reactor.run() -- Valentino Volonghi http://www.adroll.com
On Aug 16, 2011, at 3:35 PM, Valentino Volonghi wrote:
Turtl is an HTTP proxy whose purpose is to throttle connections to specific hostnames to avoid breaking terms of usage of those API providers (like del.icio.us, technorati and so on).
Awesome, Valentino! Thanks for the announcement :). Any chance you could also add this to <https://launchpad.net/tx> and/or <http://twistedmatrix.com/trac/wiki/ProjectsUsingTwisted>? -glyph
On 16 Aug 2011, at 18:55, Glyph Lefkowitz wrote:
Turtl is an HTTP proxy whose purpose is to throttle connections to specific hostnames to avoid breaking terms of usage of those API providers (like del.icio.us, technorati and so on).
Awesome, Valentino! Thanks for the announcement :).
Any chance you could also add this to <https://launchpad.net/tx> and/or <http://twistedmatrix.com/trac/wiki/ProjectsUsingTwisted>?
Hey Glyph thanks! :) I'd add it to ProjectsUsingTwisted but I don't have rights on trac. About the tx project instead AdRoll open source repositories are on bitbucket for now, unless there's a way to link to projects from outside I won't do that one just yet. -- Valentino Volonghi http://www.adroll.com
On Wed, Aug 17, 2011 at 9:02 PM, Valentino Volonghi <dialtone@gmail.com>wrote:
Hey Glyph thanks! :)
I'd add it to ProjectsUsingTwisted but I don't have rights on trac.
I added it for you.
About the tx project instead AdRoll open source repositories are on bitbucket for now, unless there's a way to link to projects from outside I won't do that one just yet.
Nah, you can totally make a Launchpad project for something (takes about 3 minutes) without using Launchpad to store code. cheers lvh
On Aug 17, 2011, at 3:02 PM, Valentino Volonghi wrote:
(...) unless there's a way to link to projects from outside I won't do that one just yet.
There are definitely ways to point at other locations for projects. For example, Twisted isn't hosted on Launchpad (it's on twistedmatrix.com), but https://launchpad.net/twisted/ can show you lots of stuff about it.
On 17 Aug 2011, at 13:10, Glyph Lefkowitz wrote:
(...) unless there's a way to link to projects from outside I won't do that one just yet.
There are definitely ways to point at other locations for projects. For example, Twisted isn't hosted on Launchpad (it's on twistedmatrix.com), but https://launchpad.net/twisted/ can show you lots of stuff about it.
Ok, I created the project and even imported the code, although it won't really be developed from there, maybe I'll even remove it from it in the future :). https://launchpad.net/turtl But the project is simple enough that it doesn't need so much development time devoted to it. -- Valentino Volonghi http://www.adroll.com
On Aug 17, 2011, at 4:15 PM, Valentino Volonghi wrote:
On 17 Aug 2011, at 13:10, Glyph Lefkowitz wrote:
(...) unless there's a way to link to projects from outside I won't do that one just yet.
There are definitely ways to point at other locations for projects. For example, Twisted isn't hosted on Launchpad (it's on twistedmatrix.com), but https://launchpad.net/twisted/ can show you lots of stuff about it.
Ok, I created the project and even imported the code, although it won't really be developed from there, maybe I'll even remove it from it in the future :).
But the project is simple enough that it doesn't need so much development time devoted to it.
Great :). One more thing though: on the right side of the page for https://launchpad.net/turtl - there's a link that says 'change details'. Click that, scroll down to 'part of', and select the 'tx' project group. Thanks again! -g
On 17 Aug 2011, at 13:21, Glyph Lefkowitz wrote:
Great :). One more thing though: on the right side of the page for https://launchpad.net/turtl - there's a link that says 'change details'. Click that, scroll down to 'part of', and select the 'tx' project group.
Thanks again!
I did that already. https://launchpad.net/turtl I see "Part of: tx" in the page. -- Valentino Volonghi http://www.adroll.com
On Aug 17, 2011, at 5:09 PM, Valentino Volonghi wrote:
On 17 Aug 2011, at 13:21, Glyph Lefkowitz wrote:
Great :). One more thing though: on the right side of the page for https://launchpad.net/turtl - there's a link that says 'change details'. Click that, scroll down to 'part of', and select the 'tx' project group.
Thanks again!
I did that already.
I see "Part of: tx" in the page.
I guess something was cached somewhere; I can see it now. Thanks again!
participants (3)
-
Glyph Lefkowitz -
Laurens Van Houtven -
Valentino Volonghi