[Twisted-Python] Making twisted web client requests on running reactor
Hi everyone, I'm writing a client library that makes (potentially long running) HTTP requests. Since the library will be used within non-Twisted code, I was thinking of running the reactor in a separate thread. The library code looks roughly like this: def start(): Thread(target=reactor.run, args=(False,)).start() def shutdown(): reactor.callFromThread(lambda x: reactor.stop(), None) def make_request(url): agent = Agent(reactor) d = agent.request( 'GET', url, Headers({ 'User-Agent': ['Twisted SSE Client'], 'Cache-Control': ['no-cache'], }), None) d.addCallback(self.cbRequest) Then I make requests from the application like so: start() make_request("http://localhost:12000") .... make_request("http://localhost:12000") ... shutdown() However, it looks like the 'd' deferred never gets fired. From a little playing around, it looks like this is because the reactor is started before the call(s) to agent.request are made. All examples in the twisted docs create the request before running the reactor. I'm sure I'm missing something very obvious here. Also, is there a better way to design such a library? -- -Ameya
Hello Ameya, On 23 Feb 2014, at 3:17, Ameya Lokare wrote:
I'm writing a client library that makes (potentially long running) HTTP requests. Since the library will be used within non-Twisted code, I was thinking of running the reactor in a separate thread.
Have you seen Crochet? That might be just what you’re looking for: https://pypi.python.org/pypi/crochet/ Cheers Hynek
On 02:17 am, lokare.ameya@gmail.com wrote:
Hi everyone,
I'm writing a client library that makes (potentially long running) HTTP requests. Since the library will be used within non-Twisted code, I was thinking of running the reactor in a separate thread. The library code looks roughly like this:
def start(): Thread(target=reactor.run, args=(False,)).start()
def shutdown(): reactor.callFromThread(lambda x: reactor.stop(), None)
def make_request(url): agent = Agent(reactor) d = agent.request( 'GET', url, Headers({ 'User-Agent': ['Twisted SSE Client'], 'Cache-Control': ['no-cache'], }), None) d.addCallback(self.cbRequest)
Then I make requests from the application like so:
start() make_request("http://localhost:12000") .... make_request("http://localhost:12000") ... shutdown()
However, it looks like the 'd' deferred never gets fired. From a little playing around, it looks like this is because the reactor is started before the call(s) to agent.request are made. All examples in the twisted docs create the request before running the reactor. I'm sure I'm missing something very obvious here. Also, is there a better way to design such a library?
It sounds like you want to use Crochet (which Hynek already linked you to). The reason your code doesn't work, though, is that Twisted APIs are not thread safe. You may *only* call them in the thread the reactor is running in. Your `start` function runs the reactor in a new thread. Then the code proceeds to try to use `Agent` from the original thread. This is unsupported. Jean-Paul
Thanks, Hynek and Jean-Paul! Crochet looks like exactly what I'm looking for. -Ameya On Sun, Feb 23, 2014 at 5:11 AM, <exarkun@twistedmatrix.com> wrote:
On 02:17 am, lokare.ameya@gmail.com wrote:
Hi everyone,
I'm writing a client library that makes (potentially long running) HTTP requests. Since the library will be used within non-Twisted code, I was thinking of running the reactor in a separate thread. The library code looks roughly like this:
def start(): Thread(target=reactor.run, args=(False,)).start()
def shutdown(): reactor.callFromThread(lambda x: reactor.stop(), None)
def make_request(url): agent = Agent(reactor) d = agent.request( 'GET', url, Headers({ 'User-Agent': ['Twisted SSE Client'], 'Cache-Control': ['no-cache'], }), None) d.addCallback(self.cbRequest)
Then I make requests from the application like so:
start() make_request("http://localhost:12000") .... make_request("http://localhost:12000") ... shutdown()
However, it looks like the 'd' deferred never gets fired. From a little playing around, it looks like this is because the reactor is started before the call(s) to agent.request are made. All examples in the twisted docs create the request before running the reactor. I'm sure I'm missing something very obvious here. Also, is there a better way to design such a library?
It sounds like you want to use Crochet (which Hynek already linked you to).
The reason your code doesn't work, though, is that Twisted APIs are not thread safe. You may *only* call them in the thread the reactor is running in.
Your `start` function runs the reactor in a new thread. Then the code proceeds to try to use `Agent` from the original thread. This is unsupported.
Jean-Paul
_______________________________________________ Twisted-Python mailing list Twisted-Python@twistedmatrix.com http://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python
-- -Ameya
participants (3)
-
Ameya Lokare -
exarkun＠twistedmatrix.com -
Hynek Schlawack