[Twisted-Python] Twisted and ipython
Hi, I'm starting a project where I plan to use twisted for the first time. I'd like to practice using ipython and after looking around I found a recepe [1] by Metthew Scott and a comment by Bob Ippolito on that recepe that suggests a different approch [2]. In the ipython manual there's a reference to twisted in the -*thread section, but it's unclear to me what should be the option (-pylab as for matplotlib?): -gthread, -qthread, -q4thread, -wthread, -pylab Only ONE of these can be given, and it can only be given as the first option passed to IPython (it will have no effect in any other position). They provide threading support for the GTK, QT3, QT4 and WXWidgets toolkits, for the mat- plotlib library and Twisted reactor. So what's the suggested way to play with twisted? thanks in advance sandro *:-) [1] http://code.activestate.com/recipes/410670-integrating-twisted-reactor-with-... [2] http://bob.pythonmac.org/archives/2005/04/17/twisted-and-foreign-event-loops... -- Sandro Dentella *:-) http://www.reteisi.org Soluzioni libere per le scuole http://sqlkit.argolinux.org SQLkit home page - PyGTK/python/sqlalchemy
On 11 January 2011 07:48, Alessandro Dentella <sandro@e-den.it> wrote:
So what's the suggested way to play with twisted?
It depends what you're trying to do. I'd suggest you get some form of skeleton program running first. At that point you can interact with the program using twisted manhole. ipython has a task interface with supports starting asynchronous jobs, a quick search turns up http://mail.scipy.org/pipermail/ipython-user/2009-February/006148.html Regards, Michael
thanks, short abstact: manhole is very good, thanks. But I'd need to add readline to make it usable and I'd need tab-completion too. On Thu, Jan 13, 2011 at 12:06:10PM +0000, Michael Thompson wrote:
On 11 January 2011 07:48, Alessandro Dentella <sandro@e-den.it> wrote:
So what's the suggested way to play with twisted?
It depends what you're trying to do. I'd suggest you get some form of skeleton program running first.
I do have already: it uses SerialPort, queries a serial bus and returns info on a domotic system (switches & co). Apart from my queries it also listen to whatever the serial bus issues and react changing the internal state of some objects. What I'd like to do is to play interactively with these objects (Eg: a digital module with 8 switches: I want to switch on/off the exits).
At that point you can interact with the program using twisted manhole.
this is interesting. I found a working example:: from twisted.internet import reactor from twisted.conch import telnet def createShellServer( ): print 'Creating shell server instance' factory = telnet.ShellFactory() port = reactor.listenTCP( 2000, factory) factory.namespace['x'] = 'hello world' factory.username = 'sd' factory.password = 'vega' print 'Listening on port 2000' return port if __name__ == "__main__": reactor.callWhenRunning( createShellServer ) reactor.run() that raises /usr/lib/python2.6/dist-packages/twisted/manhole/telnet.py:8: DeprecationWarning: As of Twisted 2.1, twisted.protocols.telnet is deprecated. See twisted.conch.telnet for the current, supported API. from twisted.protocols import telnet I couldn't make out a correct syntax using the new API. Can somebody point me to the correct solution? Adding readline ---------------- To make this solution viable to me I really need to have readline and completion. I found a thread hinting of a patch that should implement it but no real patch to see [1]. I tryied to start an IPythonShell at the prompt but it simply hangs forever. sandro *:-) [1] http://www.mail-archive.com/twisted-web@twistedmatrix.com/msg02241.html -- Sandro Dentella *:-) http://www.reteisi.org Soluzioni libere per le scuole http://sqlkit.argolinux.org SQLkit home page - PyGTK/python/sqlalchemy
On 13 Jan, 11:43 pm, sandro@e-den.it wrote:
thanks,
short abstact: manhole is very good, thanks. But I'd need to add readline to make it usable and I'd need tab-completion too.
If by "add readline" you mean handle input using the stdlib readline module, you can't really do this (or, I'm sure you can, but it would involve a ton of coding). If you only mean "have some commonly useful line editing features" then you have at least a couple options: * invective includes a fully unit-tested line input widget (for the insults widget library). This really belongs somewhere other than invective, it'd be nice if someone pushed it back into Twisted. However, you can find invective at svn://svn.twistedmatrix.com/svn/Twisted/sandbox/exarkun/invective/trunk until that happens. * urwid supports Twisted and has lots of fancy things, including an input editing widget. Also, we really should get rid of twisted.manhole, because these days manhole is supposed to refer to the stuff in twisted.conch. I'm not sure if you found the latter, but from the warning you got about twisted.protocols.telnet it seems like you may have only found the former. Jean-Paul
thanks Jean-Paul for the hints, On Sun, Jan 16, 2011 at 12:50:05AM -0000, exarkun@twistedmatrix.com wrote:
On 13 Jan, 11:43 pm, sandro@e-den.it wrote:
thanks,
short abstact: manhole is very good, thanks. But I'd need to add readline to make it usable and I'd need tab-completion too.
If by "add readline" you mean handle input using the stdlib readline module, you can't really do this (or, I'm sure you can, but it would involve a ton of coding). If you only mean "have some commonly useful line editing features" then you have at least a couple options:
Well, in fact I intended mainly up/down arrows *and* tab-completion, like ipython have.
* invective includes a fully unit-tested line input widget (for the insults widget library). This really belongs somewhere other than invective, it'd be nice if someone pushed it back into Twisted. However, you can find invective at svn://svn.twistedmatrix.com/svn/Twisted/sandbox/exarkun/invective/trunk until that happens.
* urwid supports Twisted and has lots of fancy things, including an input editing widget.
From what you say I don't know if completion is something I'd get with invective of urwid.
Also, we really should get rid of twisted.manhole, because these days manhole is supposed to refer to the stuff in twisted.conch. I'm not sure if you found the latter, but from the warning you got about twisted.protocols.telnet it seems like you may have only found the former.
I was able to use only manhole.telnet. I wan't able to understand how to make twisted.protocols.telnet work in the same way and was not able to fine a demo snippet (more for completeness that else I'm still interested in this). On the other hand I managed to use ipython and twisted as I wanted and everitying is working like a charm (but I still have a question that I'll leave for next chapter). Twisted in a thread - working setup ----------------------------------- Following a hint in the ipython ml I started a second thread for twisted and I'm using blockingcallFromThread to dispatch commands to the reactor: proto = Domotik() SerialPort(proto, o.opts['device'], reactor, baudrate=int(o.opts['baudrate'])) rit = twistedutil.ReactorInThread() rit.setDaemon(True) rit.start() q = Quadro(proto, discover=False) proto.quadro = q q.discover_modules() This works exactly as I wanted. I can interact with 'q' that represents a collection of electric modules on the serial bus The unresolved doubt -------------------------------- What I can't understand is why I can't put "q.discover_modules()" inside callWhenRunning() command: if I do that it just blocks at the first iteration of the 'for' loop. reactor.callWhenRunning( start_command, proto) def start_command(proto): q.discover_modules() class Quadro(object): def __init__(self, proto): self.proto = proto self.modules = [] def discover_modules(self): """send a request to the serial bus to understand which electric modules are present""" for x in range(10): blockingCallFromThread(reactor.callLater, 1 + x/5.0, self.proto.ask_status, x) thanks sandro *:-) -- Sandro Dentella *:-) http://www.reteisi.org Soluzioni libere per le scuole http://sqlkit.argolinux.org SQLkit home page - PyGTK/python/sqlalchemy
On Jan 16, 2011, at 6:12 PM, Alessandro Dentella wrote:
thanks Jean-Paul for the hints,
On Sun, Jan 16, 2011 at 12:50:05AM -0000, exarkun@twistedmatrix.com wrote:
On 13 Jan, 11:43 pm, sandro@e-den.it wrote:
thanks,
short abstact: manhole is very good, thanks. But I'd need to add readline to make it usable and I'd need tab-completion too.
If by "add readline" you mean handle input using the stdlib readline module, you can't really do this (or, I'm sure you can, but it would involve a ton of coding). If you only mean "have some commonly useful line editing features" then you have at least a couple options:
Well, in fact I intended mainly up/down arrows *and* tab-completion, like ipython have.
* invective includes a fully unit-tested line input widget (for the insults widget library). This really belongs somewhere other than invective, it'd be nice if someone pushed it back into Twisted. However, you can find invective at svn://svn.twistedmatrix.com/svn/Twisted/sandbox/exarkun/invective/trunk until that happens.
* urwid supports Twisted and has lots of fancy things, including an input editing widget.
From what you say I don't know if completion is something I'd get with invective of urwid.
Invective is an IRC client (with a useful text-editing widget for Twisted's "insults" module). urwid is a client library. Completion is something that you'd need to implement yourself with either of these (as is a functional Python interpreter...)
Also, we really should get rid of twisted.manhole, because these days manhole is supposed to refer to the stuff in twisted.conch. I'm not sure if you found the latter, but from the warning you got about twisted.protocols.telnet it seems like you may have only found the former.
I was able to use only manhole.telnet. I wan't able to understand how to make twisted.protocols.telnet work in the same way and was not able to fine a demo snippet (more for completeness that else I'm still interested in this).
twisted.protocols.telnet is old and really should not be used. The new, good telnet implementation is in twisted.conch.telnet. You can find code that uses this (both a runnable application and an instructive example) in twisted.conch.manhole_tap, which also happens to include a python interpreter as its application.
On the other hand I managed to use ipython and twisted as I wanted and everitying is working like a charm (but I still have a question that I'll leave for next chapter).
Twisted in a thread - working setup -----------------------------------
Following a hint in the ipython ml I started a second thread for twisted and I'm using blockingcallFromThread to dispatch commands to the reactor:
You can do this in pretty much any interactive Python interpreter, including the default one. It's not very convenient, though :).
What I can't understand is why I can't put "q.discover_modules()" inside callWhenRunning() command: if I do that it just blocks at the first iteration of the 'for' loop.
Because callWhenRunning will schedule start_command to be run on the main (reactor) thread. (callWhenRunning may only be called _from_ the main thread, too, which it seems like you might not be doing). Since start_command is running on the main thread, when it gets around to blockingCallFromThread, it puts some work into the queue for the reactor to do, then goes to sleep forever, trusting that the main thread will do the work and then wake it up. Except - oops! - it just put the main thread to sleep forever, so it will never wake up. If you'd like, you can file a ticket for blockingCallFromThread to raise a loud, angry exception if it is invoked from the reactor thread. (Also, in the simple example you gave, you're invoking blockingCallFromThread with the wrong signature; the first argument must be a reactor. The rest of my explanation was based on the assumption that this is not the exact code you're running, since that would log a traceback immediately. Or, perhaps the fact that you're calling callWhenRunning from a non-main thread is blowing up in some more insidious way and that's the real problem.) Hope this helps, -glyph
Thanks Glyph, On Mon, Jan 17, 2011 at 12:07:42AM -0500, Glyph Lefkowitz wrote:
From what you say I don't know if completion is something I'd get with invective of urwid.
Invective is an IRC client (with a useful text-editing widget for Twisted's "insults" module). urwid is a client library. Completion is something that you'd need to implement yourself with either of these (as is a functional Python interpreter...)
ok, clear.
twisted.protocols.telnet is old and really should not be used. The new, good telnet implementation is in twisted.conch.telnet. You can find code that uses this (both a runnable application and an instructive example) in twisted.conch.manhole_tap, which also happens to include a python interpreter as its application.
ok, I found demo.tac and demo_manhole.tac and will have a deeper look at them.
Twisted in a thread - working setup -----------------------------------
Following a hint in the ipython ml I started a second thread for twisted and I'm using blockingcallFromThread to dispatch commands to the reactor:
You can do this in pretty much any interactive Python interpreter, including the default one. It's not very convenient, though :).
What I can't understand is why I can't put "q.discover_modules()" inside callWhenRunning() command: if I do that it just blocks at the first iteration of the 'for' loop.
Because callWhenRunning will schedule start_command to be run on the main (reactor) thread.
mmh.... in my case the reactor thread in not the main one! IPython.twistedutil.ReactorInThread does that for me I'm calling it from the main thread, that is not the reactor's thread in my case rit = twistedutil.ReactorInThread() rit.setDaemon(True) rit.start()
(callWhenRunning may only be called _from_ the main thread, too, which it seems like you might not be doing).
Since start_command is running on the main thread, when it gets around to blockingCallFromThread, it puts some work into the queue for the reactor to do, then goes to sleep forever, trusting that the main thread will do the work and then wake it up. Except - oops! - it just put the main thread to sleep forever, so it will never wake up.
If you'd like, you can file a ticket for blockingCallFromThread to raise a loud, angry exception if it is invoked from the reactor thread.
Ok I think this is the turning key. I imagine that if I start the reactor in a separate secondary thread, calling reactor.callWhenRunnning(start_command) from the main thread makes 'start_command' be run in the reactor's thread. At that point I can't call blockingCallFromThread from that thread, as you pointed out. If this assumption is wrong, please let me know and I'll go to study threads.
(Also, in the simple example you gave, you're invoking blockingCallFromThread with the wrong signature; the first argument must be a reactor. The rest of my explanation was based on the assumption that this is not the exact code you're running, since that would log a traceback immediately. Or, perhaps the fact that you're calling callWhenRunning from a non-main thread is blowing up in some more insidious way and that's the real problem.)
Sorry for the confusion. This *is* the correct code but blockingCallFromThread is really IPython.twsitedutil.blockingCallFromThread that wraps twisted.internet.threads.blockingCallFromThread passing the reactor as first arg. thanks again for the valuable explanation sandro *:-) -- Sandro Dentella *:-) http://www.reteisi.org Soluzioni libere per le scuole http://sqlkit.argolinux.org SQLkit home page - PyGTK/python/sqlalchemy
participants (4)
-
Alessandro Dentella -
exarkun＠twistedmatrix.com -
Glyph Lefkowitz -
Michael Thompson