Mailman 3 December 2007 - Twisted

[Twisted-Python] sending images via PB
by Robert Gravina 20 Dec '07

20 Dec '07

Hi All, I'm developing an application using Twisted PB, and I'd like to send image data from server to client for display there. Specifically, I'm trying to display an image (e.g. a PNG, but might be another type) in a wxPython client which should come from a Twisted server somehow, and the client is already connected using PB. I suppose I can just send the raw file data as a string and use that... but just thought I'd ask if anyone has any advice. Robert

2 2

[Twisted-Python] Server/Client bi-directional data-transfer - need a solution
by Sajit Rao 18 Dec '07

18 Dec '07

Hi, *The Goal:* - I'm trying to write a simple server/client app where the client can send the server either a small string like "hello" OR a small file (txt, pdf, ppt,.. let's say < 5MB ). - The server needs to figure out if the data it received is a simple string or a file and symmetrically may decide to send back to the client a simple string or a small file. - As a bonus, It's highly desirable that the transfer be secure - but it's ok for the first version if it isn't *Questions:* What's the cleanest way to accomplish this? I've been looking at the single Howto/Example that exists for *twisted.conch* but still can't understand several things. In particular 1. Should I create two separate channels between server and client - one for simple strings and one for files so that the client or server knows the type of data on that channel? 2. Should the files be mime-encoded at one end and mime-decoded at the other so that file-type is automatically handled? 3. I've read that there is a python package called paramiko that implements ssh2 transfer - how does this compare with twisted.conch? This is my first time playing around with twisted or any kind of networking stuff so any kind of help will be appreciated Thanks! Sajit

3 3

[Twisted-Python] SMS sending with Python
by Daniel Marfil 18 Dec '07

18 Dec '07

Hi, my name is Daniel from Spain, and this is my first post here ;-) I've been looking for something in twisted that allows me to send SMS messages through a server, but got no luck until the moment. Could anyone give me a clue to begin? Thank you in advance, Daniel

5 7

[Twisted-Python] twisted, SMP and clusters
by Doug Farrell 18 Dec '07

18 Dec '07

Hi all, I'm considering using Twisted to build a soft realtime system to control a workflow for a physical production facility. Although there would be some socket work for it to do (XML-RPC, logging, etc) I also like it because of the nice way it can be daemonized. One of my goals for this project is to take better advantage of our multiple Linux servers, each of which has multiple CPU's with dual cores. I'd like to take our workflow objects that are fairly long running state machines and cut them loose on other cores/CPUs/servers if I can. So here's my question: should I be looking at twisted.spread for this kind of work? Can the Parallel Python or Pyro projects be incorporated into Twisted, and if so, what kinds of things should I watch out for, or is it even possible? I'm thinking if I use Parallel Python or Pyro I could make remote calls to our objects in a DeferToThread object and get back the results/errors via callbacks. I'm still really pretty new to Twisted, so forgive my naïve questions if they are so. Thanks in advance, Doug

2 2

[Twisted-Python] Re: Twisted-Python Digest, Vol 45, Issue 20
by Josef Novak 18 Dec '07

18 Dec '07

Thank you for all the feedback, it was quite useful. I'm still quite new to twisted, threads and quite the novice programmer, as my questions indicate; this project has been quite enlightening. As a solution I decided that passing off the compute intensive operations in a proper callback to another (set of) machine(s), which is dedicated to these jobs was a more viable longterm solution than this start-stop madness I initially came up with. However, attempting it initially resulted in a great opportunity for me to learn a bit about threads and processes, so it wasn't necessarily a waste of time. Thanks again! -joe 2007/12/17, twisted-python-request(a)twistedmatrix.com < twisted-python-request(a)twistedmatrix.com>: > > Send Twisted-Python mailing list submissions to > twisted-python(a)twistedmatrix.com > > To subscribe or unsubscribe via the World Wide Web, visit > http://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python > or, via email, send a message with subject or body 'help' to > twisted-python-request(a)twistedmatrix.com > > You can reach the person managing the list at > twisted-python-owner(a)twistedmatrix.com > > When replying, please edit your Subject line so it is more specific > than "Re: Contents of Twisted-Python digest..." > > > Today's Topics: > > 1. Re: Re: Twisted-Python Digest, Vol 45, Issue 18 (glyph(a)divmod.com) > > > ---------------------------------------------------------------------- > > Message: 1 > Date: Sun, 16 Dec 2007 10:13:08 -0000 > From: glyph(a)divmod.com > Subject: Re: [Twisted-Python] Re: Twisted-Python Digest, Vol 45, Issue > 18 > To: twisted-python(a)twistedmatrix.com > Message-ID: > < > 20071216101308.21558.555241615.divmod.xquotient.2970(a)joule.divmod.com> > > Content-Type: text/plain; charset="utf-8"; format="flowed" > > On 10:04 am, josef.robert.novak(a)gmail.com wrote: > >Ok I see this will definitely not work as we are talking about threads > >and > >not processes so os.kill is of no use to me. I guess there is no way > >to > >track these new threads and stop/pause/restart them based on some other > >decision logic? Would have been nice if there were though... -joe > > If you really think about it (and I mean really think about it, computer > science-y stuff, understanding what sharing state in threads really > means, how the Python interpreter happens to work now, considering all > the ways it could work) the idea of stopping and restarting threads is > inherently broken. You might be able to make it work, by accident, for > a few simple cases, but there would be no reasonable way to guarantee > safety except for forcing the explicit use of channels to communicate > between threads. If you're doing that, though, you're effectively using > processes. SIGSTOP and friends work just fine on them :). > > > > ------------------------------ > > _______________________________________________ > Twisted-Python mailing list > Twisted-Python(a)twistedmatrix.com > http://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python > > > End of Twisted-Python Digest, Vol 45, Issue 20 > ********************************************** >

3 4

[Twisted-Python] Weekly Bug Summary
by exarkun＠twistedmatrix.com 17 Dec '07

17 Dec '07

1 0

[Twisted-Python] Re: Twisted-Python Digest, Vol 45, Issue 18
by Josef Novak 16 Dec '07

16 Dec '07

Hmm. This seems like a pretty nice solution, except that it seems it will be blocking incoming calls during processing. What I really need to do is pause the deferToThread objects based on their pid. How can I obtain the pid of a new deferToThread object? If I can obtain it, then I can just keep a persistent dictionary containing all the running compute-intensive processes, and unregister each one in a callback after it finishes up. Then, every time a new call comes in, I check my persistent pid dictionary/list, for pid in pid_list: os.kill(pid, signal.SIGSTOP) and every time a call finishes, I check the number of current calls, and then if we're back to zero, we resume the compute intensive processes again: if parentObj.live_calls == 0:: for pid in parentObj.pid_list: os.kill(pid, signal.SIGCONT) so, my new question is... How can I obtain the pid of a new deferToThread object? I'm pretty sure that, while not very elegant, this should solve my problem - but please let me know if this sounds really, cockamamey! -Joe > ------------------------------ > > Message: 2 > Date: Sat, 15 Dec 2007 07:50:02 -0800 > From: Ed Suominen <general(a)eepatents.com> > Subject: Re: [Twisted-Python] finer control of deferToThread ? > To: Twisted general discussion <twisted-python(a)twistedmatrix.com> > Message-ID: <4763F7AA.2090800(a)eepatents.com> > Content-Type: text/plain; charset=ISO-8859-1 > > You can do priority queuing to one or more threads using AsynQueue, > http://foss.eepatents.com/AsynQueue or python-asynqueue in Debian. Use > multiple threads and queue up the compute-intensive calls with a low > priority and the other stuff with higher priorities. > > However, you will need to chop up your compute-intensive stuff into > smaller pieces for this to be helpful. (That's good asynchronous > processing practice, generally.) The priority queuing is only effective > at deciding which calls to dispatch next, and each thread call is on its > own once it is dispatched from the queue. Each call uses an entire > thread for its entire duration, and will keep the queue from dispatching > anything else to that thread while it's squatting on it, no matter how > low-priority it is. > > Best regards, Ed > > Josef Novak wrote: > > Hi, > > I am writing a twisted application using StarPY fastagi API for > > Asterisk. My application involves answering user calls, and then > > based on their responses to an IVR dialogue, running some some > > compute-intensive applications on the same server, after they hang up. > > At the moment I am running the compute-intensive application (3rd > > party code) in a deferred.deferToThread. The application works fine, > > however if the number of callers goes above 2-3, and one of the > > compute-intensive applications from a previous call has not finished > > up, the audio for the call gets very jumpy because of CPU usage. > > This compute-intensive process needs to be run immediately after > > hangup, and I'd prefer to take care of everything on the same machine > > (rather than send the compute-intensive application request somewhere > > else). > > > > What I'd like to do is pause the thread with the > > deferred.deferToThread process any time a new call comes in (this > > sort of violates what I'm saying above but the number of calls going > > to a particular ASterisk trunk is limited so in most cases this would > > never result in more than a couple-seconds delay, which is > > acceptable). > > > > Is there any way to control these deferred.deferToThread objects in > > a more fine-grained manner? Say from a reactor factory? Can I > > register them separately somehow, and then pause these > > compute-intensive applications temporarily every time a new call comes > > in? I'm imagining something as simple as as ctrl+z and $ fg linux > > terminal commands... but I appreciate that it is probably not so > > straightforward. > > > > -Joe > > > > _______________________________________________ > > Twisted-Python mailing list > > Twisted-Python(a)twistedmatrix.com > > http://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python > > > > > > > > > > ------------------------------ > > _______________________________________________ > Twisted-Python mailing list > Twisted-Python(a)twistedmatrix.com > http://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python > > > End of Twisted-Python Digest, Vol 45, Issue 18 > ********************************************** >

2 3

[Twisted-Python] finer control of deferToThread ?
by Josef Novak 15 Dec '07

15 Dec '07

Hi, I am writing a twisted application using StarPY fastagi API for Asterisk. My application involves answering user calls, and then based on their responses to an IVR dialogue, running some some compute-intensive applications on the same server, after they hang up. At the moment I am running the compute-intensive application (3rd party code) in a deferred.deferToThread. The application works fine, however if the number of callers goes above 2-3, and one of the compute-intensive applications from a previous call has not finished up, the audio for the call gets very jumpy because of CPU usage. This compute-intensive process needs to be run immediately after hangup, and I'd prefer to take care of everything on the same machine (rather than send the compute-intensive application request somewhere else). What I'd like to do is pause the thread with the deferred.deferToThread process any time a new call comes in (this sort of violates what I'm saying above but the number of calls going to a particular ASterisk trunk is limited so in most cases this would never result in more than a couple-seconds delay, which is acceptable). Is there any way to control these deferred.deferToThread objects in a more fine-grained manner? Say from a reactor factory? Can I register them separately somehow, and then pause these compute-intensive applications temporarily every time a new call comes in? I'm imagining something as simple as as ctrl+z and $ fg linux terminal commands... but I appreciate that it is probably not so straightforward. -Joe

3 2

[Twisted-Python] disturbing trial exceptions
by Markus Schiltknecht 14 Dec '07

14 Dec '07

Hi, since my last svn update of twisted, some strange errors started popping up when testing my package with trial. I've tracked down the exact revision with which the errors started to appear: > ------------------------------------------------------------------------ > r21907 | therve | 2007-11-20 20:32:56 +0100 (Tue, 20 Nov 2007) | 9 lines > > Merge test-plugins-not-shared-942 > > Author: therve > Reviewers: exarkun, jml, glyph > Fixes #942 > > Fix twisted.test.test_plugin so that it doesn't use twisted.plugins anymore, > but create temporary plugins beneath _trial_temp instead. > > ------------------------------------------------------------------------ The first exception is: exceptions.AttributeError: 'module' object has no attribute 'running' As the errors are from seemingly dynamically generated code (at least I could only find .pyc files, no source), I didn't try to investigate further. After the errors, all tests are correctly run through, so it's just plain disturbing. Here's the full error output: > Traceback (most recent call last): > File "/home/markus/projects/twisted/twisted/python/usage.py", line 241, in parseOptions > self.postOptions() > File "/home/markus/projects/twisted/twisted/scripts/trial.py", line 292, in postOptions > self['reporter'] = self._loadReporterByName(self['reporter']) > File "/home/markus/projects/twisted/twisted/scripts/trial.py", line 278, in _loadReporterByName > for p in plugin.getPlugins(itrial.IReporter): > File "/home/markus/projects/twisted/twisted/plugin.py", line 200, in getPlugins > allDropins = getCache(package) > --- <exception caught here> --- > File "/home/markus/projects/twisted/twisted/plugin.py", line 165, in getCache > provider = pluginModule.load() > File "/home/markus/projects/twisted/twisted/python/modules.py", line 380, in load > return self.pathEntry.pythonPath.moduleLoader(self.name) > File "/home/markus/projects/twisted/twisted/python/modules.py", line 618, in moduleLoader > return self._moduleLoader(modname) > File "/home/markus/projects/twisted/twisted/python/reflect.py", line 361, in namedAny > topLevelPackage = __import__(trialname) > File "/home/markus/projects/twisted/twisted/plugins/notestplugin.py", line 18, in ? > > exceptions.AttributeError: 'module' object has no attribute 'running' > Traceback (most recent call last): > File "/home/markus/projects/twisted/twisted/python/usage.py", line 241, in parseOptions > self.postOptions() > File "/home/markus/projects/twisted/twisted/scripts/trial.py", line 292, in postOptions > self['reporter'] = self._loadReporterByName(self['reporter']) > File "/home/markus/projects/twisted/twisted/scripts/trial.py", line 278, in _loadReporterByName > for p in plugin.getPlugins(itrial.IReporter): > File "/home/markus/projects/twisted/twisted/plugin.py", line 200, in getPlugins > allDropins = getCache(package) > --- <exception caught here> --- > File "/home/markus/projects/twisted/twisted/plugin.py", line 165, in getCache > provider = pluginModule.load() > File "/home/markus/projects/twisted/twisted/python/modules.py", line 380, in load > return self.pathEntry.pythonPath.moduleLoader(self.name) > File "/home/markus/projects/twisted/twisted/python/modules.py", line 618, in moduleLoader > return self._moduleLoader(modname) > File "/home/markus/projects/twisted/twisted/python/reflect.py", line 361, in namedAny > topLevelPackage = __import__(trialname) > File "/home/markus/projects/twisted/twisted/plugins/testplugin.py", line 12, in ? > > exceptions.ImportError: cannot import name ITestPlugin Help appreciated. Regards Markus

3 3

[Twisted-Python] Conch/insults -- using HistoricRecvLine
by Bjoern Schliessmann 13 Dec '07

13 Dec '07

Hello, I'm using a "standard" Twisted setup here: a ServerFactory which has basic.LineReceiver subclass as protocol for handling connections; I'm using methods like connectionMade and lineReceived. Now I tried the example on HistoricRecvLine from Twisted Conch and I like it. I tried my protocol to subclass HistoricRecvLine directly instead of basic.LineReceiver, but the server won't work anymore (it complains about not finding certain methods as soon as a connection comes in). So HistoricRecvLine doesn't seem to be compatible to standard Core protocols. I had a closer look at the demo_recvline.py example and, besides under ten lines of real code defining functional stuff, I saw a real mess of wrapper code. Several different classes are used to hook up the DemoRecvLine class into the ServerFactory. (This simple example is, IMHO, a huge difference to the elegance of simple Twisted Core examples.) Those wrapper classes are documented in a way that you don't understand what one class does unless you understand what some few other classes and interfaces do. Those others are documented in a similar way ;) There is one tutorial for conch which is in my view completely unrelated. No matter what, I tried to adapt and/or wrap my ex-basic.LineReceiver-and-now-HistoricRecvLine subclass in a way that I can hook it up to my ServerFactory class, but to no avail for hours. That's why I'm asking here. Could you please tell me what exactly I need to do to "convert" my new HistoricRecvLine subclass in a way that ServerFactory will accept it? Or is there even a simpler way to get basic readline-like functionality in Twisted? Because that's what I'm aiming at. Regards, Björn -- BOFH excuse #403: Sysadmin didn't hear pager go off due to loud music from bar-room speakers.

2 2