Mailman 3 November 2009 - Twisted

[Twisted-Python] Farming off work to other web services
by Ted Pederson 18 Nov '09

18 Nov '09

How can my server utilize other web services? Is there part of the framework to do this, or should I just write a mini-client? Basically, I want to use a very simple web service, a url with a couple of params, and I get back xml or json data. Sorry for fundamental nature of this question. Perhaps I should be redirected to a book or docs, but the docs are pretty large and I am just getting my feet wet.

2 1

[Twisted-Python] A non-blocking socket operation could not be completed immediately
by Tommaso Mazzafico 18 Nov '09

18 Nov '09

I tried this code: ----------------------------------------- from twisted.spread import pb from twisted.internet import reactor class Echoer(pb.Root): def remote_echo(self, s): return s if __name__ == "__main__": reactor.listenTCP(8001, pb.PBServerFactory(Echoer())) reactor.run() ----------------------------------------- On Windows XP all is ok. On Windows 7 64 I receive this error message: ----------------------------------------- C:\Program Files (x86)\Python\lib\site-packages\twisted\spread\pb.py:30: DeprecationWarning: the md5 module is deprecated; use hashlib instead import md5 Traceback (most recent call last): File "C:\Users\bob\Documents\Twisted\pbs.py", line 11, in <module> reactor.listenTCP(8001, pb.PBServerFactory(Echoer())) File "C:\Program Files (x86)\Python\lib\site-packages\twisted\internet\posixbase.py", line 356, in listenTCP p.startListening() File "C:\Program Files (x86)\Python\lib\site-packages\twisted\internet\tcp.py", line 862, in startListening self._realPortNumber = skt.getsockname()[1] File "<string>", line 1, in getsockname socket.error: [Errno 10035] A non-blocking socket operation could not be completed immediately ----------------------------------------- What can I do?

2 2

[Twisted-Python] server push data to client
by vitaly＠synapticvision.com 18 Nov '09

18 Nov '09

hi, what could be the way(Producer/Consumer?) and simple,basic example to push data(~10Kb) from Twisted based server(that listening for incoming requests) into client side(for example FLEX) not as a response to the incoming request, but occasionally, triggered by server side??? In traditional client/server arch, the server is listening, accepting and responding to the client's requests, but here I'm looking for something that initiated from server itself and pushing data to client. Appreciate the help.

2 2

[Twisted-Python] Proposal (almost) to switch from Lore to Sphinx
by Kevin Horn 18 Nov '09

18 Nov '09

Back in July, there was a long discussion thread regarding Twisted Documentation, and the idea of switching to Sphinx came up. Here's probably the culmination of that thread: http://twistedmatrix.com/pipermail/twisted-python/2009-July/020114.html To summarize, Glyph laid out the following items that would need to happen in order for Twisted to move from Lore to another documentation system: 1. Someone would have to volunteer to maintain the documentation infrastructure and teach others to use it. 2. Submit a complete proposal outlining benefits for the move so that it could be discussed rationally without having to chase down lots of little bits in email, irc, etc. 3. Address the transition process. So I went away and did some thinking, and some research and some experimenting, and I decided that I really want this to happen. So I am hereby volunteering to manage a Sphinx-based documentation infrastructure (should there eventually be one) for the Twisted project as well as the transition to said system. I have written up a proposal here: http://twistedsphinx.funsize.net/proposal.html that I think addresses at least the main points of what should need to happen in order to move from Lore to Sphinx. It's not quite finished, but I think it's close enough for people to start discussing it. In particular, there are a few claims I've made about Lore and it's capabilities that I'm not quite sure of (just search for the question marks). I would very much like to get some feedback from the community (especially the core developers) on this proposal. I'd especially like to hear about: - things which are not addressed and should be. - anything that is addressed in the proposal, but you think should be done in a different way. - Lore's capabilities. Please let me know what you think. Thanks, Kevin Horn

7 21

[Twisted-Python] pb, viewable, avatars: The example chatserver is buggy
by Wolfgang Rohdewald 18 Nov '09

18 Nov '09

I tried this example: http://twistedmatrix.com/projects/core/documentation/howto/pb-cred.html#aut… (scripts chatserver.py and chatclient.py) but they do not work with python 2.6.4 (twisted 8.2.0). Before the client can do much, the server function User.detached is called resulting in an exception later when User.send is called but self.remote is None. I can fix this by changing chatclient.py: def connected(self, perspective): self.perspective = perspective ... and continue as in the original script it seems that otherwise the perspective is destroyed too early. also the documentation should perhaps mention that the client can only connect to the server once because the server code does not remove a user from her group when she detaches. Is there any more complete example anywhere with better error handling? -- Wolfgang

1 0

Re: [Twisted-Python] tee method in twisted.internet.defer.py ?
by Terry Jones 17 Nov '09

17 Nov '09

>>>>> "JP" == exarkun <exarkun(a)twistedmatrix.com> writes: JP> We implemented this 5 or 6 times at Divmod. I'm not surprised. Given that I'd have thought it would've made it into defer.py at some point. JP> The most recent incarnation looks like this: JP> http://divmod.org/trac/browser/trunk/Epsilon/epsilon/pending.py Thanks. That's more of a radial / hub pattern. I'm dealing with the case of already having a deferred around and wanting to tee off it without interfering with its callback/errback chain. E.g., I have a class that's already passed a deferred back to its caller, and I later need to arrange to do some more stuff when the deferred fires. The caller of my class may in the meantime have added callbacks to the deferred, I tee off it with no effect on my original caller. That's just one example use. I'm sure you'll have implemented something for similar situations, too. JP> Not to say it's a bad idea, but I don't think it's _obviously_ a great JP> idea. If someone wanted to contribute a documented, tested JP> implementation, particularly with examples of common patterns that it JP> simplifies, then you could probably convince someone to commit it. I don't need it committed, I was just wondering why something like that doesn't already exist. Re patterns: it would be great to have a series of diagrams illustrating situations one might want to deal with and showing how these can be done with Deferred code. Your Epsilon class could be shown as a hub with new listeners arriving, ready for the triggering of the callback. The tee diagram would show a T (or multiple Ts on the same callback chain) in the obvious way. add{Callback,Errback} is like a normal UNIX pipe and, stretching things a little, chainDeferred reminds me a bit of UNIX exec (though with real differences). There's a PyCon talk in there somewhere. Like Patterns of Asynchronous Flow of Control using Twisted Deferreds. It often feels to me like there's a lurking taxonomy of Deferred use cases or building blocks that's just waiting for someone to come along and make really clear to the world. I'd really love to do it, but I just don't have the time. Terry

2 1

[Twisted-Python] Multiple deferreds that depend on each other
by Landreville 17 Nov '09

17 Nov '09

I know people ask about deferreds all the time (and i've read through a bunch of those questions), but there is still one thing I don't quite grasp. If I want to do a number of actions in a row that each return a deferred but depend on eachother, do I need to nest those callbacks? Ie if i want to: 1. Call database function using adbabi 2. Call xmlrpc method using result from 1 3. Call database using result from 2 Would I just put 2 inside the callback for 1 and then put 3 inside the callback for 2? How would I do that in one function? All the examples I have seen have a function calling one method returning a deferred and then placing the logic (doing something with the result) in the callback, but do not deal with multiple deferreds that depend on each other.

2 2

[Twisted-Python] tee method in twisted.internet.defer.py ?
by Terry Jones 17 Nov '09

17 Nov '09

Is there a reason why the Deferred class doesn't have a tee method like the following (untested)? def tee(self, d): def _cb(arg): d.callback(arg) return arg def _eb(arg): d.errback(arg) return arg return self.addCallbacks(_cb, _eb) I suppose it could be called teeDeferred to correspond to chainDeferred. And I guess it's clear what it does and why it's quite different from chainDeferred. I at least would find it quite useful. Terry

2 1

[Twisted-Python] Telnet server: localecho "ON" for linux client but not for windows client
by filoufake-python＠yahoo.fr 17 Nov '09

17 Nov '09

Hello, Some month ago, exarkun helped me in the development of a very simplified telnet server. I observed that when I connect from a windows client, the LOCALECHO is set to "OFF" after the authentication. I cannot set it "ON" afterwards. Everything typed in the client console is hidden. When I use a linux client, the "ECHO" is OFF just for typing the password and then ON again. Does someone has an idea how to do with the windows client? I tried to set the ECHO ON using self.transport.will(ECHO) but that doesn't work. Here is the code: ############################################################ import sys from zope.interface import implements from twisted.internet import protocol from twisted.python import log from twisted.cred import error from twisted.cred import portal from twisted.cred import checkers from twisted.cred import credentials from twisted.conch.telnet import AuthenticatingTelnetProtocol from twisted.conch.telnet import StatefulTelnetProtocol from twisted.conch.telnet import ITelnetProtocol from twisted.conch.telnet import TelnetTransport from twisted.conch.telnet import ECHO class Realm: implements(portal.IRealm) def requestAvatar(self, avatarId, mind, *interfaces): if ITelnetProtocol in interfaces: av = MyTelnet() av.transport.will(ECHO) av.state = 'Command' return ITelnetProtocol, av, lambda:None raise NotImplementedError("Not supported by this realm") class MyTelnet(StatefulTelnetProtocol): def telnet_Command(self, line): print "line received via telnet", line self.sendLine('coucou man, you send me %s' % line) def main(): r = Realm() p = portal.Portal(r) c = checkers.InMemoryUsernamePasswordDatabaseDontUse() c.addUser("AA", "aa") p.registerChecker(c) p.registerChecker(checkers.AllowAnonymousAccess()) f = protocol.ServerFactory() f.protocol = lambda: TelnetTransport(AuthenticatingTelnetProtocol, p) log.startLogging(sys.stdout) from twisted.internet import reactor reactor.listenTCP(4738, f) reactor.run() if __name__ == '__main__': main() ############################################################ Best Regards, Philippe

2 2

[Twisted-Python] ReverseProxy (http) using Twisted
by Fred C 17 Nov '09

17 Nov '09

I have to write a quick and dirty ReverseProxy with Twisted, for one of our internal project. I never used Twisted before, and I was wondering of someone have an handy example of a ReverseProxy with Twisted to help that I can use as bootstrap. Many thanks. Fred

2 3

Re: [Twisted-Python] exarkun changed [#4029 - Twisted Web's wsgi.input isn't sufficiently file-like in all cases]
by Phil Christensen 17 Nov '09

17 Nov '09

-phil [sent from my phone] On Nov 16, 2009, at 4:56 PM, trac(a)twistedmatrix.com wrote: > Ticket [#4029 - Twisted Web's wsgi.input isn't sufficiently file- > like in all cases] changed by exarkun <exarkun(a)twistedmatrix.com>: > > (In [27518]) Merge wsgi-none-readline-4029 > > Author: rikyu > Reviewer: exarkun > Fixes: #4029 > Refs: #4114 > > Add a test for WSGI requests which end up with a > `tempfile.TemporaryFile` > for `Request.content` and fix the implementation to deal with this > case > by not passing `None` for the length argument to `readline` (which > temp > files do not accept, unlike StringIOs). WSGI applications can now > receive > requests with large bodies. > > --------- > status : assigned -> closed > resolution: -> fixed > --------- > http://twistedmatrix.com/trac/ticket/4029

1 1

[Twisted-Python] Caching mechanism
by vitaly＠synapticvision.com 16 Nov '09

16 Nov '09

hi, is there any cache mechanism for twisted? Could one point me please to that docs? Thank you.

7 12

[Twisted-Python] Weekly Bug Summary
by exarkun＠twistedmatrix.com 15 Nov '09

15 Nov '09

1 0

[Twisted-Python] TCP Queued connections
by tim wylie 15 Nov '09

15 Nov '09

Hello, What is the best way to set up a queue of tcp connections? I'm currently using perspective broker for authentication and communication. I want the client to register with the server, and then the server send back a message telling them their place in line. This is easy, but then how can I store the connections on the server to send the client a message when it's his turn? I realize I could store IP addresses, but then you lose connections behind firewalls/etc. I could also give the client a time estimate and he could run a deferred call after that time, but I was hoping there was a more elegant way. Any thoughts? Thanks. Tim _________________________________________________________________ Hotmail: Trusted email with Microsoft's powerful SPAM protection. http://clk.atdmt.com/GBL/go/177141664/direct/01/ http://clk.atdmt.com/GBL/go/177141664/direct/01/

1 0

[Twisted-Python] Deferred SOAP methods
by Landreville 14 Nov '09

14 Nov '09

I am making a SOAP server and all (or at least the vast majority) will be returning a deferred. The whole SOAP method is encapsulated in a deferred in most of my cases (I havent implemented this part yet.) Then I got the idea that twisted.web.soap could just wrap my SOAP method (retrieved with lookupFunction) in a deferred and return that deferred right away. For instance I would change twisted.web.soap.render where it says d = defer.maybeDeferred to d = threads.deferToThread I could also use an attribute to specify if I do not want it deferred (kinda the same way to determine whether to use keywords or not) Is this actually a good idea? I figured there might be a reason this wasn't done already considering everything in twisted is pretty well thought out.

3 7

[Twisted-Python] Getting local UDP address/port
by Olivier Parisy 14 Nov '09

14 Nov '09

I am implementing a UDP protocol where a request is sent to a server through a connected UDP transport. I listen for the answer using reactor.listenUDP(0, myDatagramProtocol). The local port is not in itself significant (hence the 0), but the protocol require that I get the local IP address and port of the socket used to send the request (so that I can properly process the answser). How can I get this information ? datagramReceived only provides me with the source address and port of the answer (as can be expected). Regards, Olivier.

2 2

Re: [Twisted-Python] synchronous/asynchronous api: possible interface
by exarkun＠twistedmatrix.com 13 Nov '09

13 Nov '09

On 04:03 pm, doug.farrell(a)gmail.com wrote: >Jean-Paul, > >Thanks for the "heads up", that certainly falls under the D'OH! >category. Would you have any suggestions for how I might do this kind >of thing? I already mentioned SynchronousDeferred, so I guess you're not interested in that. :) The main other idea that comes to mind is, in the synchronous case, to start the reactor in a separate thread and leave it running. You can ask it to do things with reactor.callFromThread. Or maybe you can use twisted.internet.threads.blockingCallFromThread. Jean-Paul

1 0

Re: [Twisted-Python] synchronous/asynchronous api: possible interface
by Tom Leys 13 Nov '09

13 Nov '09

Hi Doug. I think that a decorator is a fairly nice idea, however you will probably run into the following issue: http://twistedmatrix.com/trac/ticket/2066 Once a reactor has been started and stopped, starting the reactor again will crash your program. You will need to come up with a way to get around this problem if you plan to call your database code more than once without restarting the python process. -Tom ------------------------------ Message: 9 Date: Fri, 13 Nov 2009 09:32:25 -0500 From: Doug Farrell <doug.farrell(a)gmail.com> Subject: [Twisted-Python] synchronous/asynchronous api: possible interface To: Twisted general discussion <twisted-python(a)twistedmatrix.com> Message-ID: <80c54aa0911130632l63cad82aq54a93cff139c8b02(a)mail.gmail.com> Content-Type: text/plain; charset=ISO-8859-1 Hi all, I'd like to get some comments on the code below (including, "don't be a bonehead!" <g>). I'm trying to write a API library for database access that can be used by both synchronous (non-twisted, no reactor) code and asynchronous (twisted, with reactor) code where the methods of the library can called in either environment. What I'd like is the method to return the results of the call (database query results) when in synchronous mode, and a deferred that is a result of the database call in asynchronous mode. What I've done below is create a decorator class called CallerMode that is used to decorate a function/method that normally returns a deferred. If the CallerMode class is in synchronous mode it starts a reactor to get the default callbacks to get called, which gets the results or exceptions, and stops the reactor. Am I being a dunce doing this kind of thing? Any suggestions, comments, etc. are welcome. Thanks!

1 0

[Twisted-Python] Multi-reactor architecture
by vitaly＠synapticvision.com 13 Nov '09

13 Nov '09

hi, what would be the right thing to start from in order to build multi-reactor arch to handle thousands of concurrent connections? Appreciate the help.

4 12

[Twisted-Python] synchronous/asynchronous api: possible interface
by Doug Farrell 13 Nov '09

13 Nov '09

Hi all, I'd like to get some comments on the code below (including, "don't be a bonehead!" <g>). I'm trying to write a API library for database access that can be used by both synchronous (non-twisted, no reactor) code and asynchronous (twisted, with reactor) code where the methods of the library can called in either environment. What I'd like is the method to return the results of the call (database query results) when in synchronous mode, and a deferred that is a result of the database call in asynchronous mode. What I've done below is create a decorator class called CallerMode that is used to decorate a function/method that normally returns a deferred. If the CallerMode class is in synchronous mode it starts a reactor to get the default callbacks to get called, which gets the results or exceptions, and stops the reactor. Am I being a dunce doing this kind of thing? Any suggestions, comments, etc. are welcome. Thanks! Doug import sys from functools import wraps from twisted.internet import reactor from twisted.python import log from twisted.web.client import getPage class CallerMode(object): '''This is a decorator class to make calls to an api play nice in synchronous or asynchronous mode''' ASYNCH = 'asynch' SYNCH = 'synch' STATE = SYNCH def __init__(self): self.retval = None def __call__(self, f): '''This provides the decorator function wrapper, which will return a deferred for asynchronous mode and the results of the wrapped function in synchronous mode''' def CB(result): self.retval = result reactor.stop() def EB(failure): reactor.stop() failure.raiseException() @wraps(f) def func(*args, **kwargs): d = f(*args, **kwargs) if CallerMode.STATE == CallerMode.SYNCH: d.addCallback(CB).addErrback(EB) reactor.run() return self.retval if CallerMode.STATE == CallerMode.SYNCH else d return func class Library(object): '''This is just a class to provide an API to call for data, in this case just using getPage() to get a deferred that gets some data, vs. having to set up a database for this test program ''' def __init__(self): log.msg('MyLib has been initialized') self._retval = None @CallerMode() def page(self): return getPage('http://www.google.com') def doSomethingWithData(data): log.msg(data) log.FileLogObserver.timeFormat = '%Y-%m-%d %H:%M:%S' log.startLogging(sys.stdout) lib = Library() #CallerMode.STATE = CallerMode.ASYNCH CallerMode.STATE = CallerMode.SYNCH # call the library API method result = lib.page() # depending on the CallerMode, handle the results differently if CallerMode.STATE == CallerMode.SYNCH: doSomethingWithData(result) elif CallerMode.STATE == CallerMode.ASYNCH: result.addCallback(doSomethingWithData) reactor.run()

2 1

Re: [Twisted-Python] Amount of concurrent connections
by vitaly＠synapticvision.com 12 Nov '09

12 Nov '09

Sorry, forgot to mention, per second. Amount of concurrent connections per second. Quoting "Eero Nevalainen" <eero.nevalainen(a)indagon.com>: > vitaly(a)synapticvision.com wrote: >> hi, >> what would be expected amount(+-) of concurrent connections for >> twisted reactor if single request to gateway eq 1Kb and response >> back 30Kb (I mean response data much more bigger than request)? >> (1Gb network card on twisted machine) >> >> Appreciate the help. > > If a connection arrives once a day, I'd say a very low amount. > > Be more specific... > > -- > Eero Nevalainen >

2 1

[Twisted-Python] Amount of concurrent connections
by vitaly＠synapticvision.com 12 Nov '09

12 Nov '09

hi, what would be expected amount(+-) of concurrent connections for twisted reactor if single request to gateway eq 1Kb and response back 30Kb (I mean response data much more bigger than request)? (1Gb network card on twisted machine) Appreciate the help.

2 2

[Twisted-Python] Thrift/Twisted reconnecting client .. code review?
by David Koblas 12 Nov '09

12 Nov '09

I just got done putting together a reconnecting client (with queuing) for twisted. While it appears to work for me in practice, if anybody has a few minutes to do a quick code review that would be really handy. This is based on another thrift client, but in the end totally gutted and now hopefully works, though I have to say there are always enough edge cases with the twisted model that I'm never sure. Code is available here.. http://www.skitoy.com/p/twisted-code-review/261 Thanks, --david

1 0

[Twisted-Python] Using sqlalchemy in twisted.
by Peter Cai 12 Nov '09

12 Nov '09

If you Google "sqlalchemy twisted", you can see my post is on the top: http://groups.google.com/group/sqlalchemy/browse_thread/thread/c802749c9d0d… I've been busy for a long time so I didn't considered this question anymore. But occasionally, I got some new ideas today. I was suggested to sandbox sqlalchemy into process contains a single thread. At that time, I thought it was just the same as using deferToThread and makes no scenes to have the overhead of IPC. But now I see the benefits. A single thread process makes it possible to use a sqlalchemy session as a cache. That would greatly reduce the chances to lookup database. This could not be done with deferToThread because it uses thread pool. So there shall be more than one session and the state of cached objects will conflict. But there are still one problem : how to transfer a sqlalchemy object between 2 process? Some kind of serialization and de-serialization is needed. Without test, it's too early to tell if it works. -- look to the things around you,the immediate world around you, if you are alive,it will mean something to you ——Paul Strand

3 2

[Twisted-Python] Twisted Client
by morbidux 11 Nov '09

11 Nov '09

Hi all, I'm writing a simple client / server application, that uses a small protocol to dialog. I can't find in documentation any exemple that would correspond to what my client is supposed to do. Here is the basics : every X seconds, call a function that check a file if file is different, connect to server When the client is connected, it immediatly sends data to server, using the connectionMade method. Do I need to use while(1) loop or loopingCall ? What is the best way to write a daemon that connect to a server every X seconds ? Many thanks.

4 4