Mailman 3 November 2009 - Twisted

[Twisted-Python] Getting local UDP address/port
by Olivier Parisy 14 Nov '09

14 Nov '09

I am implementing a UDP protocol where a request is sent to a server through a connected UDP transport. I listen for the answer using reactor.listenUDP(0, myDatagramProtocol). The local port is not in itself significant (hence the 0), but the protocol require that I get the local IP address and port of the socket used to send the request (so that I can properly process the answser). How can I get this information ? datagramReceived only provides me with the source address and port of the answer (as can be expected). Regards, Olivier.

2 2

Re: [Twisted-Python] synchronous/asynchronous api: possible interface
by exarkun＠twistedmatrix.com 13 Nov '09

13 Nov '09

On 04:03 pm, doug.farrell(a)gmail.com wrote: >Jean-Paul, > >Thanks for the "heads up", that certainly falls under the D'OH! >category. Would you have any suggestions for how I might do this kind >of thing? I already mentioned SynchronousDeferred, so I guess you're not interested in that. :) The main other idea that comes to mind is, in the synchronous case, to start the reactor in a separate thread and leave it running. You can ask it to do things with reactor.callFromThread. Or maybe you can use twisted.internet.threads.blockingCallFromThread. Jean-Paul

1 0

Re: [Twisted-Python] synchronous/asynchronous api: possible interface
by Tom Leys 13 Nov '09

13 Nov '09

Hi Doug. I think that a decorator is a fairly nice idea, however you will probably run into the following issue: http://twistedmatrix.com/trac/ticket/2066 Once a reactor has been started and stopped, starting the reactor again will crash your program. You will need to come up with a way to get around this problem if you plan to call your database code more than once without restarting the python process. -Tom ------------------------------ Message: 9 Date: Fri, 13 Nov 2009 09:32:25 -0500 From: Doug Farrell <doug.farrell(a)gmail.com> Subject: [Twisted-Python] synchronous/asynchronous api: possible interface To: Twisted general discussion <twisted-python(a)twistedmatrix.com> Message-ID: <80c54aa0911130632l63cad82aq54a93cff139c8b02(a)mail.gmail.com> Content-Type: text/plain; charset=ISO-8859-1 Hi all, I'd like to get some comments on the code below (including, "don't be a bonehead!" <g>). I'm trying to write a API library for database access that can be used by both synchronous (non-twisted, no reactor) code and asynchronous (twisted, with reactor) code where the methods of the library can called in either environment. What I'd like is the method to return the results of the call (database query results) when in synchronous mode, and a deferred that is a result of the database call in asynchronous mode. What I've done below is create a decorator class called CallerMode that is used to decorate a function/method that normally returns a deferred. If the CallerMode class is in synchronous mode it starts a reactor to get the default callbacks to get called, which gets the results or exceptions, and stops the reactor. Am I being a dunce doing this kind of thing? Any suggestions, comments, etc. are welcome. Thanks!

1 0

[Twisted-Python] Multi-reactor architecture
by vitaly＠synapticvision.com 13 Nov '09

13 Nov '09

hi, what would be the right thing to start from in order to build multi-reactor arch to handle thousands of concurrent connections? Appreciate the help.

4 12

[Twisted-Python] synchronous/asynchronous api: possible interface
by Doug Farrell 13 Nov '09

13 Nov '09

Hi all, I'd like to get some comments on the code below (including, "don't be a bonehead!" <g>). I'm trying to write a API library for database access that can be used by both synchronous (non-twisted, no reactor) code and asynchronous (twisted, with reactor) code where the methods of the library can called in either environment. What I'd like is the method to return the results of the call (database query results) when in synchronous mode, and a deferred that is a result of the database call in asynchronous mode. What I've done below is create a decorator class called CallerMode that is used to decorate a function/method that normally returns a deferred. If the CallerMode class is in synchronous mode it starts a reactor to get the default callbacks to get called, which gets the results or exceptions, and stops the reactor. Am I being a dunce doing this kind of thing? Any suggestions, comments, etc. are welcome. Thanks! Doug import sys from functools import wraps from twisted.internet import reactor from twisted.python import log from twisted.web.client import getPage class CallerMode(object): '''This is a decorator class to make calls to an api play nice in synchronous or asynchronous mode''' ASYNCH = 'asynch' SYNCH = 'synch' STATE = SYNCH def __init__(self): self.retval = None def __call__(self, f): '''This provides the decorator function wrapper, which will return a deferred for asynchronous mode and the results of the wrapped function in synchronous mode''' def CB(result): self.retval = result reactor.stop() def EB(failure): reactor.stop() failure.raiseException() @wraps(f) def func(*args, **kwargs): d = f(*args, **kwargs) if CallerMode.STATE == CallerMode.SYNCH: d.addCallback(CB).addErrback(EB) reactor.run() return self.retval if CallerMode.STATE == CallerMode.SYNCH else d return func class Library(object): '''This is just a class to provide an API to call for data, in this case just using getPage() to get a deferred that gets some data, vs. having to set up a database for this test program ''' def __init__(self): log.msg('MyLib has been initialized') self._retval = None @CallerMode() def page(self): return getPage('http://www.google.com') def doSomethingWithData(data): log.msg(data) log.FileLogObserver.timeFormat = '%Y-%m-%d %H:%M:%S' log.startLogging(sys.stdout) lib = Library() #CallerMode.STATE = CallerMode.ASYNCH CallerMode.STATE = CallerMode.SYNCH # call the library API method result = lib.page() # depending on the CallerMode, handle the results differently if CallerMode.STATE == CallerMode.SYNCH: doSomethingWithData(result) elif CallerMode.STATE == CallerMode.ASYNCH: result.addCallback(doSomethingWithData) reactor.run()

2 1

Re: [Twisted-Python] Amount of concurrent connections
by vitaly＠synapticvision.com 12 Nov '09

12 Nov '09

Sorry, forgot to mention, per second. Amount of concurrent connections per second. Quoting "Eero Nevalainen" <eero.nevalainen(a)indagon.com>: > vitaly(a)synapticvision.com wrote: >> hi, >> what would be expected amount(+-) of concurrent connections for >> twisted reactor if single request to gateway eq 1Kb and response >> back 30Kb (I mean response data much more bigger than request)? >> (1Gb network card on twisted machine) >> >> Appreciate the help. > > If a connection arrives once a day, I'd say a very low amount. > > Be more specific... > > -- > Eero Nevalainen >

2 1

[Twisted-Python] Amount of concurrent connections
by vitaly＠synapticvision.com 12 Nov '09

12 Nov '09

hi, what would be expected amount(+-) of concurrent connections for twisted reactor if single request to gateway eq 1Kb and response back 30Kb (I mean response data much more bigger than request)? (1Gb network card on twisted machine) Appreciate the help.

2 2

[Twisted-Python] Thrift/Twisted reconnecting client .. code review?
by David Koblas 12 Nov '09

12 Nov '09

I just got done putting together a reconnecting client (with queuing) for twisted. While it appears to work for me in practice, if anybody has a few minutes to do a quick code review that would be really handy. This is based on another thrift client, but in the end totally gutted and now hopefully works, though I have to say there are always enough edge cases with the twisted model that I'm never sure. Code is available here.. http://www.skitoy.com/p/twisted-code-review/261 Thanks, --david

1 0

[Twisted-Python] Using sqlalchemy in twisted.
by Peter Cai 12 Nov '09

12 Nov '09

If you Google "sqlalchemy twisted", you can see my post is on the top: http://groups.google.com/group/sqlalchemy/browse_thread/thread/c802749c9d0d… I've been busy for a long time so I didn't considered this question anymore. But occasionally, I got some new ideas today. I was suggested to sandbox sqlalchemy into process contains a single thread. At that time, I thought it was just the same as using deferToThread and makes no scenes to have the overhead of IPC. But now I see the benefits. A single thread process makes it possible to use a sqlalchemy session as a cache. That would greatly reduce the chances to lookup database. This could not be done with deferToThread because it uses thread pool. So there shall be more than one session and the state of cached objects will conflict. But there are still one problem : how to transfer a sqlalchemy object between 2 process? Some kind of serialization and de-serialization is needed. Without test, it's too early to tell if it works. -- look to the things around you,the immediate world around you, if you are alive,it will mean something to you ——Paul Strand

3 2

[Twisted-Python] Twisted Client
by morbidux 11 Nov '09

11 Nov '09

Hi all, I'm writing a simple client / server application, that uses a small protocol to dialog. I can't find in documentation any exemple that would correspond to what my client is supposed to do. Here is the basics : every X seconds, call a function that check a file if file is different, connect to server When the client is connected, it immediatly sends data to server, using the connectionMade method. Do I need to use while(1) loop or loopingCall ? What is the best way to write a daemon that connect to a server every X seconds ? Many thanks.

4 4