Mailman 3 May 2005 - Twisted

[Twisted-Python] need help with waitForDeferred...
by Thomas Jung May 10, 2005

May 10, 2005

Hi everybody, I am new to twisted, and still experimenting. Found the following code somewhere, looks very interesting, but when I run the script (python 2.3.3), just nothing happens.... What am I missing ? Any hint greatly appreciated, thank you ! from twisted.internet import reactor from twisted.internet import defer waitFor = defer.waitForDeferred def getSomeDeferred(): d = defer.Deferred() reactor.callLater( 2, d.callback, 'This is a string that yells "foo!"') … [View More]

2 2

[Twisted-Python] I could swear I've seen this pattern *somewhere* in Twisted...
by Mike C. Fletcher May 10, 2005

May 10, 2005

I have a little utility module "parallel", that dispatches a callable for "count" items in parallel, collecting them as would a DeferredList (attached for reference). The idea here is that you want only count items to be actively processing at any given time, and you want to be notified when the entire suite of items completes processing (just like a DeferredList). I use this when I have, for instance, 10,000 items to dispatch as part of a "batch" process, but I don't want to negatively … [View More]impact the responsiveness of the user's front-end to the system by adding 10,000 active operations to the internal Twisted queues. The *correct* solution for these problems would probably be to have a priority-handling mechanism, to allow those 10,000 items not to interfere with high-priority events, but until we have that, it's necessary to do these kinds of things to keep the system responsive with large batch operations occurring in the background. My question being (yay, finally got around to it), where is this pattern in Twisted. As I said in the subject, I think I've seen *something* described as implementing this pattern, and I'd rather use a standard mechanism than something I've constructed myself. Enjoy all, Mike -- ________________________________________________ Mike C. Fletcher Designer, VR Plumber, Coder http://www.vrplumber.com http://blog.vrplumber.com """Parallelise X items from a set, keeping no more than count items active""" from twisted.internet import reactor, defer from twisted.python import failure class Parallel( object ): """Class managing parallel dispatch across items in an iterable Usage is intended to look very much like a DeferredList, save that items are dispatched so that no more than a given number are active at any given time, so that the system doesn't bog down processing a large number of items from the iterable. For instance, if you need to process 10,000 items with a database operation, you often do *not* want to dispatch all 10,000 database operations, as then any interactive operations will have to wait for all 10,000 to complete before the system can respond to a user query. There probably is *something* in Twisted core that can do this (I even have a rough memory of reading about it), I just couldn't find it when I went looking. p = parallel.Parallel( modems.keys(), ) return p( self.batchSize, self.processModem, modems = modems, parallelObject = p, ) or, more concisely, for when you don't need the Parallel object for display of progress bars or the like: return parallel.Parallel( modems.keys(), )( self.batchSize, self.processModem, modems = modems, parallelObject = p, ) Attributes: dispatched -- count of items dispatched returned -- count of items which have completed allDispatched -- whether all items have been dispatched allReturned -- whether all items have completed finalDF -- the final deferred object we will call iterable -- iterable which produces our items callable -- the callable object we will apply to items from iterable args, named -- extra arguments to callable Note: Despite being broken into initialisation and call steps, you should *not* treat a Parallel object as reentrant, they are *not* able to be restarted reliably. XXX confirm proper operation when iterable is empty... """ dispatched = 0 returned = 0 allDispatched = False allReturned = False finalDF = None def __init__( self, iterable ): """Setup the parallel dispatcher iterable -- produce argument for each call """ self.iterable = iter(iterable) self.live = [] def __call__( self, count, callable, *args, **named ): """Call the parallel dispatcher, dispatching count items in parallel count -- number of "parallel" operations to start callable -- callable function: callable( currentValue, *args, **named ) Must return a Deferred instance args, named -- extra parameters for callable The reason we have callable, args and named specified here is that we often want access to the ParallelDispatcher instance in the arguments to the iterable function (to calculate fraction completed, for instance). returns deferred returning [(success,result) for item in self.iterable] """ self.callable = callable self.args = args self.named = named if self.finalDF is None: self.finalDF = defer.Deferred() # We could raise an error here, but this allows us to later decide to # kick off a few *extra* parallel operations, should we desire that... for x in range( count ): if not self.next(): break return self.finalDF def next( self ): """Iterate to schedule the next item""" try: try: value = self.iterable.next() except (StopIteration,IndexError), err: self.allDispatched = True if self.dispatched == 0: self.allReturned = True if self.finalDF: self.finalDF.callback( self.live ) self.cleanup() return False else: df = self.callable( value, *self.args, **self.named ) index = self.dispatched self.dispatched += 1 self.live.append( None ) if not hasattr( df, 'addCallbacks' ): raise TypeError( """Callable %r (with arguments) (%s,%s) did not return a Deferred (no addCallbacks method): %r"""%( self.callable, ", ".join( [str(x) for x in (value,) + self.args] ), ", ".join( ["%s=%r"%(key,value) for key,value in self.named.items()] ), df, ) ) df.addCallbacks( self.oneFinished, self.oneFailed, callbackKeywords={'index':index}, errbackKeywords={'index':index}, ) return True except Exception, err: self.finalDF.errback( failure.Failure()) def doNext( self ): """Decide whether to dispatch another request""" if self.allDispatched: if self.returned >= self.dispatched: self.allReturned = True self.finalDF.callback( self.live ) self.cleanup() else: self.next() def oneFailed( self, reason, index ): """Retire one of our items""" self.returned += 1 self.live[index] = (False, reason) self.doNext() def oneFinished( self, result, index ): """Retire one of our items""" self.returned += 1 self.live[index] = (True, result) self.doNext() def cleanup( self ): """Eliminate all of our potentially circular references""" try: del self.callable del self.args del self.named del self.iterable del self.finalDF except AttributeError, err: pass [View Less]

5 7

[Twisted-Python] Twisted on Mac OS X 10.4
by Max May 9, 2005

May 9, 2005

Hi, do you know how to install Twisted on Mac OS X 10.4 Tiger? I need it to deploy Punjab... Thank you Max

10 14

[Twisted-Python] Weekly Bug Summary
by James Y Knight May 9, 2005

May 9, 2005

Bug summary ______________________ Since 2005-05-03 14:10:29.270000: Bugs opened: 16 Bugs closed: 2 Total open bugs: 364 (+14) Urgent: 9 (+2) Bug: 147 (+10) Feature: 138 (+2) Wish: 70 (+0) New / Reopened Bugs ______________________ ==== Bug ==== [TRIAL] trial's default timeout isn't enforced (yesterday) http://twistedmatrix.com/bugs/issue1028 opened by warner [CONCH] Can't logout of an SFTP server (yesterday) http://twistedmatrix.com/bugs/… [View More]

1 0

[Twisted-Python] Q about handling CTRL-C correctly
by Stefan Näwe May 9, 2005

May 9, 2005

Hi. I'm new to twisted. I'm trying to create a simple twisted server that sends some telegrams (created with struct.pack()) every second to the connected clients. Attached is what I came up with so far. My problem is, that I don't know how to handle CTRL-C correctly. If I run my test server and press CTRL-C while a client is connected I get something like: 2005/05/03 09:23 [__main__.SimpleFactory] Received SIGINT, shutting down. and that's it. The server simply keeps running. But I … [View More]

2 2

[Twisted-Python] Call for documentation reviewers
by Mary Gardiner May 8, 2005

May 8, 2005

Hey, They have been some people here pretty keen to have better documentation. One concrete thing you could do right now to help that take place is to review existing documentation for me. Steps: 1. Write a brief description of your general Twisted user profile (how much do you know about Twisted? what have you used it for?) Note I don't care what this profile is exactly, it can range from "novice programmer" to "author of reactor framework." 2. Find an existing howto or … [View More]

1 0

Re: [Twisted-Python] Using Twisted inside mod_python
by Bob Ippolito May 5, 2005

May 5, 2005

On May 5, 2005, at 4:01 PM, jarrod roberson wrote: > > > On 5/5/05, Bob Ippolito <bob(a)redivi.com> wrote: > On May 5, 2005, at 3:34 PM, jarrod roberson wrote: > > > I want to embed a Twisted Server ( specifically a ZeroConf > > implemenation I wrote ) into Apache2. > > > > I have it running from a PythonImport and attaching to an > > interpreter and all that goodness. > > It prints out all the queries and answers it received to the … [View More]

1 0

[Twisted-Python] Using Twisted inside mod_python
by jarrod roberson May 5, 2005

May 5, 2005

I want to embed a Twisted Server ( specifically a ZeroConf implemenation I wrote ) into Apache2. I have it running from a PythonImport and attaching to an interpreter and all that goodness. It prints out all the queries and answers it received to the apache log just fine. Apache works just fine serving normal stuff. The problem is anytime I make any kind of method calls into my global object from a handler it just hangs Apache2 :-( I have tried it with worker.c and prefork.c and both … [View More]

2 1

[Twisted-Python] twistedmatrix.com gets a touch-up
by Alex Levy May 5, 2005

May 5, 2005

Given the recent discussion of Twisted's "professional face", this seems more than a little apropos. twistedmatrix.com has been updated with some new CSS and a little more color. I've tested it on Firefox/Win and IE6/Win, but it needs a lot more scrutiny. Everyone, please throw it into your browser(s) of choice and give a yell if something looks awry. Thanks to Chris Armstrong for giving me permission to run amok in the WebSite branch. -- Alex Levy WWW: http://mesozoic.geecs.org/ "Never … [View More]

5 5

[Twisted-Python] Building Twisted on Windows
by Justin Johnson May 5, 2005

May 5, 2005

Will someone who has experience building the latest Twisted code on Windows using Python2.4 please shed some light on the problems I'm experiencing, documented in http://twistedmatrix.com/bugs/issue1017? Thanks! Justin

1 0

[Twisted-Python] Reactors and New Asynchronous APIs
by Patrick Mézard May 5, 2005

May 5, 2005

Hello, I have several questions about twisted reactors and extending the twisted framework with new asynchronous APIs. 1- What is the status of the IOCP reactor? Is it stable? If it is, is there any reason for not selecting it as the default reactor for win32 NT platforms? 2- I am considering the integration of a custom database API (like Berkeley DB) in a twisted application. I could easily wrap the API in a C++ python module and access it synchronously from twisted, at least in a … [View More]

3 2

[Twisted-Python] Closing Jabber connection
by Carolina Ardohain May 4, 2005

May 4, 2005

I try to launch several jabber connections (BTW I use twisted 1.3). When the login is successful there's no problem, but whe the password is wrong I wanna to disconnect, I call connector.disconnect() but the event client.BasicAuthenticator.AUTH_FAILED_EVENT keep calling again and again. How I should close the connection so that event don't be called anymore? from twisted.protocols import jabber, xmlstream from twisted.protocols.jabber import client, jid from twisted.internet import reactor, … [View More]

11 24

[Twisted-Python] twisted.news
by Manlio Perillo May 4, 2005

May 4, 2005

Regards and thanks for your work on Twisted. I have noted that twisted.news isn't installed with the last version of the installer for Windows. Why? Manlio Perillo

3 2

[Twisted-Python] Issue with stopListening
by Örjan Reinholdsen May 4, 2005

May 4, 2005

All, I have written a Twisted server that will be used as a test server to test the client software ability to gracefully handle server shutdown and thereafter reconnect. For that purpose I need my twisted server to periodically stop listening on ports for a while, and then commence normal operation again. For some reason I don't seem to succeede since the twisted server never seems to stop listening on a port even if I call the stopListening method on the IListeningPort object. Anyone got a … [View More]

2 1

Re: [Twisted-Python] blocking and threads
by Joachim Boomberschloss May 3, 2005

May 3, 2005

Thanks, how embarassing... I suggest adding it to the "using threads" howto (I'd do it myself if I knew how). Another idea: make an asynchronous file-handling module that wraps Python's file IO with something that has a deferred-oriented interface. Is there already something like that too? Joe --- Jp Calderone <exarkun(a)divmod.com> wrote: > On Wed, 27 Apr 2005 05:49:01 -0700 (PDT), Joachim > Boomberschloss <boomberschloss(a)yahoo.com> wrote: > >Hi all, > > > … [View More]

5 5

[Twisted-Python] Weekly Bug Summary
by James Y Knight May 3, 2005

May 3, 2005

Bug summary ______________________ Since 2005-04-25 14:26:00.540000: Bugs opened: 11 Bugs closed: 7 Total open bugs: 350 (+4) Urgent: 7 (+1) Bug: 137 (+2) Feature: 136 (+1) Wish: 70 (+0) New / Reopened Bugs ______________________ ==== Urgent ==== [RELEASE MANAGEMENT] release 2.0.1 (4 days ago) http://twistedmatrix.com/bugs/issue1007 opened by jknight ProtocolWrapper's behavior is unspecified (6 days ago) http://twistedmatrix.com/bugs/… [View More]

1 0

[Twisted-Python] Another win32 installer release
by Cory Dodt May 2, 2005

May 2, 2005

-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1 As usual, they're here. As usual, this is based on OFFICIAL, RELEASED Twisted 2.0.0 but the *installers themselves* are considered RC's. This fixes issue 991. It also fixes issue 996 for Windows. (Issue 996 pertains to other platforms as well.) http://twisted.sf.net/Twisted_NoDocs-2.0.0.win32(installer-rc4)-py2.4.exe http://twisted.sf.net/Twisted_NoDocs-2.0.0.win32(installer-rc4)-py2.3.exe C -----BEGIN PGP SIGNATURE----- Version: GnuPG v1.2.… [View More]

2 1

[Twisted-Python] Newbie reactor.crash()/reactor.stop() question
by T Barket May 2, 2005

May 2, 2005

Hello, I have set up a server and client with perspective broker and can successfully move data back and forth between them. however, i want to use the client interactively at the python prompt. ie, i want to create a client object, make a request of the server and get the data returned to me. then i want to do some work with the data interactively at the python command prompt, until i need new data. then i want to use the client again to make another request of the server, etc. i have … [View More]

3 2

[Twisted-Python] dbus + twisted
by Anthony Baxter May 2, 2005

May 2, 2005

So it turns out that the python dbus bindings just use glib - this means that if you're using gtk2reactor, you can use dbus and it all Just Works. This is very cool. It would be nice if there was a way to say (in dbus) "Give me any pending event" in a non-blocking way -- we could then write something similar to the current wxsupport module and use dbus from any reactor. I'm not sure that this would be all that easy, as dbus supplies bindings to other event loops rather than to the raw … [View More]

1 2