Mailman 3 December 2011 - Twisted

Re: [Twisted-Python] How to Solve a Problem Like Santa with Stackless and Twisted
by Andrew Francis Dec. 14, 2011

Dec. 14, 2011

Hi Glyph: Message: 3 Date: Tue, 13 Dec 2011 23:45:54 -0500 From: Glyph <glyph(a)twistedmatrix.com> Subject: Re: [Twisted-Python] How to Solve a Problem Like Santa with Stackless and Twisted To: Twisted general discussion <twisted-python(a)twistedmatrix.com> Message-ID: <F1901420-DAD9-4879-8E65-CAFC123BACA4(a)twistedmatrix.com> Content-Type: text/plain; charset="iso-8859-1" On Dec 13, 2011, at 2:04 PM, Andrew Francis wrote: Hi Glyph: G>The only reason that the Santa problem, expressed as a thread-synchronization problem, has to deal with G>"simultaneous" arrival of 9 reindeer and 3 elves is a quirk of thread scheduling: the Santa thread can be ready to run G>but not running. When the final reindeer arrives, a final elf may arrive before the scheduler decides it's time to give G>Santa a chance to run, and then Santa has the opportunity to make the choice between the two. I agree: it is the scheduler that introduces quirks. Not that it matters but the important edge case is when three elves have arrived. Santa is awoken and placed on the queue. However additional reindeer have arrived bringing the count to nine. So when Santa runs, both the group of nine reindeer and group of three elves are ready. Hence the need for Santa to check when he is awake. In reality, I have to leave the programme running for a long time to see this happen. G>In an event-driven system, this state doesn't naturally exist: when you call Deferred.callback(), the callback runs the G>relevant application code immediately. In regards to Stackless, if I wanted to greatly simplify the problem, I would immediately schedule Santa once reindeer or a group of elves were ready. This is what Twisted is doing. Perhaps in a real-life case, this would be the correct action since it is obvious that Santa is a high priority task. However I am trying to be true to the original problem. Also a good solution should not depend on scheduling tricks. Otherwise Twisted with deferred lists would otherwise be pretty effective. Again the secret sauce is that deferred lists are a join pattern of sorts. The component is that Twisted is essentially a non-preemptive first come first served scheduler. This behaviour also simplifies the problem. G>Despite the fundamentally sequential nature of reality, Maybe this is the reality of computers based on von Neuman architecture. However reality is fundamentally concurrent. G>Twisted reactor's scheduling mechanism does leave somethings to be desired; the quantum time-slice of Twisted is G>effectively one reactor iteration, and there aren't really any tools for working with those. There's some latency G>between becoming aware of some work that needs to be done (select(), or your chosen multiplexor, returning) and G>actually doing the work (calling doRead, doWrite, or your timed event), but you don't get to decide on priorities once G>Twisted's aware of the work; they're executed in a quasi-arbitrary order. I feel a Twisted callback and a Stackless tasklet in non-preemptive mode suffer from a bigger problem: one does not know how long the code fragment will execute. The programmer is betting that the CPU slice will be relatively short. An additional assumption is that all tasklets/callbacks are created equally (actually in Stackless, one can use preferences to give some tasklets a limited priority). I feel the reason we have sophisticated schedulers is when throughput and fairness are a big deal. G>Ultimately however, the order of the work will be arbitrary; if it's not influenced by the vagaries of Twisted's scheduler, G>it will be influenced by something in the kernel scheduler that you don't understand or can't control, or in the IP stack, G>or in a switch on your network, or in your hosting provider's firewall configuration, or something on a client machine G>that you truly have no influence over at all. Having something give priority to an event that occurs at "the same" G>time given all of these potential sources of interference is basically pointless. I am not an expect in real-time systems. However I would argue that at a pragmatic level, timestamps have to be of a resolution that allows the application to make reasonable decisions. So if the application is measuring time in seconds to say two decimal places, as far as the application is concerned, it can have a reindeer and elf arriving at the same time if they have the same timestamps. Perhaps my original example leaves something to be desired. However my point still remains. If we constructed a schedule so that the ninth reindeer always arrived at T+11 seconds and three elves always arrived at T+9, T+10, and T+11 seconds, it would be a coin toss in Twisted as to whether reindeer or elves were serviced. Rather than a reindeer being in a queue, in Twisted, its file descriptor would be in a reader set. All I can say is that a Twisted solution would be unbiased (or biased towards elves if there we are playing with 10 of them). When the problem requires it is biased towards reindeer. If one had a burning desire, we could construct cases to measure if there was a bias. Cheers, Andrew

1 0

[Twisted-Python] How to Solve a Problem Like Santa with Stackless and Twisted
by Andrew Francis Dec. 14, 2011

Dec. 14, 2011

Hi Folks: I don't know what to file this under but here goes: Santa repeatedly sleeps until wakened by either all of his nine reindeer, back from their holidays, or by a group of three of his ten elves. If awakened by the reindeer, he harnesses each of them to his sleigh, delivers toys with them and finally unharnesses them (allowing them to go off on holiday). If awakened by a group of elves, he shows each of the group into his study, consults with them on toy R&D and finally shows them each out (allowing them to go back to work). Santa should give priority to the reindeer in the case that there is both a group of elves and a group of reindeer waiting I recently modified the stackless.py library to support yet another new concurrency feature: join patterns. Well I think of my version as a sawed-off version of join patterns. Join patterns are a synchronization and message passing construct. The closest thing to a join pattern in Twisted is a deferred list. Stackless has nothing of the sort. The Santa Claus Problem is a notorious problem in concurrency (http://www.youtube.com/watch?v=pqO6tKN2lc4). I have come up with a relatively short solution. I think one would be hard pressed to come up with a shorter solution in Python. My desire was to compete with Simon Peyton Jones Haskell version in "Beautiful Code." The full solution is here: http://andrewfr.wordpress.com/2011/11/30/the-santa-claus-problem-with-join-…. The nucleus is: def santa(reindeer, elves): joinObject = stackless.join().addPattern([ch for _, ch, _ in reindeer]).\ addPattern([ch for _, ch, _ in elves],3) reindeerPattern, elfPattern = joinObject.patterns while True: pattern = joinObject.join() if reindeerPattern.ready(): print "*** REINDEER GET PRIORITY ***" reindeerPattern.join() pattern = reindeerPattern if pattern is reindeerPattern: harness(reindeerPattern) deliveryToys(reindeerPattern) unharness(reindeerPattern) elif pattern is elfPattern: consultWithSanta(elfPattern) stopTwisted() In this solution I use Twisted to control the timer (I use the blockOn technique that Christopher Armstrong first used). def tick(seconds): tickCh = stackless.channel() reactor.callLater(seconds, tickCh.send, None) tickCh.receive() However I could just as easily make reindeer and elves web services or RestFull interfaces. I am still working on the Join Pattern API. Also it has also been a goal of mine to write a stacklessreactor to better integrate Stackless with the Twisted Reactor. Cheers, Andrew

4 4

[Twisted-Python] Autobahn WebSockets 0.4.10
by Tobias Oberstein Dec. 14, 2011

Dec. 14, 2011

Hi, Autobahn WebSockets 0.4.10 is done: + various improvements to RPC / PubSub + RFC 6455 support + kqueue support on BSD + various smaller issues fixed + demos updated, docs improved Special thanks to Peter Thorson for RFC6455 changes and test cases. Cheers, Tobias Project Link: http://www.tavendo.de/autobahn Package: http://pypi.python.org/pypi/autobahn Docs: http://www.tavendo.de/autobahn/doc/python/index.html Source: https://github.com/oberstet/Autobahn

2 1

Re: [Twisted-Python] Twisted-Python Digest, Vol 93, Issue 12
by Scott M Ferguson Dec. 12, 2011

Dec. 12, 2011

Hi all, A little intro first, I'm a developer by day, sql/plsql/java/perl/c/c++/... I've been looking for an opensource project to get to know and contribute to and came across twisted a week or so ago. It looks pretty interesting, years ago I spent time writing networking/remote access code. Anyway, I want to help and am starting to look through the 'easy' bugs - Python is pretty new to me so I figure that's where I should be. I took a look at http://twistedmatrix.com/trac/ticket/4966, which might be out-of-date. I don't see __all__ in twisted.web.server.py. Am I missing something there? Also, does the email verification on the website work? I've tried it a few times and haven't received an email, also nothing in my spam/trash related to it - sorry if this shouldn't go to the list. Thanks, Scott Ferguson On Sat, Dec 10, 2011 at 12:00 PM, <twisted-python-request(a)twistedmatrix.com> wrote: > Send Twisted-Python mailing list submissions to > twisted-python(a)twistedmatrix.com > > To subscribe or unsubscribe via the World Wide Web, visit > http://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python > or, via email, send a message with subject or body 'help' to > twisted-python-request(a)twistedmatrix.com > > You can reach the person managing the list at > twisted-python-owner(a)twistedmatrix.com > > When replying, please edit your Subject line so it is more specific > than "Re: Contents of Twisted-Python digest..." > > > Today's Topics: > > 1. Re: Boston Twisted Sprint: Last-Minute December Edition (Tenth) > 2. Re: Boston Twisted Sprint: Last-Minute December Edition > (Kevin Horn) > 3. MemoryError in twisted.internet.abstract.FileDescriptor > (Augusto Mecking Caringi) > 4. Re: MemoryError in twisted.internet.abstract.FileDescriptor > (Itamar Turner-Trauring) > 5. Re: Boston Twisted Sprint: Last-Minute December Edition (Glyph) > > > ---------------------------------------------------------------------- > > Message: 1 > Date: Fri, 9 Dec 2011 13:32:23 -0500 > From: Tenth <tenth(a)twistedmatrix.com> > Subject: Re: [Twisted-Python] Boston Twisted Sprint: Last-Minute > December Edition > To: Twisted general discussion <twisted-python(a)twistedmatrix.com> > Message-ID: > <CAKJpRM0NWgs47FvsodcPKEBpX_D8fbnSGXQ+OSQp1+WiXWPknQ(a)mail.gmail.com> > Content-Type: text/plain; charset="iso-8859-1" > > I should also mention that the Sprint will now officially be starting at > Noon, if that makes it more convenient for anyone... > > Thanks, > > - Dave >

2 3

[Twisted-Python] Weekly Bug Summary
by exarkun＠twistedmatrix.com Dec. 11, 2011

Dec. 11, 2011

1 0

[Twisted-Python] Boston Twisted Sprint: Last-Minute December Edition
by Tenth Dec. 10, 2011

Dec. 10, 2011

The good news is that it's December, and we're hosting another Boston Twisted Sprint! The bad news is that it's going to be this Saturday, December 10th, and I'm only just announcing it on the list now, due to last minute venue and schedule change shenanigans. This will be the last sprint that Exarkun can join us for in Boston, though, so it's also your last chance to visit and bother him to review your tickets in person! As with the last few sprints, we'll be at the Smarterer.com offices in South End Boston, from about 2:00 PM to 10:00 PM; If you're interested, please contact me to be added to the official invitation (which includes more specific info about how to get there, and gain access to the building, which is locked on weekends). Thanks, - Dave

3 4

Re: [Twisted-Python] Log output formatting (was Re: logging question)
by Tom Sheffler Dec. 5, 2011

Dec. 5, 2011

I am happy with the information I get out of the logging system and the formatting of errors and tracebacks. We've recently come across two things that could be handled nicer however. 1) we use multiple logfiles. one for *all* messages, and another for *special* messages. It would be nice if the concept of multiple logfiles and the routing of messages to files was part of the system. 2) the above, coupled with the use of twisd and starting the logfiles was a little weird. Our main logfile is attached to the application like this application.setComponent(ILogObserver, FileLogObserver(logfile).emit) and the other logfile just starts on its own. I had expected to attach both of them to the application and set up an event filter for each. It's a little non-symmetric the way it is. -T

2 1

[Twisted-Python] Moving Twisted off Trac and SVN to somewhere nicer
by Laurens Van Houtven Dec. 5, 2011

Dec. 5, 2011

Hi, As some of you may already know (either through a backchannel or because you talked to me at Europython), there has been some talk about moving Twisted way from Trac+SVN to somewhere that isn't Trac+SVN. A lot of the devs do like SVN. My guess is that that's mainly because they don't actually use SVN, they use Combinator, or something. On the other hand, I do think that Trac is pretty universally loathed, and it would be a good idea to get away from it. There's a few existing hosting solutions: 1. Launchpad (+ Bazaar as the default vcs) 2. Bitbucket (+ Mercurial as the default vcs) 3. Github (+ Git as the default vcs) Although I've hated git for a long while (and I still don't like it very much), I firmly believe Github is the right thing for Twisted. My incredibly unscientific poll amongst people who like Twisted but aren't devs is that they all love or at least like Github, and a surprising number has a distaste for Launchpad (unfamiliarity with Bazaar, perceived developer-unfriendly UI, slowness). Unless someone is going to go all "NO GITHUB IS TERRIBLE AND YOU ARE A BAD PERSON FOR EVEN SUGGESTING IT" on me, maybe we can talk about planning the transition? :) -- cheers lvh

32 97

Re: [Twisted-Python] Telnet negotiation with Twisted
by Lee Orsino Dec. 4, 2011

Dec. 4, 2011

Hi Patrick! Thanks for the suggestion. After I call self.will(LINEMODE), I get Failure: twisted.conch.telnet.OptionRefused: twisted.conch.telnet.OptionRefused: '"' So I know at least something is happening. I guess this just became a Telnet question instead of a twisted question. But I still don't understand how enableRemote and requestNegotiation are supposed to be used with this. Anyway, this happens in PuTTY as well as the Windows telnet client. Below is the code, if it helps, as simple as I could get it. Maybe there is something obvious that I need to do: from twisted.internet.protocol import Protocol, Factory from twisted.conch.telnet import TelnetTransport, TelnetProtocol, ITelnetProtocol from twisted.internet import reactor import sys class EchoTransport(TelnetTransport): def __init__(self, protocolFactory=None, *a, **kw): TelnetTransport.__init__(self) if protocolFactory is not None: self.protocolFactory = protocolFactory self.protocolArgs = a self.protocolKwArgs = kw def connectionMade(self): TelnetTransport.connectionMade(self) print ("connection made") def connectionLost(self, reason): print ("connection lost") reactor.stop() def applicationDataReceived(self, bytes): TelnetTransport.applicationDataReceived(self, bytes) bytes = bytes.strip() if (bytes == 'b'): self.will(chr(34)) # Error here if (bytes == 'x'): self.transport.write("goodbye") TelnetTransport.loseConnection(self) def main(): f = Factory() f.protocol = lambda: EchoTransport(TelnetProtocol) reactor.listenTCP(8023, f) reactor.run() if __name__ == '__main__': main()

5 13

[Twisted-Python] Weekly Bug Summary
by exarkun＠twistedmatrix.com Dec. 4, 2011

Dec. 4, 2011

1 0