Mailman 3 January 2013 - Twisted

[Twisted-Python] Twisted-related talks and tutorials at PyCon
by Itamar Turner-Trauring 28 Jan '13

28 Jan '13

Hi, Thought I'd point out some Twisted-related talks at PyCon; I may have missed a few. *Tutorials* Jessica "jesstess" McKellar is teaching two tutorials, one of which teaches the basics of contributing to open source projects, in case you've wanted to contribute to Twisted but haven't yet figured out the process: https://us.pycon.org/2013/schedule/presentation/4/. I will be teaching a tutorial on test-driven development with Twisted, explaining how to build robust, well-tested applications: https://us.pycon.org/2013/schedule/presentation/15/ *Talks* * * Jessica McKellar will be giving a talk on "How the Internet Works", which will be including some demonstrations using Twisted: https://us.pycon.org/2013/schedule/presentation/50/. Ashwini "ashfall" Oruganti, one of our Google Summer of Code interns, will be talking about endpoints, Deferreds and more: https://us.pycon.org/2013/schedule/presentation/40/ Glyph Lefkowitz will be rudely-yet-jokingly insulting Javascript programmers in his talk on event-driven programming: https://us.pycon.org/2013/schedule/presentation/120/ Armin Rigo will be talking about Software Transactional Memory and removing the GIL from PyPy (we will hopefully have a multi-core reactor soon after this becomes usable): https://us.pycon.org/2013/schedule/presentation/37/ Allen "dash" Short will be talking about Parsley, his parsing library: https://us.pycon.org/2013/schedule/presentation/84/ Laurens "lvh" Van Houtven will be teaching Crypto 101: https://us.pycon.org/2013/schedule/presentation/74/ Brad Kuhn, of the Software Freedom Conservancy (Twisted's non-profit sponsor), will be talking about funding open source projects, using among others Twisted as an example: https://us.pycon.org/2013/schedule/presentation/48/ *Other* * * Jessica McKellar will be giving one of the keynote addresses, though it probably has nothing to do with Twisted. (The posters list isn't up yet, but I'm betting Jessica will also have a poster.) * * I believe that as with every year we will be having a development sprint after the main conference, and a birds-of-feather session at some point. -Itamar

2 2

Re: [Twisted-Python] inlineCallbacks: multiple yielding same Defferred behaviour
by Sergey V. Gerasimov 28 Jan '13

28 Jan '13

My target is just to deny request duplications. Reg. inlineCallbacks.. What do you think is it possible to implement inlineCallbacks generator in the way allows multiple yields? (but do not modify Defferred) From: Laurens Van Houtven [mailto:_@lvh.cc] Sent: Monday, January 28, 2013 1:43 PM To: sergun(a)gmail.com Subject: Re: [Twisted-Python] inlineCallbacks: multiple yielding same Defferred behaviour The short answer is that those models don't support chaining the way Deferreds do, IIUC. I think a very similar thing was a bonus question at the tutorial Stephen Thorne and I gave at EuroPython last year. Do you want to cache results as well, or only not make duplicate requests when one is in-flight? On Mon, Jan 28, 2013 at 10:36 AM, Sergey V. Gerasimov <sergun(a)gmail.com> wrote: Hi Laurens, Thanks for your reply! I need something like this. A server that processes requests and handles situation when there are 2 requests with same key. class Server(object): def __init__(self): self.requestsInProgress = {} # dict: key -> Deferred with result @defer.inlineCallbacks def request(self, key): in_progress = key in self.requests if in_progress: d = self.requestsInProgress [key] result = yield d else: d = process_request(key) self.requests[key] = d result = yield d del self.requestsInProgress [key] defer.returnValue(result) How can I implement the logic correctly? In my opinion it's very simple and clear concept to consider Deferred just like deferred execution of some action. In this case you should be able to get it's result (doing yield) many times (why not?). I have deal with Task-based programming and async functions (mechanism similar to Deferred and inlineCallbacks decorated functions in Twisted) in Microsoft .NET. MS people have implemented Task (analogue of Deferred) in the way I described. From: Laurens Van Houtven [mailto:_@lvh.cc] Sent: Monday, January 28, 2013 12:47 PM To: sergun(a)gmail.com; Twisted general discussion Subject: Re: [Twisted-Python] inlineCallbacks: multiple yielding same Defferred behaviour I'm not entirely sure why you feel this is unintuitive or why you should need a workaround; in any other case, deferreds work pretty much the same way. When reading this code, the first question that pops up in my head is: why are you doing this instead of just using the result that you already have (and assigned to a name)? How did this waiting-twice-on-one-deferred happen? On Mon, Jan 28, 2013 at 8:47 AM, Sergey V. Gerasimov <sergun(a)gmail.com> wrote: It's not intuitively clear why multiple yielding the same Defferred in function decorated with defer.inlineCallbacks get different results: @defer.inlineCallbacks def f(): d = defer.succeed(1) result = yield d print result result = yield d print result Output: 1 None Probably we SHOULD NOT think if "d" was or was not yielded before when using inlineCallbacks.. Are they some workarounds? _______________________________________________ Twisted-Python mailing list Twisted-Python(a)twistedmatrix.com http://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python -- cheers lvh -- cheers lvh

2 1

[Twisted-Python] inlineCallbacks: multiple yielding same Defferred behaviour
by Sergey V. Gerasimov 28 Jan '13

28 Jan '13

It's not intuitively clear why multiple yielding the same Defferred in function decorated with defer.inlineCallbacks get different results: @defer.inlineCallbacks def f(): d = defer.succeed(1) result = yield d print result result = yield d print result Output: 1 None Probably we SHOULD NOT think if "d" was or was not yielded before when using inlineCallbacks.. Are they some workarounds?

2 2

[Twisted-Python] Weekly Bug Summary
by exarkun＠twistedmatrix.com 26 Jan '13

26 Jan '13

1 0

[Twisted-Python] Amazing exception "GeneratorExit"
by Аркадий Левин 25 Jan '13

25 Jan '13

Hi all, i got strange error If run this: # -*- coding: UTF-8 -*- import os import sys from twisted.python import log from twisted.internet import reactor from twisted.internet.defer import Deferred, inlineCallbacks @inlineCallbacks def _(): try: (yield Deferred()) except: log.err() reactor.callLater(0, _) reactor.run() i got exception "GeneratorExit" Why?! ... this code work: # -*- coding: UTF-8 -*- import os import sys from twisted.python import log from twisted.internet import reactor from twisted.internet.defer import Deferred, inlineCallbacks @inlineCallbacks def _(): deferred = Deferred() try: (yield deferred) except: log.err() reactor.callLater(0, _) reactor.run() Change "Deferred()" to "deferred = Deferred()"

2 3

Re: [Twisted-Python] Stopping Reactor in UnitTest Callback
by Fabian Rothfuchs 25 Jan '13

25 Jan '13

Hey again! Having the setup doing quite well so far, I just ran into a problem that I do not know how to debug: Testcase `setUp()` starts a server that subclasses `twisted.web.resource.Resource` , as it's going to be a lightweight SOAP server, responding with hard-coded xml strings. Purpose is to check the request for validity. The Testcase's `test_*()` function initiates a client request by using the suds module. I now discovered that the server process started in `setUp()` hangs as long as the suds connection did not finish. Which is a deadlock, as it cannot finish unless the server sent the response. When trying to connect to the server via telnet, I do not get any response unless the suds connection is lost. Snippets: TestCase: http://dpaste.org/pmg6Y/ TestServer: http://dpaste.org/NOLb8/ Client: http://dpaste.org/2wxIV/ Thanks for any help! Fabian From: Fabian Rothfuchs <fabian.rothfuchs(a)googmail.com> Date: Thursday, January 17, 2013 11:27 AM To: Twisted general discussion <twisted-python(a)twistedmatrix.com> Subject: Re: [Twisted-Python] Stopping Reactor in UnitTest Callback While trying to minimalize my code, I discovered that the following line somewhere down deep the code is causing the whole test to hang: > adbapi.ConnectionPool('MySQLdb', system.host, system.username, > system.password, system.database) > > It seems that the connection is not being closed properly, causing the whole test chain to hang. From: Itamar Turner-Trauring <itamar(a)futurefoundries.com> Reply-To: Twisted general discussion <twisted-python(a)twistedmatrix.com> Date: Tuesday, January 15, 2013 6:22 PM To: Twisted general discussion <twisted-python(a)twistedmatrix.com> Subject: Re: [Twisted-Python] Stopping Reactor in UnitTest Callback On Tue, Jan 15, 2013 at 10:26 AM, Fabian Rothfuchs <fabian.rothfuchs(a)googlemail.com> wrote: > Yes, I thought so, too. Indeed it happens that the reactor is started > automatically, but after the test succeeds, it's not being stopped (only > chance is to kill 9 it). Perhaps the problem isn't with the test framework, but with your test. Can you show us a minimal piece of code that reproduces the problem? _______________________________________________ Twisted-Python mailing list Twisted-Python@twistedmatrix.comhttp://twistedmatrix.com/cgi-bin/mailman/lis tinfo/twisted-python

2 2

Re: [Twisted-Python] Twisted Names strings
by exarkun＠twistedmatrix.com 22 Jan '13

22 Jan '13

On 03:37 pm, angelo.dellaera(a)gmail.com wrote: > >Hi, >I didn't have the possibility to test the fix during the last days >(hopefully will do >in the next hours) but taking a look at the fix proposed by ralphm > >http://twistedmatrix.com/trac/changeset/36867 > >it seems like the patch is not complete because a similar check (and >potential >conversion) should be inserted in the twisted.names.dns.Name __init__ >method >too. Am I wrong? Hi Angelo, Thanks for taking a look at the changeset, at least! I think that you are correct. I would even go so far as to say that the change should be reverted, as it does not actually fix the *regression*, it works around it in a higher-level API. As you say, this results in an incomplete fix for the issue since many applications may (and likely are) using `Name` directly rather than via SRVConnector. I hope Ralph or Tom will chime in with their thoughts on this. Jean-Paul

1 0

[Twisted-Python] Twisted client memory leak
by Diego Woitasen 22 Jan '13

22 Jan '13

I have an Twisted client app that makes hundreds of connections per minute. I discover that I have a memory leak un my app and I'm almost sure that is related to the ClientFactory() derived class that is never deleted. I reproduce the problem with a modification of Echo client example from Twisted documentation: from twisted.internet.protocol import Protocol, ClientFactory from twisted.internet import reactor from twisted.internet.task import LoopingCall from sys import stdout class Echo(Protocol): def connectionMade(self): print 'MADE' self.transport.write('XXXX') def dataReceived(self, data): print 'RECV', data self.transport.loseConnection() def __del__(self): print 'DEL PROTOCOL' class EchoClientFactory(ClientFactory): def startedConnecting(self, connector): print 'Started to connect.' def buildProtocol(self, addr): print 'Connected.' return Echo() def clientConnectionLost(self, connector, reason): print 'Lost connection. Reason:', reason def clientConnectionFailed(self, connector, reason): print 'Connection failed. Reason:', reason def __del__(self): print 'DEL FACTORY' def connector(): print 'CONNECTOR' factory = EchoClientFactory() reactor.connectTCP('localhost', 7, factory) #reactor.callLater(2, connector) register_loop = LoopingCall(connector) register_loop.start(1) reactor.run() With this code I discover that the instances of EchoClientFactory() are only deleted when the program shutdowns. They are not deleted when the connections finish. I haven't found in the documentation if I need to do some to get factory instances deleted. -- Diego Woitasen

6 14

[Twisted-Python] Twisted Names strings
by Angelo Dell'Aera 22 Jan '13

22 Jan '13

After updating Twisted to 12.3.0 I started getting errors like the following one 2013-01-17 14:38:23+0100 [-] File "/usr/lib64/python2.7/site-packages/twisted/names/common.py", line 93, in lookupAddress 2013-01-17 14:38:23+0100 [-] return self._lookup(name, dns.IN, dns.A, timeout) 2013-01-17 14:38:23+0100 [-] File "/usr/lib64/python2.7/site-packages/twisted/names/client.py", line 384, in _lookup 2013-01-17 14:38:23+0100 [-] d = self.queryUDP([dns.Query(name, type, cls)], timeout) 2013-01-17 14:38:23+0100 [-] File "/usr/lib64/python2.7/site-packages/twisted/names/dns.py", line 484, in __init__ 2013-01-17 14:38:23+0100 [-] self.name = Name(name) 2013-01-17 14:38:23+0100 [-] File"/usr/lib64/python2.7/site-packages/twisted/names/dns.py", line 357, in __init__ 2013-01-17 14:38:23+0100 [-] raise TypeError("%r is not a byte string" % (name,)) 2013-01-17 14:38:23+0100 [-] TypeError: u'example.com' is not a byte string Taking a look at documentation I found this document http://twistedmatrix.com/documents/current/core/howto/python3.html which states "twisted.names.dns deals with strings with a wide range of meanings, often several for each DNS record type. Most of these strings have remained as byte strings, which will probably require application updates (for the reason given in the FilePath section above). Some strings have changed to text strings, though. Any string representing a human readable address (for example, Record_A's address parameter) is now a text string. Additionally, time-to-live (ttl) values given as strings must now be given as text strings." which seems to describe the problem I'm experiencing. But let me say that I'm currently using Python 2.7.3 and I was surprised in realizing that a simple update from a minor 12.x release was enough to disrupt my application. Don't you think that a kind of compatibility mechanism (maybe with a warning message) would have been a more gentle way of doing this transition especially for those who are still using Python 2? Thanks and regards, -- Angelo Dell'Aera 'buffer' Antifork Research, Inc. http://buffer.antifork.org Sysenter Honeynet Project http://www.sysenter-honeynet.org

6 11

[Twisted-Python] Weekly Bug Summary
by exarkun＠twistedmatrix.com 19 Jan '13

19 Jan '13

1 0