Mailman 3 July 2013 - Twisted

[Twisted-Python] defer.gatherResults & Deferreds chaining
by Jorge Gonzalez 01 Aug '13

01 Aug '13

I'm trying to understand why a chain of Deferreds I'm dealing with hangs and why return values from some callbacks are not always making it to other callbacks down the chain of Deferreds. Is it possible that there could be a race condition between Deferred chaining and Deferred result gathering<http://twistedmatrix.com/documents/current/core/howto/defer.html> ? I.e., Chaining Deferreds<http://twistedmatrix.com/documents/current/core/howto/defer.html#auto13> says that "*If you need one Deferred to wait on another, all you need to do is return a Deferred from a method added to addCallbacks. Specifically, if you return Deferred B from a method added to Deferred A using A.addCallbacks, Deferred A's processing chain will stop until Deferred B's .callback() method is called; at that point, the next callback in A will be passed the result of the last callback in Deferred B's processing chain at the time*." Now at the same time, DeferredList<http://twistedmatrix.com/documents/current/core/howto/defer.html#auto8> (on top of which gatherResults is built) warns that "*If you want to apply callbacks to the individual Deferreds that go into the DeferredList, you should be careful about when those callbacks are added. The act of adding a Deferred to a DeferredList inserts a callback into that Deferred ... The important thing to remember is that it is this callback which records the value that goes into the result list handed to the DeferredList's callback.* *Therefore, if you add a callback to the Deferred after adding the Deferred to the DeferredList, the value returned by that callback will not be given to the DeferredList's callback. To avoid confusion, we recommend not adding callbacks to a Deferred once it has been used in a DeferredList*" Given these two explanations, say I have something like this: --------------------------------------------------------------------------- def slowFuncReturningDeferred_A(): ... def slowFuncReturningDeferred_B(): ... d = defer.Deferred() d.addCallback(labmda _: slowFuncReturningDeferred_A()) d.addCallback(labmda _: slowFuncReturningDeferred_B()) defer.gatherResults([d]) --------------------------------------------------------------------------- Wouldn't it be possible that defer.gatherResults inserts its callback into d before say slowFuncReturningDeferred_B gets to run and return its own Deferred (and therefore chain it to what gatherRestults is ultimately waiting for?). If so, wouldn't the results returned by slowFuncReturningDeferred_B never make it to the results gathered by defer.gatherResults? Thanks, Jorge

2 4

[Twisted-Python] ugh open-ssl and twisted python
by gary clark 28 Jul '13

28 Jul '13

Hello, This problem has been driving me crazy for a couple of days. I'm hoping someone can shed some light on this. I have a twisted server its using open-ssl (using certificates) and runs on linux and communicates great with a windows client thats send messages every second. However when I send data back to the client "sometimes" it doesnt receive the packet and there is no indication of failure at the client whats more I dont see any failure on the twisted server end. Is there any way I can tell what could cause this failure? It just look like the transport.write(data) fails just dont know why it would when receiving works like a charm. Any help on this would relieve the pain I'm feeling right now. Much Appreciated, Gazza

4 6

[Twisted-Python] Trial & the mock library
by James Broadhead 27 Jul '13

27 Jul '13

Hey all - I've recently started working with the 'mock' library in our trial tests, and am looking for some best-practice advice. I'm really just starting to get used to the library, so it might well have functionality that I'm unaware of or am misusing. I very quickly ran into a problem where I mistakenly returned a Mock() in the place of a deferred, causing the asserts in callbacks to not be called, and for the test to spuriously pass. A basic example: def test_foo(): d = Mock() def check_result(res): self.assertEqual(res.code, expected) # never called d.addCallback(check_result) return d # Mock is truthy, test passes This occurred where I was mocking some internals of the class under test; something like the below A slightly more believable example: == myclass.py == def some_function(...): d = self.authenticate() d.addCallback(foo) # foo never called d.addErrback(bar) # bar never called return d == test_myclass.py == def setUp(self): self.resource.authenticate = Mock(return_value=Mock()) def test_foo(): d = self.resource.some_function def check_result(res): # never called self.assertEqual(res.code, expected) d.addCallback(check_result) return d # test passes Currently, I'm experimenting with wrapping Mock instantiations by defining common deferred methods on them in advance; this approach would eventually lead to extending Mock itself with this functionality. def nonDeferredMock(): m = Mock() def notimpl(*args, **kwargs): raise NotImplementedError('You treated a Mock like a Deferred!') m.addCallback = notimpl m.addErrback = notimpl m.addBoth = notimpl m.addCallbacks = notimpl return m Another approach might be extending TestCase to check that return values are always not Mock objects. Does anyone on the list have experience with this? Obviously, this only happens when mistakes are made when writing tests, but I'd rather have confidence that when my tests pass, that they've passed for the right reasons. Another antipattern that I've come across has been: resource.mymethod = Mock(return_value=defer.succeed(None)) which works fine for tests in which mymethod() is called once, but always returns the same deferred object if multiple calls are made. What would be a better approach? Cheers- James

7 13

[Twisted-Python] Multi-service, multi-protocol examples
by Justin Chiu 26 Jul '13

26 Jul '13

Hi all, A couple weeks ago I posted a question to this list titled "AMQP listening and user-facing daemon". The responses were very helpful; thank you. Does anyone have any examples/tutorials on writing with multiple protocols doing totally different things? I found a thread on Stackoverflow [1] and the Twisted Finger tutorial [2]. Anything along these lines -- possibly more in-depth? I have also been working thorugh the recommended Krondo tutorial [3] which has been helpful in learning the mechanisms of Twisted. [1] http://stackoverflow.com/questions/4084090/multiple-reactors-main-loops-in-… [2] https://twistedmatrix.com/documents/current/core/howto/tutorial/index.html [3] http://krondo.com/blog/?p=1247 Thanks, Justin

2 1

[Twisted-Python] Don't Finalize Connection
by Thomas Hommers 23 Jul '13

23 Jul '13

Hi, i am fairly new to twisted and try to work my way trough the documentation. There is a problem I can't find a solution for. Maybe somebody can give me a hint or point me into the right direction. I am trying to create a TCP server. The client connects to the server, then the server will send multiple commands and binary data to the client. After the data has been send, the connection should stay open. I subclass "Protocol" to create my protocol. It seems that after i send data with self.transport.write(data) the last packet will be send with the "FP." flag, what seems to "Finalize" the connection from this end. Is there a way to force twisted to just "Push" my data, but not to "Finalize" the connection? Thanks. Thomas

3 3

[Twisted-Python] Best way to trigger a future connection with data
by Nick Johnson 22 Jul '13

22 Jul '13

Hi, I'm porting some code to twisted and got a little stuck. The current (non-twisted) code connects to a server, gives it some data (a job to do), gets a job-id and then terminates the connection. Sometime later, it reconnects and gets the output using the job-id. I've a factory and protocol set up, with buildProtocol over-riden in the Factory class to supply some extra data to each Protocol instance when it's created. This is fine when I submit a job as I can call a factory function to generate the required data (ie the job to do). My problem is how to initiate a connection at some point in the future AND pass it some specific info (ie the job-id) for it to pass to the server. I start the connections using task.callingLoop and reactor.connectTCP so my first guess was to add: task.callLater(time, reactor.connectTCP, ...) to the connectionLost method of my Protocol. However, I cannot see how to pass in any extra information (ie the job-id) this way? I know I can store state in the Factory, so I guess I just need to know how to pass this to a scheduled invocation of the Protocol. Is anyone able to point me in the right direction here? Cheers, -Nick. -- Nick Johnson, Applications Developer, EPCC 2407 JCMB, King's Buildings, Mayfield Road, Edinburgh, EH9 3JF e: Nick.Johnson(a)ed.ac.uk t: 0131 651 3388 The University of Edinburgh is a charitable body, registered in Scotland, with registration number SC005336.

2 8

[Twisted-Python] Setting socket options before connect
by Phil Mayers 21 Jul '13

21 Jul '13

I have a use case (namely setting the Linux-specific SO_MARK socket option) where I need to set the socket option after the socket object is created but before Twisted starts to try and connect. Is there a clean way to do this? Ideally there would be some sort of pre-connect function, similar to the preexec_fn for child processes, but any reasonably clean solution will do. (By clean I mean "doesn't have me duplicating hundreds of lines of Twisted code into my own Connector/Port/etc. classes" ;o)

5 12

[Twisted-Python] July Report
by Tom Prince 19 Jul '13

19 Jul '13

http://labs.twistedmatrix.com/2013/07/july-report.html This will be my last report, until the beginning of September. I have, for many years, been a volunteer at the Edmonton Folk Music Festival[1]. For the next month, I will be busy setting it up (and then taking it down, and so will not have time to devote to twisted. This month, 23 tickets got some attention that they would not have received without the sponsored development. The result was 5 closed tickets and 11 other tickets unblocked for other developers to resume work on. This month, I made a number of improvements to the buildobt: - Switched most of the buildslaves to use git to checkout the source. - All the git-using builders merge to trunk, before running tests, so the tests results reflect the code as it would be merged. In particular, all the lint steps should now accurately report only the regressions relative to current trunk of the merged result. - I added a bunch of new builders on VMs provided by Rackspace[1] (thanks to Jesse Noller[3]) and the Fedora Project[4] (thanks to Seth Vidal[5]). These cover all recent versions of Debian, Fedora, Ubuntu and RHEL. As part of this, I automated the deployment of buildslaves, (currently only Debian and Fedora derivatives are supported). This work is made possible by the sponsorship of individuals and organizations which have donated to the Twisted project, part of the Software Freedom Conservancy, a not-for-profit organization that helps promote, improve, and develop open source software. Thanks! The tickets I reviewed, that have now been closed: - #5574: Add support for SOCK_DGRAM to IReactorSocket - #6221: Replace Deprecated Test Case Assertion Methods in twisted.internet - #6393: Utility function to pretty-format list of things - #6475: FileAuthority._lookup incorrectly returns AuthoritativeDomainError for non-subdomain names which happen to share the same suffix as the zone (eg the-example.com and example.com) - #6572: twisted.mail.smtp.sendmail should return a cancellable Deferred Other tickets I have reviewed: - #2699: UDP DatagramProtocol reconnecting - #3926: twisted.positioning -- a better positioning framework - #4173: WebSocket server support - #4735: Implement something like ReconnectingClientFactory for endpoints - #5001: replace call to functions from the string module in twisted.scripts.tap2deb - #5004: replace call to functions from the string module in web - #5685: SMTP Client Tutorial should be self-contained - #6556: LineReceiver rejects lines of length MAX_LENGTH sometimes - #6557: LineOnlyReceiver doesn't disconnect the transport when it detects an overlarge line - #6558: LineOnlyReceiver doesn't pass the entire contents of its buffer to lineLengthExceeded() - #6580: twisted.names.server.DNSServerFactory should query child zones before parent zones Tickets I have worked on: - #5513: Detailed documentation on Proxy by adding docstrings to many of the methods. - #6537: Update coding-standard to reflect that documentation should be sentence per line. - #6620: Document conventions for writting test assertions. - #6626: Remove all uses of deferredGenerator. [1]: http://www.edmontonfolkfest.org/ [2]: https://www.rackspace.com/ [3]: https://twitter.com/jessenoller/status/355453772803211264 [4]: https://fedoraproject.org/ [5]: https://www.redhat.com/about/news/archive/2013/7/thank-you-seth-vidal

1 0

[Twisted-Python] SQL ORM for Twisted & PostgreSQL
by Matt Haggard 19 Jul '13

19 Jul '13

> On Jul 17, 2013, at 5:04 PM, Jeffrey Ollie <jeff at ocjtech.us> wrote: > > > I know that this has been asked before, but it's been a while and I'm > > hoping for some good news. Is there a SQL ORM that works well with > > Twisted and PostgreSQL? In particular I'm hoping to find something > > that works with txpostgres as that's the library I prefer to access > > the database with. > > There's bits of one in Calendar Server: <http://trac.calendarserver.org/browser/CalendarServer/trunk/twext/enterpris…> It doesn't use txpostgres, but that's because it relies on adbapi2 (also in calendar server, hopefully to be moved into Twisted at some point). > > -glyph I've made a thing that has a similar API to txpostgres and will use txpostgres underneath if it's installed: https://github.com/iffy/norm I welcome feedback/patches. - Matt Haggard (I wasn't on the mailing list when the original message was sent, so this "reply" may not appear quite correctly)

1 0

[Twisted-Python] Testing AMP-based code
by Free Ekanayaka 18 Jul '13

18 Jul '13

Hi, following up from ticket #6502, I'm looking for recommendations/best practices for writing unit-tests for AMP-based code. As described in the ticket, the issue I'm currently facing is that the AMP implementation is subtly not re-entrant safe and doesn't work with a synchronous transport, for example this code raises an exception: http://twistedmatrix.com/trac/attachment/ticket/6502/example.py I'm starting to think that the most appropriate testing strategy would be to mock/stub AMP.callRemote (or the protocol class altogether) instead of trying to use a fake transport. Thoughts? Cheers, Free

4 6