Mailman 3 May 2014 - Twisted

[Twisted-Python] ANN: Eliot: a framework for Logging as Storytelling
by Itamar Turner-Trauring May 16, 2014

May 16, 2014

Eliot provides a structured logging and tracing system for Python that generates log messages describing a forest of nested actions. Actions start and eventually finish, successfully or not. Log messages thus tell a story: what happened and what caused it. Here's what your logs might look like before using Eliot: Going to validate http://example.com/index.html [1]. Started download attempted. Download succeeded! Missing <title> element in "/html/body". Bad HTML entity in "/html/body/p[2]". 2 validation errors found! After switching to Eliot you'll get a tree of messages with both message contents and causal relationships encoded in a structured format: * {"action_type": "validate_page", "action_status": "started", "url": "http://example.com/index.html" [1]} * {"action_type": "download", "action_status": "started"} * {"action_type": "download", "action_status": "succeeded"} * {"action_type": "validate_html", "action_status": "started"} * {"message_type": "validation_error", "error_type": "missing_title", "xpath": "/html/head"} * {"message_type": "validation_error", "error_type": "bad_entity", "xpath": "/html/body/p[2]"} * {"action_type": "validate_html", "action_status": "failed", "exception": "validator.ValidationFailed"} * {"action_type": "validate_page", "action_status": "failed", "exception": "validator.ValidationFailed"} Features: * Structured, typed log messages. * Ability to log actions, not just point-in-time information: log messages become a trace of program execution. * Excellent support for unit testing your logging code. * Emphasis on performance, including no blocking I/O in logging code path. * Optional Twisted support. * Designed for JSON output, usable by Logstash/Elasticsearch. * Supports CPython 2.7, 3.3 and PyPy. Eliot is released by HybridCluster [2] under the Apache 2.0 License. To install: $ pip install eliot Downloads are available on PyPI [3]. Documentation can be found on Read The Docs [4]. Bugs and feature requests should be filed at the project Github page [5]. Links: ------ [1] http://example.com/index.html [2] https://hybridcluster.github.io [3] https://pypi.python.org/pypi/eliot [4] https://eliot.readthedocs.org/ [5] https://github.com/hybridcluster/eliot

1 0

[Twisted-Python] TCP and ICMP
by Jay Shah May 13, 2014

May 13, 2014

Hiee All, I have very simple question. I have already implemented custom reactor for handling ICMP requests (reference: txnettools) . My problem is: i am implementing tcp traceroute. I have handled all icmp time exceed messages. But, i want to handle tcp syn-ack reply which i am getting from remote server. How can i do that ? Can i start listening to TCP syn req or replies on the same reactor on which i am listening to icmp replies ? or should i install new custom reactor for listening to tcp replies ? I want to parse SYN ACK request ONLY. Neither I want to make a new connection with remote server nor handling them! I just need syn ack replies to be parsed so that i can figure out whether i have reached my destination or not. thank you. Jay Shah (Technische Universität München, München, Deutschland Tel. +49 176 7266 7259 )

6 5

[Twisted-Python] Twisted 14.0.0 Release Announcement
by HawkOwl May 12, 2014

May 12, 2014

On behalf of Twisted Matrix Laboratories, I am honoured to announce the release of Twisted 14.0! It has been a long road to get here, but we’ve done it! The highlights of this release are: - Twisted Positioning (`twisted.positioning`) makes its entry into Twisted! It comes ready to talk with common GPS devices, and will supersede `twisted.protocols.gps`. - A wealth of SSL/TLS improvements, including ECDHE support, TLS Service Identity (with service_identity on PyPI), a stronger default set of ciphers, and strengthening against attacks such as CRIME. A Twisted Web server with pyOpenSSL 0.14 is capable of getting an A in Qualys SSL Labs tests out of the box, and A+ with small application modifications. Twisted Agent can also now do HTTPS hostname verification. - Python 3 improvements, including the ability for `pip install` to install all ported modules. - Twisted Pair’s TUN/TAP support has been overhauled, with documentation and full test coverage. - Significant documentation improvements, including more API documentation for Twisted Mail & Twisted Names, narrative documentation for Twisted Names, and a migration to Sphinx for building Twisted narrative docs. - Support is dropped for pyOpenSSL older than 0.10 and Windows XP. You can find the downloads at <https://pypi.python.org/pypi/Twisted> (or alternatively <http://twistedmatrix.com/trac/wiki/Downloads>) . Many thanks to everyone who had a part in this release - we’ve got some big things landed, and if it weren’t for the support of developers (both core and occasional), the Twisted Software Foundation, or people giving feedback and filing bugs, we’d have never got it done. Twisted Regards, HawkOwl

3 2

[Twisted-Python] Distributing apilinks.py
by Adi Roiban May 11, 2014

May 11, 2014

Hi, Right now apilinks.py is located in Twisted at docs/_extensions/apilinks.py, but a copy can also be found in pydoctor project. While reporting a bug for apilinks.py in Twisted I was ask to send the patch upstream in pydoctor [1] During the review of the patch in pydoctor, pydoctor's maintainer agreed that there should be a single copy of this file but don't know what's involved in achieving that. [2] Can someone involved in using apilinks.py in Twisted provide some feedback regarding distribution of apilinks.py file? Many thanks! [1] https://twistedmatrix.com/trac/ticket/6954 [2] https://code.launchpad.net/~adiroiban/pydoctor/1282458-apilinks-anchor/+mer… -- Adi Roiban

3 5

[Twisted-Python] Weekly Bug Summary
by trac＠twistedmatrix.com May 11, 2014

May 11, 2014

1 0

[Twisted-Python] The Twisted 14.0 Release Pre-Post-Mortem, and Where To From Here
by HawkOwl May 7, 2014

May 7, 2014

Hi everyone, I’m sure that some of you have been following the past seven or so weeks of Twisted 14.0 release shenanigans, and this email hopes to explain what went wrong, what we can do better next time, and where we can go from here. Problem 1: Twisted 14.0.0pre1 had a regression. This was not noticed in the prerelease stage because it was not marked as a regression, where the RM does a check for open regressions on the milestone. What we can do better next time: Tickets that are regressions need to be marked as regressions and applied to the release milestone. If you think it might be a regression - even slightly - mark it as such, and comment that you are not sure. It’s easier to find the ticket later and decide it is not actually a regression than have to abort a release because it’s come up after a prerelease. Problem 2: The fix for the regression was not merged into pre1, the release was rerolled from trunk. This meant some pyOpenSSL and TLS improvements got into the 14.0 release from pre2 onwards, but introduced new regressions. What we can do better next time: Do not reroll from trunk to get bug fixes - merge them into the release branch. Problem 3: The fixes for the regressions were finished after some delay, since the fixes had to be written and reviewed. This introduced delays into the 14.0 release cycle. What we can do better next time: Rather than fix regressions introduced, the ticket that introduced them should be reverted. Problem 4: The fixes for the regressions did not merge cleanly with the release branch. Some 35+ tickets were merged between pre1 and the release of the regression fix into trunk. What we can do better next time: Bug fixes should be based off the release branch, not trunk. This reduces the likelihood of code churn or unknown dependencies causing problems during the merge. Problem 5: There was mixed communication whether one of the regression fixes was to be introduced in 14.0 or in a bug fix release (14.0.1). What we can do better: If a fix is intended for merging in to a prerelease, it should be raised on the mailing list, so that there is more visibility for its intentions. Problem 6: I personally made several mistakes along the way - from screwing up svn merges to interpreting the “abort the release and incorporate the bugfix” to apply the initial regression fix. Since the TLS changes were topical, I decided that having them out ASAP would be better than not. What we can do better: Improved docs/automation to reduce the margin for RM error, and better automation to make a new release to get out important features really easy. These are the major problems which I have identified - I’m sure there’s plenty more, and I would like people to list them if I have not - even if they make me look like an idiot ;). We can learn from it, I’m sure. So, this leaves where to from now. I see a few options, with my estimates for work and risk that it’ll explode: 1 - Most work, high risk - Work on making the regression fixes merge cleanly with 14.0.0pre5. This is big-ish task with room for error, since there was some underlying code churn. 2 - Some work, medium risk - Release 14.0.0pre5 as 14.0 final, and I (or another RM if I’m no longer trusted ;) ) initiate the 14.1 release immediately. 3 - Least work, highish risk - Scrap 14.0, begin the 14.1 release immediately. since 14.0 tags become 14.1 tags, and we just have to hope that there’s no regressions in the 39 tickets fixed between pre1 and now. This may introduce issues (since 14.0 is an un-release, and there are questions about what this does to our deprecation windows). If I am to be honest, I much prefer option #3, but I would like opinions from other developers, before I go causing more problems than I already have :) Regards, HawkOwl

5 6

[Twisted-Python] qt4reactor on PySide (Was: Serial interface for software program)
by anatoly techtonik May 7, 2014

May 7, 2014

On Mon, May 5, 2014 at 10:06 PM, <exarkun(a)twistedmatrix.com> wrote: > On 06:49 pm, sank.daniel(a)gmail.com wrote: >>> >>> There's a 3rd party event loop for Twisted that runs on top of Qt >>> (qt4reactor), which would >>> allow Twisted and Qt code to run in the same thread. >> >> Speaking of which, why isn't that thing officially part of twisted? Super >> useful. > > The PyQt team insists that the PyQt license is incompatible with Twisted's > license (this is an intentional simplification, you can read more about the > issue on PyQt's website if you really care). > > Since PySide now exists this is potentially no longer an issue. However, the > last thing I heard about using PySide to run the Qt reactor is that it > segfaulted sometimes. > > So perhaps the reason is that someone needs to do some work to determine if > PySide is actually safe to use with the Qt reactor now and, if it is, > shuffle some code around. I think Twisted users just need to start using it, have this doc at hand - https://code.google.com/p/spyderlib/wiki/HowToDebugQtCrash - and register on https://bugreports.qt-project.org/ if there is a trouble (or just mail PySide list). > However, I'm not even sure if the GUI reactors *should* be distributed as > part of the main Twisted package. They are already first-class reactors via > the plugin system. Independent releases let them track changes in the GUI > library they're each concerned with more quickly than they could if they > were part of Twisted itself. > > Perhaps there are other reasons, like simplicity of packaging or quality of > continuous integration testing, that would override those issues and make it > worth bringing Qt reactor back into Twisted. Sorting that out is also a > task that requires someone interested to step up an do some work, though. :) The question was - why super useful Qt reactor loop is not a part of Twisted. I think that if it is really useful, it should, -- anatoly t.

2 1

[Twisted-Python] ReactorBuilder doesn't support external reactors ticket reminder (Was: Serial interface for software program)
by anatoly techtonik May 7, 2014

May 7, 2014

On Tue, May 6, 2014 at 4:02 AM, Itamar Turner-Trauring <itamar(a)itamarst.org> wrote: > On 05/05/2014 03:06 PM, exarkun(a)twistedmatrix.com wrote: >> >> However, I'm not even sure if the GUI reactors *should* be distributed as >> part of the main Twisted package. They are already first-class reactors via >> the plugin system. Independent releases let them track changes in the GUI >> library they're each concerned with more quickly than they could if they >> were part of Twisted itself. > > > Someone remind me to file a "ReactorBuilder doesn't support external > reactors" ticket tomorrow... File a "ReactorBuilder doesn't support external reactors" today. ) -- anatoly t.

1 0

[Twisted-Python] Serial interface for software program
by anatoly techtonik May 7, 2014

May 7, 2014

Hello, network hackers, Recently I've got a Raspberry Pi and a friend showed me its serial interface and helped to setup it. I was stunned. What I saw - you open serial terminal, and you jump in into the middle of boot section. Just open a terminal and you see what's going on inside of R.Pi. Any time. That's was an intro. =) For a long time I wanted the same interface for debugging software programs. When I was hacking on Spyder I found that my human brain is too limited to squeeze details of dynamic of interaction of objects at run-time. Spyder is written in Python with Qt, and it is a parallel application much like any Twisted app is. But I wanted to push the progress, I have to deal with complexity, so I badly needed to be able to draw a video of interaction at run-time. That was the problem I tried to solve. And for the first step I needed a way to connect to Spyder at run-time and see what's going on. I didn't know how to call that way before, but now I have a rather accurate wording - "a serial interface for an application". That is the background for the question I am trying to ask. What is the best way to implement such interface considering the following properties: 1. it needs to be simple (so that you can quickly create python script that reads the info) 2. it needs to be universal (so that script for one app worked for other) 3. it should be asynchronous (connected terminal should not degrade performance) 4. it needs to be reliable (at least detecting missing packets) 5. and cross-platform (and pure python) Why Twisted? For the first I think that this problem is actual for people who are dealing with complexity of dynamic and interconnected systems built with the help of Twisted. For the second I believe it can only be solved or analyzed by people skilled in interprocess and internetwork communications. One button test: 1. program behaves weird 2. user hits the button 3. terminal pops up 4. shows a lot of data for a running program I am interested to know. 1. how the button should discover the program network port seems cool, but what about two parallel programs? what about different programs that provide this interface? 2. how terminal receives the data i see it as line based - one line - one event, everything is text 3. how to make it fast TCP clearly won't here 4. how to send the data what if two threads (or parallel code) produce serial data? who should run the server that serves connected terminals, and how other thread will send message to this server? you can not run two TCP servers on the same port on the app side, so is there an alternative? how to detect missing data in a stream? how to avoid dealing with incomplete packets and low level network details? Thanks. I'd like to get back to hacking on Spyder one day, but this is a stumbling block for me every time I think about it or any program of compared complexity. -- anatoly t.

6 8

[Twisted-Python] overly specific ImportError assert for python 3?
by Glyph May 5, 2014

May 5, 2014

cyli was working on <https://twistedmatrix.com/trac/ticket/7144> and encountered this peculiar failure on her branch (and not on trunk): <https://buildbot.twistedmatrix.com/builders/python-3.3-tests/builds/2212/st…>. The interesting bit of the failure is that in one case, the ImportError's message is "import of 'service_identity' halted; None in sys.modules" and on trunk it's apparently 'No module named service_identity'. The branch in question changes conch, which isn't even tested on python 3. So... anyone have an idea what's going on? -g

3 3