Mailman 3 January 2011 - Twisted

[Twisted-Python] HTTPClient handling LF only servers
by Jason J. W. Williams Jan. 26, 2011

Jan. 26, 2011

Hi Y'all, Certain servers (notably Hacker News) break HTTPClient because they use LF instead of CRLF to terminate header lines. I've uploaded a patch with tests to ticket 4814 (merged into ticket 2842). I'd appreciate a review and some discussion about integrating it. -J Sent via iPhone Is your e-mail Premiere?

5 9

[Twisted-Python] Names DNS Client too chatty
by Jason J. W. Williams Jan. 25, 2011

Jan. 25, 2011

Hi Guys, I'd like to get ticket 4021 integrated in some fashion into Twisted. Currently, the Twisted Names client issues a "starting" and "stopping" log message on every resolution. It fills up the logs and makes seeing valuable log messages really difficult in the sea of DNS requests. 150,000 resolutions per day is generating 700mb of logs just from the start/stop. The problem is actually the UDP datagram transport that Resolver is based on. 4021 proposes to address the issue by wrapping the two log lines with "if self.noisy". I'd advocate this solution as it leverages the noisy attribute already expected to tamp down log chattiness. Also, it requires the smallest amount of code change as opposed to implementing a filter on the logging system. Honestly this has been a problem for me on more than one project I've used the Names client in. It's just untenable in a project with a high volume. Jean-Paul asked me to bring this up for discussion if I wanted to fix it. What do others think? -J Sent via iPhone Is your e-mail Premiere?

7 16

[Twisted-Python] Porting twisted methods to a synchronous environment
by Don Dwiggins Jan. 25, 2011

Jan. 25, 2011

I have a server that provides access to a database in the usual way, using adbapi to talk to an MSSQL server. Now, I'm facing the need to create an application that can access the server directly. (In addition to, not instead of.) In the spirit of "write it once", I'd like to be able to "import" the server's database read/write methods into a module in the application, possibly driven by a reactor of some sort. In this case, there'd be no client/server protocol, just calls from other modules in the application. I realize that this is pretty vague; I'm still in the process of conceptualizing it. I'm looking for any advice that might help avoid the all-too-obvious approach of copying code and performing surgery on it, with the all-too-well-known drawbacks. Thanks for any good words, -- Don Dwiggins Advanced Publishing Technology

2 2

[Twisted-Python] Weekly Bug Summary
by exarkun＠twistedmatrix.com Jan. 24, 2011

Jan. 24, 2011

5 4

[Twisted-Python] Refactoring Documentation
by Tom Davis Jan. 24, 2011

Jan. 24, 2011

I've been using (and threatening to work on) Twisted for a few years now. It seems like every time I get back into it, I need to dig up old code or Google queries just to get started. Yesterday, Jean-Paul introduced me to the trial-tutorial branch<http://twistedmatrix.com/trac/browser/branches/trial-tutorial-2443-2> which has shed some light on the basics of using trial and testing Twisted client/server applications in general. Until he mentioned it in IRC, I was stuck looking at the actual tests for protocols and deciding which parts of that were generically useful to me. I agreed to finish up that documentation so that it could finally (four years later) be added to trunk (and more importantly, twistedmatrix.com). But after thinking about it, I believe the problem runs much deeper than just the lack of a branch merge. Reading code to find answers isn't rocket science; I've been programming long enough to be comfortable doing it. But I probably have to resort to reading Twisted's code about 8 billion percent more often than any other codebase. And reading code is a hurdle. Reading through Twisted's semi-random, 45-point FAQ is a hurdle—and recommending it as a starting point is unhelpful at best. The core documentation isn't awesome either, given that it has a tendency to be overly cryptic and link to API documentation that is often incomplete or generally unhelpful. As one very basic example, see: http://twistedmatrix.com/documents/current/core/howto/servers.html. Let's just review a few things wrong with this page: - It's tutorial page #1, but basically tells me I need to read howto/plugins.html first if I am writing an "application" (whatever that is), as opposed to a "TCP, SSL, and Unix socket server". And it's the wrong place for UDP. - It attempts to introduce Protocols and Factories—two of Twisted's most important concepts—and does neither particularly well. I know that Protocols (usually) inherit from t.i.p.Protocol and may be instantiated for a variety of reasons and aren't (usually) persistent. I also know that Factories instantiate Protocols and give a reference to themselves so protocols can "access and modify the persistent configuration". I am told I need to implement some interface (or something) to actually listen on a host/port. I think. - At one point "TCP4ServerEndpoint" is instantiated (but never imported); its explanation is left to a digression into the endpoints API, which has its own issues. Suffice it to say the document doesn't give me sufficient reason to actually use the endpoints API. - Later on, we just use reactor.listenTCP()—which our previous digression (if bothered to click through and read) claims is not preferable. By the end of the *servers* tutorial (and after reading some linked documentation), here's all I *really* know: 1. Factories create protocols somehow 2. Protocols have methods like connectionMade, connectionLost, and dataReceived to handle events 3. There are other protocols with other methods. One that I know of, anyway. 4. I may need to write a state machine (???) 5. I should use an Endpoint or maybe a Service or reactor (but probably not!) 6. I should also use Application for serious business Moving forward, howto/clients.html duplicates a lot of these things and fills in some gaps in knowledge while creating more holes. Meanwhile, I still never wanted to create a QOTD or Echo server. I think the point has been made. My *real* point, though, is that I love Twisted. And I'm constantly wishing it was more accessible to newcomers. Twisted is Python's oldest and most mature event-based networking engine and despite its decade of existence it remains largely confusing and obscure to the majority of Python programmers who come upon it. It contains concepts and standards that are alien to the average Python programmer, but they make a lot of sense and have a lot more consistency and predictability than the documentation conveys. I want to fix that, among other things. And as luck may have it, I like writing documentation. And I know at least enough Twisted to get the high-level stuff in order and improve the documentation to the point that people will keep reading long enough to make sense of the "idea of Twisted" and be able to implement some basic things and expand upon them later. There are some things I *do* want to accomplish early on: - Make the docs accessible (a lot is hidden and hard to find) - Make them more concise and useful to somebody who doesn't want to know the 50 different ways to skin every cat (including the ones you should never use) - Make them introduce and explain Twisted in a way that somebody as dumb as me can understand it. This means talking about protocols, factories, deferreds, etc. in a way that doesn't require thousands of words of circular explanations, digressions, and duplications. - Document the different Twisted projects as what they are: really useful libraries and/or clients/servers built on top of Twisted that just so happens to also ship with the core. One of them is likely to do what the prospective user wants, too. There are also some things I *don't* want to accomplish, at least initially: - Explain asynchronous programming in depth - Get into the level of detail that the Krondo series does (but I plan to borrow from it!) There are things I'll *need help with*: - The final word on what is/isn't to be used in what context; practical, real-world explanations of why I would use A instead of B (not just "it's more flexible"). - Better documentation from project creators/maintainers/lovers on their project. These projects make a lot of hard stuff trivial and many do not convey that very well. I'll do my best to reorganize and improve the existing documentation, but most of the projects cater to a specific niche that requires its own core knowledge. Most of this was discussed with Jean-Paul and Glyph at tonight's Python meet-up in Cambridge. Some work has already begun on the new docs here: https://github.com/tdavis/twisted-docs I hope the above came off as optimistic and constructive as I meant it: it's not my intention to insult the documentation or those who wrote it. I just think with a dose of hindsight and a lot of effort, the docs can be awesome. And they would improve the entire Twisted ecosystem along with them. Cheers, Tom

9 30

[Twisted-Python] incomplete reviews
by Glyph Lefkowitz Jan. 23, 2011

Jan. 23, 2011

Some reviewers (you know who you are!) have gotten into the habit of doing incomplete reviews, that note one or two trivial things about a patch or branch, but don't say whether the code is ready to land after fixing just those trivial things. I've updated the review process documentation to address this, so every reviewer, please have a look: <http://twistedmatrix.com/trac/wiki/ReviewProcess?action=diff&version=44>

1 0

[Twisted-Python] Flags with MaildirMailbox
by goffi Jan. 20, 2011

Jan. 20, 2011

G'day everybody, I'm playing a little bit with twisted.mail.maildir.Mailbox which seems to work nice, but it doesn't seem to manage flags (cf http://cr.yp.to/proto/maildir.html "What can I put in info"). Is there an easy/elegant way to do this ? Is it planed to implement this feature ? Thanks for any answer Regards Goffi

3 5

[Twisted-Python] Question about SerialPort
by Albert Brandl Jan. 20, 2011

Jan. 20, 2011

Hi! I am trying to read Modbus RTU messages over the serial line. There is a library called pyModbus which is very usable for Modbus TCP/IP, but the support for RTU is a bit lacking. The main problem seems to be the specification of Modbus RTU. RTU messages are separated by time gaps that are at least 3.5 characters long. Unfortunately, this is the only reliable way for separating messages - there is no length header or something similar. The specification does define a trailing CRC, though. As far as I understand the event loop, I will not be able to guarantee that will not miss such a gap - the reactor might be busy handling some other events while the gap arrives. But when the reactor assigns some time to read from the serial port, I'd like it to read until the gap is detected. My current attempt is to override the doRead method of SerialPort by something like this: def doRead(self): port = self.fileno() result = [] while True: # The following code is stolen from fdesc.readFromFD try: output = os.read(port, 1) if output: result.append(output) else: break except (OSError, IOError), ioe: if ioe.args[0] in (errno.EAGAIN, errno.EINTR): return else: return CONNECTION_LOST if not result: return CONNECTION_DONE else: self.protocol.rawDataReceived("".join(result)) If the serial port is configured with a timeout about the size of the gap, os.read() should return an empty string, thus indicating that the message is complete. Most of the code above is a copy of t.i.fdesc.readFromFD. The main difference is that I don't read 8192 bytes, but only one at a time, to detect a time gap. Does this make sense? Do you think it would be possible to replace the constant 8192 in readFromFD by a parameter (that defaults to 8192)? This would allow me to replace the content of the while loop with a simple function call. TIA and best regards, Albert -- Albert Brandl Weiermayer Solutions GmbH | Abteistraße 12, A-4813 Altmünster phone: +43 (0) 720 70 30 14 | fax: +43 (0) 7612 20 3 56 web: http://www.weiermayer.com

3 3

[Twisted-Python] (conch) credentials.signature always None?
by Saúl Ibarra Corretgé Jan. 20, 2011

Jan. 20, 2011

Hi all, I'm experiencing something weird while writing a simple SSH server, and I wonder if anyone did run into this before: On every snippet I see around regarding public key authentication, the signature is checked as follows: # if not credentials.signature: # return failure.Failure(ValidPublicKey()) # try: # public_key = keys.Key.fromString(data=credentials.blob) # except (keys.BadKeyError, keys.EncryptedKeyError): # return failure.Failure(ConchError("Public key error")) However, I'm always getting None there and thus the authentication cannot proceed. I printed its value in the sshsimpleserver.py from the examples directory and same seems to be happening, so I guess it's not only my script :-S Any clue? Thanks in advance! -- /Saúl http://saghul.net | http://sipdoc.net

3 5

Re: [Twisted-Python] Boston Area Mini-Sprint
by Tenth Jan. 18, 2011

Jan. 18, 2011

The Mini-Sprint was pretty successful... The seven attendees (and three remote contributors) managed to close 12 tickets, and start work on a several more. The Official Report is here: http://labs.twistedmatrix.com/2011/01/january-twisted-sprint-report.html Hopefully we'll do this again soon! If you weren't on the invitation list for the January sprint but would like to hear about our plans for February, let me know. (I'll post again once we figure out a date.) - Dave

1 0