Mailman 3 November 2013 - Twisted

[Twisted-Python] carbon-relay eating CPU - EAGAIN (Resource temporarily unavailable)?
by Quan Tong Anh 26 Nov '13

26 Nov '13

- Python 2.7.3 - [twisted, version 13.1.0] - xen-domU `atop` shows that `carbon-relay` is eating 80, 90% USRCPU. From the `strace`: accept(7, {sa_family=AF_INET, sin_port=htons(60649), sin_addr=inet_addr("192.237.222.81")}, [16]) = 257 accept(7, {sa_family=AF_INET, sin_port=htons(51564), sin_addr=inet_addr("166.78.1.48")}, [16]) = 257 accept(7, 0x7ffff4679550, [16]) = -1 EAGAIN (Resource temporarily unavailable) accept(7, {sa_family=AF_INET, sin_port=htons(33654), sin_addr=inet_addr("198.61.194.248")}, [16]) = 257 accept(7, {sa_family=AF_INET, sin_port=htons(50037), sin_addr=inet_addr("166.78.181.204")}, [16]) = 257 accept(7, 0x7ffff4679550, [16]) = -1 EAGAIN (Resource temporarily unavailable) The strange thing is: even restart the service, it seems stuck at fd 7 everytime running `strace`. Does it mean this fd is not being cleanup properly? I have increased the number of open files: **/proc/2891/limits** Limit Soft Limit Hard Limit Units Max cpu time unlimited unlimited seconds Max file size unlimited unlimited bytes Max data size unlimited unlimited bytes Max stack size 8388608 unlimited bytes Max core file size 0 unlimited bytes Max resident set unlimited unlimited bytes Max processes 15834 15834 processes Max open files 16384 16384 files Max locked memory 65536 65536 bytes Max address space unlimited unlimited bytes Max file locks unlimited unlimited locks Max pending signals 15834 15834 signals Max msgqueue size 819200 819200 bytes Max nice priority 0 0 Max realtime priority 0 0 Max realtime timeout unlimited unlimited us then it decreases down to ~ 50%. My problem looks like similar to this [thread]( http://twistedmatrix.com/pipermail/twisted-python/2008-September/018361.html) but since we have a few sockets in TIME_WAIT state, I don't think that enable the `tw_recycle` can help. About the `tcp_syncookies`, I don't see any related message in the syslog. This is what I get when trying to start `carbon-relay` in debug mode: 26/11/2013 02:22:14 :: [listener] MetricPickleReceiver connection with 50.56.249.127:48772 lost: Connection to the other side was lost in a non-clean fashion: Connection lost. 26/11/2013 02:22:14 :: [listener] MetricPickleReceiver connection with 198.101.241.101:50672 lost: Connection to the other side was lost in a non-clean fashion: Connection lost. 26/11/2013 02:22:14 :: [listener] MetricPickleReceiver connection with 166.78.2.167:43346 lost: Connection to the other side was lost in a non-clean fashion: Connection lost. This is from `twisted`: class ConnectionLost(ConnectionClosed): """Connection to the other side was lost in a non-clean fashion""" def __str__(self): s = self.__doc__.strip().splitlines()[0] if self.args: s = '%s: %s' % (s, ' '.join(self.args)) s = '%s.' % s return s I also have tried to [debug with `gdb`]( https://wiki.python.org/moin/DebuggingWithGdb) but `pystack` returns nothing.

3 3

[Twisted-Python] Bristol (England!) sprint hosted by HybridCluster
by exarkun＠twistedmatrix.com 25 Nov '13

25 Nov '13

Greetings all, I'm pleased to invite you to join Tom Prince (2013 Twisted Fellow), Luke Marsden (founder of HybridCluster), and the HybridCluster team (including myself!) for the first Twisted sprint hosted by HybridCluster! The sprint will take place on December 7th in central Bristol (exact location to be determined) from 11AM until no one remains capable of higher brain function(*). As usual sprint topics will surely be wide ranging - bring your favorite pet project, a personal loathing for unresolved bugs, or just find someone interesting to work with when you show up (personally, I'll probably be trying to add APIs to support more easily testing Agent- using applications). HybridCluster is a small Bristol-based company building high density web hosting platform using Twisted for all manner of tasks - from SSH, HTTP, SMTP, POP3, and IMAP4 to internal cluster coordination and asynchronous SQL access. Watch this space for a follow-up post giving details about the venue (and free food!). In the meantime feel free to contact me or post to the list with any other questions. Jean-Paul (*) Due to excessive hacking. Certainly not because of any kind of mind control technology deployed by the TSF.

2 3

[Twisted-Python] [twisted.mail][newbie] return tempfail (4xx) after eomReceived()
by Fabio Sangiovanni 25 Nov '13

25 Nov '13

Hello list, I'm trying to code an SMTP server that stores received messages in a queue (es. RabbitMQ or Redis). I'm figuring out how to do it with Twisted, and so far I managed to print to console the messages the way I want. I'm trying to understand now how to return a tempfail code (4xx) from within eomReceived(), so to simulate (and, in the future, handle) temporary unavailabilities of the queuing system, and thus to delegate to the client the handling/retrying of the delivery (there will be postfix in front of my smtp server, so it will keep the message in its spool). I ended up digging in the code until I found the _messageHandled callback of the SMTP class, that's added to the deferred returned by eomReceived. I admit I'm not very accustomed to deferreds yet, but I can only see 250 and 550 response code inside it. I suspect that I should return the 4xx code in one of the deferred's callbacks/errbacks *before* _messageHandled, but I really can't imagine how. Would you please give me some hints about how to accomplish the task? Am I looking to the right direction at all? Thanks a lot, Fabio

2 3

[Twisted-Python] How to get ServerOptions in tac file.
by Jonas Brunsgaard 25 Nov '13

25 Nov '13

Is there an easy way get an options object from the .tac file? I my .tac file i have the following. fh = logging.FileHandler('/var/log/ocmg/datastore.log') I would like to be able et replace '/var/log/ocmg/datastore.log' with the logfile option given by the user at the command line. Any suggestions?

2 3

[Twisted-Python] txThings - simple library for CoAP protocol
by Maciej Wasilak 25 Nov '13

25 Nov '13

Hello everyone, I would like to announce the first release of txThings. It is a simple library for Constrained Application Protocol (CoAP). CoAP is a relatively new protocol designed for "Internet of Things" and M2M communications. txThings currently implements a subset of CoAP features: - standard messaging based on draft-ietf-core-coap-13 (no proxying support yet) - blockwise transfers based on draft-ietf-core-block-12 - limited suport for RFC6690 Core Link Format (for servers only) I hope to add new features like Observe (draft-ietf-core-observe-09) soon. TxThings is my first Twisted project, so the code quality can probably be improved, also better unit testing is necessary. However I decided to release the code in its current form to discover, if there is any interest in it. Internet of Things is currently very interesting field of development. It is approaching a phase where many opportunities for both large and small players. I think Twisted is well suited for development of IoT apps: - it's stable - it's Python based and portable - it has good support for UDP and TCP, which makes it ideal for proxying IoT protocols to HTTP In my opinion in the next three years, at least 20% of Twisted apps will be IoT related (educated guess, no hard data :) ) There are many new protocols out there designed for IoT, in my opinion CoAP and MQTT have the biggest chance of adoption. CoAP is developed by IETF, and backed by companies like ARM. You can download txThings from Github: https://github.com/siskin/txThings Description: http://sixpinetrees.blogspot.com/2013/09/txthings-good-enough-is-good-enoug… Few more remarks: - txThings works on RaspberryPi (tested) - txThings plays well with Kivy Python GUI Library http://kivy.org Kivy is new GUI library that supports Twisted, and allows building apps for Android and iOS - I've ran a few tests, and it's very promising :) Photo: http://bit.ly/1aPZ1ag Best Regards Maciej Wasilak

5 10

[Twisted-Python] How to debug the client-server pyqt project using twisted framework?
by yangyouxiu 25 Nov '13

25 Nov '13

Hi, all! I'm developing an client-server pyqt application. Is there an easier way to debug? Now, I can only run client and server, then click the button on the client and watch responce of the server? What's the method and what's the tool? Thanks all.

2 4

[Twisted-Python] Weekly Bug Summary
by trac＠twistedmatrix.com 24 Nov '13

24 Nov '13

1 0

[Twisted-Python] Some thoughts hacking with AMP (was: something about multiplexing and accessing protocols)
by Laurens Van Houtven 23 Nov '13

23 Nov '13

Hi, Some thoughts while hacking: - The argument having access to the responder locator and then eventually it going to a responder function is kind of useless, usually. The responder function already has a reference to the locator, usually called "self" :) - The locator object can be shared between different connections, so a hack using the current status quo where you get the proto using the responder locator is... well, hacky. Also, I do really really want the protocol and not the transport. This is because I want to pass a reference to the protocol around so that later I can call callRemote on it. That I can also get the transport is mostly just gravy so that I can return nice things for my fake transport's getHost/getPeer. So, to reiterate: - I think all the docstrings should reflect the real situation. I guess I need to file a ticket for that. - Maybe there should be a new API that passes the proto (and actually means "proto" ;)) I think I have some code up (or will have some code up soon, depending on when you read this email) that does have sort-of working multiplexed transports: https://github.com/lvh/txampext When done, this will be available from pypi as txampext==0.0.6. If anyone is interested in this, the next big feature mark will be integrating _habnabit's AMP producer stuff, so that it's not just transports, but IConsumer/IProducer transports as well. cheers lvh

2 2

[Twisted-Python] Update: FreeBSD builders
by Tobias Oberstein 20 Nov '13

20 Nov '13

Hi, the 3 builders for the new FreeBSD 9.2 x86-64 slave are up and running: http://buildbot.twistedmatrix.com/buildslaves/tavendo-freebsd-9.2-amd64 Nov 20 09:42 601a9d1cb6eb... failure freebsd-9.2-amd64-python2.7 #12 Failed 1 test (select) 1 test (poll) 3 tests (kqueue) Nov 20 09:42 601a9d1cb6eb... failure freebsd-9.2-amd64-pypy2.2 #8 Failed 28 tests (select) 28 tests (poll) 93 tests (kqueue) Nov 20 09:42 601a9d1cb6eb... failure freebsd-9.2-amd64-pypy-nightly #6 Failed 28 tests (select) 28 tests (poll) 31 tests (kqueue) CPy 2.7: this is a vanilla build from Python 2.7.6 sources PyPy 2.2: this is a vanilla build from PyPy 2.2 sources PyPy Nightly: this is build by the PyPy buildslave (running on the same host) from PyPy head All binaries are compiled using Clang 3.3 (which will be the default toolchain beginning with FreeBSD 10, kernel to userland). The biggest issue that was causing a bunch of IPv6 tests to fail is already solved and merged: https://twistedmatrix.com/trac/ticket/6843 Thanks Jean-Paul (exarkun) and Laurens (lvh)! Regarding the remaining failing tests: 1) CPy 2.7 / select & poll reactors There is 1 identified issue left: https://twistedmatrix.com/trac/ticket/6846 2) CPy 2.7 / kqueue reactor Apart from #6846, in above run, there are 2 failing tests. My problem is: I think I have seen the _same revision_ yield between 0 - 4 failing tests for kqueue. I'll watch that, and then see. 3) PyPy 2.2 & nightly / select & poll reactors I would focus on these 28 failing tests first, since they are "stable" both across PyPy 2.2 / Nightly and select/poll reactors. >From a first quick lock, there might be some low hanging fruit - tests that probably test CPy specific behavior like. I guess analyzing and fixing the PyPy related issues won't happen tomorrow. It'll take some time. Cheers, Tobias

1 0

[Twisted-Python] AMP Argument.toBox's proto argument is a locator, not the proto?
by Laurens Van Houtven 20 Nov '13

20 Nov '13

Hi everyone, I think I've hit one of those cases where AMP really seems to want everything (locator, receiver, sender) to be an instance of t.p.amp.AMP :-( I've written some code that tries to multiplex stream transports over AMP: https://github.com/lvh/txampext/blob/multiplexing/txampext/multiplexing.py The repo contains an example server and client, which demonstrate the issue: https://github.com/lvh/txampext/blob/multiplexing/docs/examples/multiplexin… https://github.com/lvh/txampext/blob/multiplexing/docs/examples/multiplexin… In order to do some of this multiplexing, I need access to the protocol instance inside the responder on the server side. Fortunately, I already had some code that exposed box senders (after a lot of advice from Glyph). I modified it to expose the protocol as well: https://github.com/lvh/txampext/blob/multiplexing/txampext/exposed.py#L41 However, it turns out fromBox gets called with the *responder locator* as the "proto" argument, not the actual protocol. The server has a pudb call that makes it easy (?!) to trace this down. The CommandLocator class, inside doit (a function defined in _wrapWithSerialization) passes "self" to command.parseArguments: https://twistedmatrix.com/trac/browser/trunk/twisted/protocols/amp.py#L1015 This is the part where I think the contract is broken, since parseArguments claims to want the protocol (well, it says it wants the AMP protocol, which, subclassing everything, is also all of the things, of course), but receives the responder locator. What am I doing wrong? Is this a bug? confusedly, lvh

2 6