Mailman 3 February 2014 - Twisted

[Twisted-Python] Status of twistedchecker
by Adi Roiban 22 Feb '14

22 Feb '14

Hi, While submitting a patch on Twisted I got a review response that my code is not complaint with `twistedchecker`. My first reaction was: Holy code, what is twistedchecker! Searching the source code for `twistedchecker` gave no results and on the website there were only 2 wiki pages... none talking about how to use twistedcheck as a Twisted developer. Hitting the #twisted-dev channel I found out about twistedchecker and install version 0.2.0 Now that I got some info about twistedchecker and wiki write access I would like to update the https://twistedmatrix.com/trac/wiki/TwistedDevelopment wiki page with information about how a Twisted developer is expected to use twistedchecker. I run twistedcheck on trunk and got a lot of errors. Am I suppose to look through all those error and filter the ones generated by my changes? Since twistedchecker is only about twisted, shouldn't it be included as a dependency to twisted-dev-tools so that as I developer I would only need to install one extra package and it will pull all development/testing depenencies? Thanks! -- Adi Roiban

2 1

[Twisted-Python] Implement HTTP request acception / rejection as defined in RFC 2616 - 8.2.3
by Adi Roiban 21 Feb '14

21 Feb '14

Hi, As requested during review I am moving this to mailing list for discussions. The initial ticket is here : https://twistedmatrix.com/trac/ticket/6928 ------- The current Twisted implementation of RFC 2616 - 8.2.3 is very simple and Twisted will always reply with continue to Accept: 100-continue requests. I am working at web application based on Twisted web so I did a quick security check for current twisted web code. The major problem is that when reading POST data, requests under 1M are kept in memory, the other being stored in temporary file... but for form-mulipart request the whole data is loaded again in memory for parsing. Now, if the client initially reports that content is less than 1M, but then continue to send more than 1M, twisted.web will continue to keep all data in memory. To address the problem of unwanted big post requests I first start with implementing support for well behaved HTTP/1.1 clients. In current code, Resource objects are called only after whole request data was received and processed so a resource can not decide whether or not it accepts the request, just by looking at headers. Also the Request object is called after headers are received but with calls like parseCookie or gotLength. So there is no API for Request to accept or reject a request based on headers. -------- To solve this problem, I add a new `headersReceived` API call on Request and Resource so that Requests and Resources can decided if they want to accept the request. The biggest change required to implement this API is that URL traversal can now begin before a resource has received all its content. If a resource needs full request content before making a traversal decistion, then this API does not work. This is why this is an optional API not enforced on all resources. For now the ticket only has demo code as I wanted to highlight the API changes. I have already implemented this code in my project and wrote full test for these changes. If this kind of API is accepted in Twisted I can port the code and tests, otherwise I can try to work on a different implementation. Thanks! -- Adi Roiban

1 0

[Twisted-Python] serializing inline callbacks
by Axel Rau 19 Feb '14

19 Feb '14

I have a web application, which runs several subprocesses to build a page. This runs pretty well with subprocess.check_output(…). To improve concurrency with other web sessions, I’m converting this to ireactorprocess.spawnProcess with inline callbacks: —- class SubprocessProtocol(ProcessProtocol): outBuffer = "" errBuffer = "" def connectionMade(self): self.d = Deferred() def outReceived(self, data): self.outBuffer += data def errReceived(self, data): self.errBuffer += data def processEnded(self, reason): if reason.check(ProcessDone): self.d.callback(self.outBuffer) else: self.d.errback(reason) def subprocess(args, ireactorprocess=None): """ :type args: list of str :type ireactorprocess: :class: twisted.internet.interfaces.IReactorProcess :rtype: Deferred """ deblog('Running ' + str(args)) if ireactorprocess is None: from twisted.internet import reactor ireactorprocess = reactor pprotocol = SubprocessProtocol() ireactorprocess.spawnProcess(pprotocol, args[0], args) return protocoled —- I have chained up the calls to subprocess with the expectation, that they will be called sequentially one after the previous has been finished: —- def render_GET(self, request): ... self.A() return NOT_DONE_YET @inlineCallbacks def A(self): ... tmp = yield subprocess(args) ... self.B() @inlineCallbacks def B(self): ... tmp = yield subprocess(args) ... self.C() def C(self): ... x = self.D(...) y = self.D(...) z = self.D(...) ... self.E() @inlineCallbacks def D(self): ... tmp = yield subprocess(args) ... returnValue(tmp) @inlineCallbacks def E(self): ... tmp = yield subprocess(args) ... request.write(...) request.finished —- In C, the three calls to self.D run in parallel. How can I serialize this? Thanks, Axel --- PGP-Key:29E99DD6 ☀ +49 151 2300 9283 ☀ computing @ chaos claudius

4 12

[Twisted-Python] Deadlocks when launching processes - how to investigate?
by Orestis Markou 19 Feb '14

19 Feb '14

Hello, I just filed https://twistedmatrix.com/trac/ticket/6972 The issue I'm facing is a deadlocked Python on OS X when a lot of processes are spawned. In the repro script we do this very aggressively to trigger the deadlock quickly, but the actual program that does this "ticks" every minute. There is a possibility that this is either a Python bug or an OS X issue as the same program used to run fine in 10.5 and after some upgrades to 10.7 this issue appeared. We worked around it by using 2.6 but now we need 2.7. I know this is going to be difficult for people to reproduce, so I wonder if someone can help me investigate the issue further. I found this https://dev.launchpad.net/Debugging/GDB but it doesn't work - I believe, without being able to confirm, that the issue is that GDB can't really work with clang-built executables? Or perhaps I don't have the debugging symbols. Any help would be greatly appreciated. Orestis

6 9

[Twisted-Python] Weekly Bug Summary
by trac＠twistedmatrix.com 16 Feb '14

16 Feb '14

1 0

[Twisted-Python] positioning framework
by Kasun Thennakoon 13 Feb '14

13 Feb '14

Hi, It took long time to go through the ticket[1] and understands, still i couldn't find any issue with it, instead its working perfectly for me. and thanks to the twisted positioning NMEA adapter, it become easy to me to write a adapter for the device(MVT380 device[2]) which i'm using in my project[3]. IMHO, 1. isn't it good to use items() rather than iteritems() since python 3.x has deprecated iteritems(), 2. In nmea._UNIT_CONVERTERS dictionary mapper, can't see any usage of key 'K' which convert Km/h unit to meters/s because it has a FIXER for speedInKnots but not for speedInKMh, i think in $GPVTG NMEA string type (it is not defined in _SENTENCE_CONTENTS) string they use text 'K' to indicates that speed over ground is in kilometers/hour. other than those suggestions,its works perfectly well. i wonder why this positioning module didn't merged with the trunk yet,and i'll be happy if i can help to get it merged. [1] https://twistedmatrix.com/trac/ticket/3926 [2] http://www.mvt380.com/ [3] https://github.com/tmkasun/port/blob/kasun/server_Program/server_twisting/s… -- ~ Kasun Thennakoon <http://me.knnect.com>

2 2

[Twisted-Python] Pure twisted load balancer (possible?)
by Oscar Campos 11 Feb '14

11 Feb '14

Hi folks. It's possible to run several instances of the same twisted based application sharing the same port and balancing the access to them? If it's possible, can you give me some link to any documentation about that or the code were it is implemented? Thank you. Oscar Campos. -- Join the free software foundation and become free as in freedom

2 2

[Twisted-Python] Like to write some code for Twisted
by Surya 10 Feb '14

10 Feb '14

Hello there, I'm Surya - github.com/ksurya I have been working on Django and consider as beginner in Python and Django or may be a little bit more than a beginner. After dealing with pretty high level layers, I am curious to work on the low level layers and contribute some code. That said, I haven't really worked on Twisted anytime and didn't dwell into this stack. So, I was wondering if there anyone who can offer some specific help to get me started. Cheers Surya

4 3

[Twisted-Python] Weekly Bug Summary
by trac＠twistedmatrix.com 09 Feb '14

09 Feb '14

1 0

Re: [Twisted-Python] how to hire a Twisted developer to implement the feature?
by Laurent Domenech-Cabaud 08 Feb '14

08 Feb '14

Hi Laurens, Thanks. I’ll email you privately. Cheers, Laurent On 7 févr. 2014, at 20:00, twisted-python-request(a)twistedmatrix.com wrote: > Message: 3 > Date: Fri, 7 Feb 2014 18:44:49 +0100 > From: Laurens Van Houtven <_(a)lvh.io> > To: Twisted general discussion <twisted-python(a)twistedmatrix.com> > Subject: Re: [Twisted-Python] how to hire a Twisted developer to > implement the feature? > Message-ID: > <CAE_Hg6aQWOo7gpHxv18=MRPHD8hAj2jxpg-cZfwFsUYOpktfmw(a)mail.gmail.com> > Content-Type: text/plain; charset="iso-8859-1" > > Hi Laurent, > > Apologies for brevity; I am on my phone. > > I am a Twisted core developer, and the author of minitrue, a software > package that uses the proxy parts of Twisted. I am currently available for > contracting. > > hth > lvh > On Feb 7, 2014 5:12 PM, "Laurent Domenech-Cabaud" <ldc(a)ldom.net> wrote: > >> Hello, >> >> We're looking to hire (contract) someone with a good knowledge of Twisted, >> ideally a dev of this great project. >> >> What we need is fairly simple: implement an HTTPS transparent proxy to >> work with a basic HTTP proxy (based on the example from the doc). The HTTPS >> proxy will just have to pass the data to/from the destination server >> (untouched, but some filtering will be added by us on the destination host >> name) and the HTTP proxy will do a little more filtering (this portion is >> not part of the work to be done). >> >> Please let me know. I'm at your disposal for more details. If it's not the >> proper place to ask, please accept my apologies. >> >> Cheers, >> Laurent >> _______________________________________________ >> Twisted-Python mailing list >> Twisted-Python(a)twistedmatrix.com >> http://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python >> >

1 0