Mailman 3 March 2014 - Twisted

[Twisted-Python] Configuration options for twisted code
by Adi Roiban 04 Mar '14

04 Mar '14

While using twisted.web (but not only) I had to sub-class twisted code to implement the following: HTTPChannel * reduce limit of HTTP headers length * reduce number of HTTP headers * connection timeout HTTPRequest * change HTTP server signature * change HTTP cookie name and secure/HTTPOnly flag Static.File * resource for child not found * resource for forbidden * template for directory listing. During review process, one feedback was to add these options as part of __init__ method for the required classes and another one was to design some configuration system, which allows changing these options without sub-classing. What do you think? Should I go ahead and add options as __init__ arguments or think of a configuration system? Thanks! -- Adi Roiban

4 6

[Twisted-Python] treq // Agent and Content-Length
by Donald Stufft 04 Mar '14

04 Mar '14

I’m making a small proxy server in Twisted which is supposed to essentially "mask" the original destination of an URL. Specifically it does the following: 1. Receives an HTTP request on /<hmac>/<url encoded url> 2. Verifies that the hmac in the URL matches the url decoded url 3. Makes an HTTP request to the url decoded url 4. Verify the response 4a. Verify that the *on the wire* size is not greater than a set number of bytes, prior to downloading the body. 4b. Verify that the content-type is acceptable prior to downloading the body. 5. Download the body of the response without modifying it (No Gzip decode) 6. Return the unmodified downloaded response as the resposne to the HTTP request in 1, as well as copying over several Response Headers. This should be able to pass through ``Accept-Encoding`` from the request in #1 to the request made in #3. I've had a problem getting 4a to happen, no matter what I've tried if the Content-Encoding is set to gzip it appears that the length of the response is unknown. Additionally the "connection headers" are split out from the normal headers, so I cannot seem to locate the original ``Content-Length`` header to simply check that. I got frustrated and implemented it using requests and that has worked OK. It however uses deferToThread and I would like to get back to using the built in tooling. So my question is, how can I make Agent/treq do this? The requests version of the code is here: https://gist.github.com/dstufft/7c8aac898de0ad359675 The treq versions of the code is here: https://gist.github.com/dstufft/0dffb12201d235f35bc7 ----------------- Donald Stufft PGP: 0x6E3CBCE93372DCFA // 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA

1 0

[Twisted-Python] Fixing pydoctor / epydocs errors
by Adi Roiban 03 Mar '14

03 Mar '14

Hi, I would like to fix pydoctor / epydocs errors generated while building twisted api reference documentation. The goal is to have a clean build so that when a developer sees an error in his/her branch, the errors is for sure produced by his/her changes. My current problem is that I can not reproduce the results generated by Twisted buildbot. For example for pair, Twisted buildbot gives [1]: twisted.pair.tuntap._IInputOutputSystem.ioctl:143 invalid ref to fcntl.ioctl twisted.pair.tuntap._IInputOutputSystem.ioctl:143 invalid ref to fcntl.ioctl twisted.pair.testing.MemoryIOSystem.ioctl:442 invalid ref to fcntl.ioctl twisted.pair.testing.MemoryIOSystem.ioctl:442 invalid ref to fcntl.ioctl While on Travis-CI builder, and my local system I get [2]: twisted.pair.testing.MemoryIOSystem.receiveUDP:494 invalid ref to socket.socket twisted.pair.testing.MemoryIOSystem.receiveUDP:494 invalid ref to socket.socket twisted.pair.testing._FakePort.recv:527 invalid ref to socket.socket.recv twisted.pair.testing._FakePort.recv:527 invalid ref to socket.socket.recv twisted.pair.testing._FakePort.recv:527 invalid ref to socket.socket.recv twisted.pair.testing._FakePort.recv:527 invalid ref to socket.socket.recv twisted.pair.tuntap._IInputOutputSystem.receiveUDP:182 invalid ref to socket.socket twisted.pair.tuntap._IInputOutputSystem.receiveUDP:182 invalid ref to socket.socket ----- The command used is python bin/admin/build-apidocs . apidocs Does anyone has any clue what can go wrong and why I get this differences ? Thanks! [1] https://travis-ci.org/chevah/twisted/jobs/19809459 [2] http://buildbot.twistedmatrix.com/builders/documentation/builds/4469/steps/… -- Adi Roiban

1 0

[Twisted-Python] [twisted-web] I/O error with twisted.web.wsgi and Django
by Orestis Markou 03 Mar '14

03 Mar '14

Hi all, I am using Twisted Web WSGIResource to host a Django site. Here's the code I use to setup the WSGI resource (other boilerplate skipped) def wsgi_resource(): from django.core.handlers.wsgi import WSGIHandler pool = threadpool.ThreadPool() pool.start() # Allow Ctrl-C to get you out cleanly: reactor.addSystemEventTrigger('after', 'shutdown', pool.stop) wsgi_resource = wsgi.WSGIResource(reactor, pool, WSGIHandler()) return wsgi_resource This generally worked quite well. However I have an API endpoint that gets hit with multiple POST requests per second (not very much - I'd say 10-20 req/sec at most). Sometimes I see this in the logs: Internal Server Error: /cas/eventlog/ Traceback (most recent call last): File "<pythonpath>/site-packages/django/core/handlers/base.py", line 114, in get_response response = wrapped_callback(request, *callback_args, **callback_kwargs) File "<pythonpath>/site-packages/django/views/decorators/csrf.py", line 57, in wrapped_view return view_func(*args, **kwargs) File "<djangoapp>/views.py", line 488, in event_log data = json.loads(request.body) File "<pythonpath>/site-packages/django/http/request.py", line 192, in body self._body = self.read() File "<pythonpath>/site-packages/django/http/request.py", line 246, in read return self._stream.read(*args, **kwargs) File "<pythonpath>/site-packages/django/core/handlers/wsgi.py", line 45, in read result = self.buffer + self._read_limited() File "<pythonpath>/site-packages/django/core/handlers/wsgi.py", line 39, in _read_limited result = self.stream.read(size) File "<pythonpath>/site-packages/twisted/web/wsgi.py", line 94, in read return self._wrapped.read(size) ValueError: I/O operation on closed file. Any pointers on why would this happen? Why would Twisted close the file early? Could it be that I need to increase the threadpool size? I use treq 0.2.0 for the requests, with the "persistent" flag turned on, if that makes any difference. Twisted 13.2.0 and Django 1.6, python 2.7.1 on OS X 10.7.5. Many thanks, Orestis

2 1

[Twisted-Python] Stream an mp3 file + IP Multicasting
by Jay Kay 03 Mar '14

03 Mar '14

Hi everybody, I am trying to stream an mp3 file via ip multicasting and have found the following script: https://sites.google.com/site/ka010notes/python/twisted#TOC-Multicast-ENCOD… When I run this script on the server I am able to open the stream on VLC player from another device on the subnet but it is laggy/missing/choppy. When I try streaming this mp3 file using VLC player on the server, I am able to successfully open the stream with good quality, without any lags. I have tried playing around with the parameters in this script such as changeing the interval in l.start(0.2) but nothing has worked =/. It would be great if somebody can help me out here. Thanks in advance!

1 0

[Twisted-Python] Weekly Bug Summary
by trac＠twistedmatrix.com 02 Mar '14

02 Mar '14

1 0

[Twisted-Python] Help non-commiter to fully test their changes locally or remote
by Adi Roiban 01 Mar '14

01 Mar '14

Hi, I am trying to gather documentation to help non-commiters test their changes before sending them to review. When run locally some tests fail with a lot of errors and is hard for a developer to filter the errors generated by his/her changes. Ex: * twistedchecker * pyflakes * api-reference I have created patches to clean pyflakes and I also plan to start cleaning the api-reference build errors >From information gathered from #twisted-dev it seems that twistedchecker is not maintained and that some of its errors are invalid... so one should not try to fix all its errors. ... I wander why twistedchecker is still used ? I would like to investigate how we can fix api-reference build errors. ------ I have created a travis-ci configuration file which can be used as a reference for running tests. It somehow copies the builders observed on buildbot.twistedmatrix.com A sample build result can be found here: https://travis-ci.org/chevah/twisted/builds/19803969 Code changes are here. If you find them useful I can refine them into a public patch. https://github.com/chevah/twisted/pull/1/files Only python 2.7 and 3.3 builders are enabled... but 2.6 builders can also be enabled. py-select-gc tests are not enabled since they are very, very slow. Running all tests on a single process could take about 1 hour. I guess that running only tests with default reactor and then pyflakes/pydoctor on changed files should cover most errors. A developer should run at least the following tests before sending a patch for review: * trial with default reactor * pyflakes for changed files * pydoctor for changed files If documentation was changed he/she should also run documentation tests. ---- Running pydoctor on my computer takes about 3 minutes... a bit long for my taste. I hope that pydoctor can be run on a independent python files so that I can create a separate linter test which will run pyflakes and pydoctor only on changes files. This should provide a big improvements in speed. I used travis-ci as it provides an isolated / reference test environment. Once the configuration is ok, it can be translated into tox [1], private buildbot [2] or custom script to allow developers to run test on their computers. If a Twisted developer has time, I need some help to check the current travis-ci errors and suggest how to fix them. Thanks! [1] https://github.com/msabramo/python-panci [2] https://github.com/isotoma/buildbot_travis -- Adi Roiban

2 2