Mailman 3 January 2014 - Twisted

[Twisted-Python] twisted.web secured session cookies and multiple sites on same host
by Adi Roiban 28 Jan '14

28 Jan '14

Hi, I am looking for the followings: * Force twisted.web to set session cookie as secured when HTTPS is used. * Allow 2 separate twisted.web Sites to run on same host/IP but with different sessions. I have checked the current code for getSession() [1] but i can see that session cookie is harcoded and I don't know how to use self.sitepath . I tried to search the code for sitepath, but I only got this comment, which I things is used to suppport some some sort of load balancing / high availability. # Sitepath is used to determine cookie names between distributed # servers and disconnected sites. Can this be done with current code? If not, I am happy to submit a ticket and a patch. To solve my problems I rewrote getSession to not use harcoded session name and to check for self.isSecure before setting a new session cookie. Many thanks! [1] https://github.com/twisted/twisted/blob/trunk/twisted/web/server.py#L384 -- Adi Roiban

2 2

[Twisted-Python] Weekly Bug Summary
by trac＠twistedmatrix.com 25 Jan '14

25 Jan '14

1 0

[Twisted-Python] Announcing pyOpenSSL 0.14a3
by exarkun＠twistedmatrix.com 23 Jan '14

23 Jan '14

Hello all, I'm happy to announce the third alpha leading up to the release of pyOpenSSL 0.14. pyOpenSSL is the library Twisted uses for all of its SSL functionality. pyOpenSSL 0.14 is bringing some major changes compared to pyOpenSSL 0.13. Any and all testing is greatly appreciated. You can read more about the changes and download the source tarball from the github release page: https://github.com/pyca/pyopenssl/releases/tag/0.14a3 Feel free to leave feedback here, on a github issue, or on the pyOpenSSL mailing list. Thanks! Jean-Paul

4 7

[Twisted-Python] Limit headers by size and not by number in twisted.web
by Adi Roiban 22 Jan '14

22 Jan '14

Hi, I have checked twisted.web.http.HTTPChannel and I see that it limits the headers by number and not by size. https://github.com/twisted/twisted/blob/trunk/twisted/web/http.py#L1596 Maybe I don't see the big picture, but wouldn't this allow the server to load in memory a request with a single header of 1G ? Other than changing the HTTPChannel.lineReceived() implementation, is there a way to limit header size? I did a quick test and the server accepted an 1M cookie value without any problem. curl --cookie cookies.txt http://localhost:8080 cookies.txt in in this format https://gist.github.com/adiroiban/8557725 It this a feature or a bug? Many thanks! -- Adi Roiban

2 3

[Twisted-Python] PLEASE TEST: the new pyOpenSSL alpha release is a major change, and if you use TLS at all with Twisted you should test it
by Glyph Lefkowitz 21 Jan '14

21 Jan '14

Hi everyone, Since this particular detail was in the middle of Jean-Paul's release announcement, I feel like it's important to draw attention to it. The new pyOpenSSL v0.14 is a substantial rewrite of all of pyOpenSSL's bindings to OpenSSL: while this rewrite should be a vast improvement, because eliminates lots of error-prone hand-written C code, replacing it with much smaller, much safer declarative C bindings via cffi, it also means that things are different now. Different means “possibly broken” so please test any Twisted applications that use TLS with this pre-release. Sorry for all the bold text and capital letters, but this is both a big change and also a change to a particularly security-sensitive area of the Twisted stack. If there were ever a change that you should really test before it's marked as “final release”, this is it. -glyph

2 1

[Twisted-Python] Twisted and symlink
by Toph Bei Fong 21 Jan '14

21 Jan '14

Hello everyone, i was trying to port Python, Twisted, etc.. to VxWorks. My Problem now is: twisted/python/lockfile.py (import symlink). VxWorks doenst support symlink. My Questions are: * For what is the symlink used? * Can i disable them? * Is there an old version without symlink? Is there any other alternative or solution for the symlink? Best regards.

2 1

[Twisted-Python] possible error in twisted app
by Jonathan Vanasco 19 Jan '14

19 Jan '14

a portion of my twisted app is having some problems. i think i figured out the issue -- but if I'm right.. i'll be a bit lost. this portion of the app is essentially a web scraper. it grabs a batch of X urls from a data broker , and then updates a database with data about the URL ( which either comes from an oEmbed endpoint , a third party data provider, or scraping the page if needed ) there's a lot of code that would be messy to follow, so i'll just explain it as best as possible, and provide some highlights. the underlying logic is basically this: reactor starts an UpdateLinksService, that checks for new batches every 30 seconds the UpdateLinksService has an internal marker to check if it's still processing the last batch - or if it's safe to process to process the urls, the UpdateLinksService runs them in a request wrapper , that is supposed to be run through a defer.DeferredSemaphore() service when i'm done with the batch, i clear out internal marker via a `deferred_list_finish` method. looking at some aggressive debugging output, it looks like my work to process a url is happening /after/ i call deferred_list_finish. in other words, i've somehow structured this so that i'm instantly finished. i *thought* i was running out of memory because i had some phantom deferreds running around. now i'm starting to think that i'm just stacking the queue faster than i work on it. i've tried changing things around and using different return values, but then started getting "exceptions.AssertionError:" because "assert not isinstance(result, Deferred)" ( twisted/internet/defer.py", line 381, in callback ) the following is a rough composite of what is going on. if anyone sees an obvious fix, i'd be greatly appreciative. thanks! ================= class UpdateLinksService(): def process_urls(self, urls): requests = [] for url in urls: wrapper = requestWrapper( self.semaphoreService, dbPool ) d = wrapper.queue_url(url) updates.append(d) self.d_list = defer.DeferredList( updates )\ .addCallback( self.deferred_list_finish ) class RequestWrapper(): def __init__(self, semaphore_service, dbPool): self.semaphoreService=semaphore_service self.dbPool = dbPool def queue_url( self, url ): self.url = url d = self.semaphoreService.run( self._to_thread ) return d def _to_thread( self ): d = threads.deferToThread( self._thread_begin ) return d def _thread_begin(self): worker = UrlWorker() d = self.dbPool.runInteraction( worker.process_url , self.url ) class UrlWorker(): def process_url(self,txn, url): #blocking stuff return True/False The reason why I have _to_thread + _thread_begin as 2 functions, and UrlWoker separate is for code re-use. The RequestWrapper functions are mostly all in a base class; i just subclass RequestWrapper and override _thread_begin and an error callback (not shown) UrlWorker's various methods are used througout my twisted daemon.

4 4

[Twisted-Python] Weekly Bug Summary
by trac＠twistedmatrix.com 18 Jan '14

18 Jan '14

1 0

[Twisted-Python] starting point to contribute
by Kasun Thennakoon 18 Jan '14

18 Jan '14

Hi all, I'm kasun from srilanka, and i'm developing a vehicle tracking system for one of our government authorities,when i start coding for the project, I used the support of basic python modules(socket,threading)[1], then I got to know about the usability of Twisted and re-factored the code[2] to use Twisted because it's a great python network framework. so I have hands on experience with the framework.I'm happy to start contributing to twisted community.and i have already gone through two articles about the development process (ContributingToTwistedLabs and TwistedDevelopment).So if you can suggest me a good starting point I would be more than happy to contribute reference: [1]: https://github.com/tmkasun/port/blob/kasun/server_Program/server_tcp.py [2]: https://github.com/tmkasun/port/blob/kasun/server_Program/server_twisting/g… -- ~ Kasun Thennakoon <http://me.knnect.com>

4 6

[Twisted-Python] strategies for tracking down memory issues
by Jonathan Vanasco 17 Jan '14

17 Jan '14

some recent changes to a very-happy twisted daemon have resulted in a process that grows in memory until it crashes the box. boo! looking through the code and logs, i'm wondering if i''ve coded things in such a way that defferds or deferrd lists are somehow not getting cleaned up if an unhandled exception occurs. i've been looking through all my former notes and some questions on stack overflow, and I've seen a lot of info on using heapy and other tools to find issues on a function-by-function basis. i'm wondering if anyone has experience in simply monitoring the lifecycle of deferreds ?

4 4