Mailman 3 December 2014 - Twisted

[Twisted-Python] not terrible object oriented rpc system in js?
by Daniel Sank Dec. 28, 2014

Dec. 28, 2014

Are you guys aware of anything which solves the same kind of problem as t.s.pb but in js? I asked about this on the js stack exchange chat room and the general response was that there's isn't anything. I figured y'all might know more. -- Daniel Sank

4 4

[Twisted-Python] Sigs/Archive
by Clayton Daley Dec. 28, 2014

Dec. 28, 2014

I forgot to delete my tel from my sig in a reply. Is there anything I can do to avoid leaving it published in the archives indefinitely? Thanks, Clayton Daley

1 0

[Twisted-Python] Scrapy spiders waiting in reactor thread when callFromThread gets call repeatedly
by Adi Lavi Dec. 23, 2014

Dec. 23, 2014

Hi, I am using Pika's asynchronous consumer implementation with Scrapy and Twisted. I have twisted reactor running on the main thread, and Rabbit consumer running on a background thread. When I get a message and want to start my spider, I use 'callFromThread' to wake the reactor thread, init the spider and start crawling. Alas, on high load of Q messages, I find that because 'callFromThread' is called all the time, Scrapy does not start downloading until there is some 'break' in these calls. I am wondering what is the best approach to gain high scale with Scrapy, Twisted and RabbitMQ. Should I continue using the current design, and simply do some buffering or batching to reduce the 'callFromThread' frequency? Perhaps I should use a synchronous design? Thanks

2 1

[Twisted-Python] Weekly Bug Summary
by trac＠twistedmatrix.com Dec. 21, 2014

Dec. 21, 2014

1 0

Re: [Twisted-Python] Python Twisted Questions
by Adam Michalski Dec. 18, 2014

Dec. 18, 2014

Hi All! Thanks so much for the response. We’ve looked into it and here are our findings: We have implemented Linereceiver in our original script to receive the data. It’s working successfully if we connect to the server through terminal – i.e. we’re getting proper input from terminal client and also proper response is received to the terminal client. It fails when we try to integrate this with the iOS. i.e. it’s connecting to the server successfully but not getting the inputs properly from the iOS client. e.g., if the command is: "demo:{"userid":"1","chatroomid":"1"}" On server I am getting the below: demo:{ Please find attached the python script. Note: This seems to be an iOS specific Python issue - when running the same script from Mac OS X it is functioning as expected. If there’s anyone you believe we should redirect the question towards, just let me know. :) Again, I really appreciate your help with this. You’ve been super helpful. -Adam Founder, DormChat dormchat.com Begin forwarded message: On Tuesday, Dec 2, 2014 at 8:22 PM, Glyph <glyph(a)twistedmatrix.com>, wrote: On Dec 2, 2014, at 23:30, Adam <adam(a)dormchatapp.com> wrote: I assume you're using this tutorial? <http://www.raywenderlich.com/3932/networking-tutorial-for-ios-how-to-create…> It is unfortunate that this has not been updated with better information, it seems to confuse a lot of people making iOS applications ;). 1. Please find attached Python script. We have integrated the Twisted framework with iOS for chat module. Attached is our Python demo code in which we have implemented this. The issue is when we try to connect with the server through the terminal, we are getting the complete correct response without any error. But when it was integrated on the iOS app, data is getting received in chunks (i.e. if data exceeds 1366 bytes or above in json size it comes in pieces which creates a problem when sending and receiving chats) This is a frequently asked question: <https://twistedmatrix.com/trac/wiki/FrequentlyAskedQuestions#Whyisprotocol.…> In other words, if you want a message-oriented protocol, you can't just use dataReceived. For sending short JSON structures back and forth, line-delimited data structures are a reasonable choice. On the Twisted side, you can use twisted.protocols.basic.LineReceiver, and override lineReceived instead of dataReceived. On the iOS side, you need a radically different structure for buffering your input; you can't just stuff it into a buffer on the stack. One quick hacky way to do this would be to give your object a NSString bufferString instance variable, and do bufferString = [bufferString stringByAppendingString: [[NSString alloc] initWithBytes:buffer length:len encoding:NSASCIIStringEncoding]]; NSArray* lines = [bufferString componentsSeparatedByString: @"\r\n"];, bufferString = [lines lastObject];, then loop over the other items in lines. Keep in mind you also need to do this for output; as you send output to the server, you may not be able to send all of it at once, so you need to keep a buffer, and keep track of how much you've written, etc. Or you could just build Python into your app on iOS and run Twisted in there. If you manage to get that working be sure to let us know how it goes :-). I am lead to believe that <http://omz-software.com/pythonista/> contains a template for at least getting the Python runtime going... -glyph

1 0

[Twisted-Python] Weekly Bug Summary
by trac＠twistedmatrix.com Dec. 14, 2014

Dec. 14, 2014

1 0

[Twisted-Python] Tubes!
by exarkun＠twistedmatrix.com Dec. 12, 2014

Dec. 12, 2014

Hello all, Some of you may have heard rumors of some work in progress on a replacement for Twisted's IConsumer/IProducer interfaces. Tubes have been largely Glyph's effort (though a lot of people have contributed in one way or another). And a large effort it's been. Development is proceeding in a Twisted branch and comes to over three thousand lines of additions so far. Given the large size of the implementation and the long time that this effort has been underway (I remember the Twisted meetup at the Rackspace offices that *I* attended when I was visiting SF... a year and a half ago... at which point tubes wasn't exactly a brand new project) I'd like to re-raise the idea that the best next step for the project is to see some distribution in its *current* state. Specifically, I think it would be beneficial to set up a tubes project on Github under the Twisted organization and try for a release in the very near future. I think this has several advantages over the status quo: 1) As an independent project, tubes will attract more attention than it presently gets as a relatively unknown ticket & branch of Twisted. 2) As a separate Python package, the logistics of actually using tubes are simpler (just consider how you might declare a dependency on a branch of Twisted - keeping in mind you may want to use tubes in a project that already depends on some version of Twisted). It may not make sense to say that it is the same quality as Twisted proper right off the bat (on the other hand, it may well - I suspect tubes in its current form actually is a lot higher quality than large sections of Twisted) but that doesn't mean people (not to mention the tubes project) can't benefit from being able to experiment with it. 3) Decoupling tubes from Twisted frees tubes from certain of Twisted's policies which are more challenging to follow for the kind of non- trivial, brand new code base that tubes is. Technically we could just say that these policies don't apply to a tubes package *in* Twisted but this kind of subtle distinction is often lost on users (ie application developers). a) Twisted's compatibility policy need not apply. It could either be sped up or abandoned more thoroughly. I'm generally a fan of being backwards compatible even when you have few users because it actually makes development easier, but loosening the policy to say things might break if it's just really inconvenient to keep them working (whereas Twisted goes to the inconvenience to keep them working) seems reasonable. b) tubes can undergo a faster release cycle to benefit more from user feedback. 4) At this point, a normal review of the tubes branch is going to be a problem. We do not have good tools or mechanisms for dealing with branches this large. The code in the current tubes branch can just become master of a new project. Development going forward from this point should continue to follow the feature-branch, small-changes, pre- commit-peer-review process. But those 3k lines are written already. Short of an extremely expensive effort to break the work up into smaller, self-contained pieces there's simply never going to be a *good* review in the typical style. Additionally, it may turn out that tubes can remain independent indefinitely. Someday perhaps Twisted would come to depend on it to allow the various protocols and applications implemented in Twisted to benefit from the superior abstractions it provides. Or maybe once it has undergone a few iterations it will make sense to bring it back to Twisted. I don't think this needs to be decided now. There are downsides, of course. All of the boring maintenance involved with having a separate project - setting up CI, actually doing the releases, etc. Perhaps we could find some volunteers to help out with these tasks, though, in exchange for getting some great code out there? I'm curious what the folks out there who develop applications using Twisted would find to be the easiest path forward. I'm also curious to hear what Glyph thinks about all this. ;) Jean-Paul

10 14

[Twisted-Python] ANN: Eliot 0.6: Logging for Complex & Distributed Systems
by Itamar Turner-Trauring Dec. 10, 2014

Dec. 10, 2014

Eliot is a logging system designed not only for simple applications but for complex applications as well, including distributed systems. Eliot supports simple structured messages but can also record a causal chain of actions happening within and across process boundaries: a logical trace of the system's operation. Structured, action-oriented logging is a great help when debugging problems. For example, here are the combined logs of a request originating from a client process being sent to a server. Notice how easy it is to figure out the cause of the problem, even though it's opaque to the client: process='client' task_uuid='40be6df2' task_level=[1] action_type='main' action_status='started' process='client' task_uuid='40be6df2' task_level=[2, 1] action_type='http_request' action_status='started' x=5 y=0 process='server' task_uuid='40be6df2' task_level=[2, 2, 1] action_type='eliot:remote_task' action_status='started' process='server' task_uuid='40be6df2' task_level=[2, 2, 2, 1] action_type='divide' action_status='started' x=5 y=0 process='server' task_uuid='40be6df2' task_level=[2, 2, 2, 2] action_type='divide' action_status='failed' exception='exceptions.ZeroDivisionError' reason='integer division or modulo by zero' process='server' task_uuid='40be6df2' task_level=[2, 2, 3] action_type='eliot:remote_task' action_status='failed' exception='exceptions.ZeroDivisionError' reason='integer division or modulo by zero' process='client' task_uuid='40be6df2' task_level=[2, 3] action_type='http_request' action_status='failed' exception='requests.exception.HTTPError' reason='500 Server Error: INTERNAL SERVER ERROR' process='client' task_uuid='40be6df2' task_level=[3] action_type='main' action_status='failed' exception='requests.exception.HTTPError' reason='500 Server Error: INTERNAL SERVER ERROR' WHAT'S NEW IN 0.6.0 Incompatible output format change! In previous versions the ordering of messages and actions was ambiguous and could not be deduced from out-of-order logs, and even where it was possible sorting correctly was difficult. To fix this the action_counter field was removed and now all messages can be uniquely located within a specific task by the values in an _improved task_level field_ [1]. Features: * Eliot tasks can now _span multiple processes and threads_ [2], allowing for easy tracing of actions in distributed applications. * _eliot.add_global_fields_ [3] allows adding fields with specific values to all Eliot messages logged by your program. This can be used to e.g. distinguish between log messages from different processes by including relevant identifying information. Bug fixes: * On Python 3 files that accept unicode (e.g. sys.stdout) should now work. Downloads are available on PyPI [4]. Documentation can be found on Read The Docs [5]. Bugs and feature requests should be filed at the project Github page [6]. You can ask for help on IRC at the #eliot channel on irc.freenode.net [7]. Links: ------ [1] http://eliot.readthedocs.org/en/0.6.0/fields.html#task-fields [2] http://eliot.readthedocs.org/en/0.6.0/threads.html#cross-process-tasks [3] http://eliot.readthedocs.org/en/0.6.0/output.html#add-global-fields [4] https://pypi.python.org/pypi/eliot [5] https://eliot.readthedocs.org/ [6] https://github.com/ClusterHQ/eliot [7] http://irc.freenode.net

1 0

[Twisted-Python] Mixing Twisted with Kivy
by Mike Epps Dec. 9, 2014

Dec. 9, 2014

Hello, Im looking for examples of mixing twisted with kivy. The only examples I have seen are from the kivy site but they do not include examples that show how to use the kv language files separate from the python code. I would appreciate any links that show such examples. thnx..

1 0

[Twisted-Python] Mixing Python threads and Twisted code
by Alexandre Quessy Dec. 7, 2014

Dec. 7, 2014

Hello everyone, What happens if some code starts a thread while I am using Twisted's reactor? Here is what I need to use: https://github.com/ekulyk/PythonPusherClient/blob/master/pusherclient/conne… This thread polls some socket to handle received TCP messages. It appears it has been very slow in production. Is it this mix of twisted and non-twisted threading that causes this brutal slowdown? If so, how can I easily speed it up without having to twistify the library I need to use? (and practically rewrite it) Thank you, -- Alexandre Quessy - 亚历山大 - ألكسندر http://alexandre.quessy.net - Artiste en nouveaux médias http://perte-de-signal.org - Membre de Perte de signal http://sourcelibre.net - Développement logiciel et conception de sites Web http://labalab.ca - Impliqué dans LabàLab - communauté de pratique en arts numériques

2 2