Mailman 3 March 2013 - Twisted

[Twisted-Python] Reason of calling Stale Broker inside loopback
by Sergey Gerasimov 30 Mar '13

30 Mar '13

Hello, 1. What is the reason of error "DeadReferenceError: Calling Stale Broker"? My program uses PB inside loopback, so I shouldn't have any disconnects. 2. I call many time remote method returning pb.Referenceable. Can it be a reason of topic 1 or generally lead to system's degradation? Sergey.

2 1

[Twisted-Python] Weta Digital is hiring
by Donal McMullan 29 Mar '13

29 Mar '13

"Weta Digital is a world leading visual effects company based in Wellington, New Zealand." The "Production Engineer" roles on our jobs page are heavily Python focused. Some of that Python code will be inside applications like Autodesk's Maya or the Foundry's Nuke, but most of it will solve the kinds of problems I'm sure most of you work on routinely. http://www.wetafx.co.nz/jobs For example, because our render farm is on the large side, we face many of the challenges of scale that those of you who work on internet-facing applications will be familiar with. On the client side, we have an ocean of data to track, and we try to provide our artists with simple tools to get the most from that data. Frameworks like PySide, Bottle and TxWeb help us to get there. Unsurprisingly, Twisted is well established in the pipeline at Weta, and it's used in a diverse set of client, server and automated/unattended applications. We've had superb service from Twisted, and we only expect itsusage here to grow. We have roles in Production Engineering that we'd like to fill as soon as we can, but we're protective of the great team that we have, and we try to grow it carefully. We prefer to wait for the right people. If you're interested in helping us to deliver some extraordinary software and film projects, we'd love to hear from you, and the recruitment form on our site takes just a few minutes to fill in. http://www.wetafx.co.nz/recruitment-form.php?job-title=Production%20Enginee…<http://www-ext.wetafx.co.nz/recruitment-form.php?job-title=Production%20Eng…> Thank you Donal McMullan Senior Production Engineer Weta Digital http://www.wetafx.co.nz

1 0

[Twisted-Python] is twisted compatible with pickle?
by succer110＠tiscali.it 29 Mar '13

29 Mar '13

I have made 2 application: The client extract data from a sql server (10k lines), and send every line pickled to a "collector" server via socket.The server uses twisted and receive every line, unpikle it and store the data in another sql server. Everytime i start sending data from client to server, in the first 200 line (everytime a different line) **the server** throws an exception:SOMETIMES it something like: Traceback (most recent call last): File "collector2.py", line 81, in dataReceived self.count,account = pickle.loads(data) File "/usr/lib/python2.6/pickle.py", line 1374, in loads return Unpickler(file).load() File "/usr/lib/python2.6/pickle.py", line 858, in load dispatch[key](self) File "/usr/lib/python2.6/pickle.py", line 1138, in load_pop del self.stack[-1] IndexError: list assignment index out of range But it's NOT every time the same. Printing my exception i red: Exception: pop from empty listException: list index out of rangeException: "'"Exception: list assignment index out of range Another strange errors is:File "/usr/lib/python2.6/pickle.py", line 1124, in find_class __import__(module)exceptions.ImportError: No module named ond' for i in listaSAI: crm={} try: crm['uid']=i[0] except: crm['uid']=None try: crm['type_cond']=i[01] except: crm['type_cond']=None try: crm['population_id']=i[2] except: crm['population_id']=None try: crm['product_id']=i[3] except: crm['product_id']=None try: crm['a_id']=i[4] except: crm['a_id']=None try: crm['status']=i[5] except: crm['status']=None #time.sleep(0.001) serialized = pickle.dumps((count,crm)) #print "sent num", count, crm s.sendall(serialized) count += 1 And my server: def dataReceived(self, data): try: self.count,account = pickle.loads(data) except Exception as e: print "Eccezione:", e print self.count+1 print data print traceback.print_exc() Printing the data in my client tells me that everything it's ok.*If i try to slow down the process of sending using time.sleep(0.01) in my client, EVERYTHING IS FINE, and no exception are raised.* What can i do to debug my code? p.s. I suspect that exceptions.ImportError: No module named ond' refers to "type_cond" key in crm. Invita i tuoi amici e Tiscali ti premia! Il consiglio di un amico vale più di uno spot in TV. Per ogni nuovo abbonato 30 € di premio per te e per lui! Un amico al mese e parli e navighi sempre gratis: http://freelosophy.tiscali.it/

4 3

[Twisted-Python] Problems with TLS connection not shutting down.
by Michael Williamson 28 Mar '13

28 Mar '13

Hi, I'm having an issue where a tls connection never seems to shutdown properly, and as a result, all the associated cleanup never happens. Basically we're trying to implement a jabber client and allow the user to sign out and then re-sign back in without completely quitting the program. When the user tries to sign back in, Twisted goes nuts because the original connection was never shutdown properly. So, I traced through the code with the debugger and I came to the _shutdownTLS() function in tls.py. I believe that what's supposed to happen is that the first time through, the self._tlsConnection.shutdown() returns False because the shutdown is initiated. Then, when the 2nd part of the shutdown handshake comes back from the server, dataReceived is called, which calls shutdownTLS again, and then the cleanup can continue. The problem is that after the initial call to shutdownTLS, I never see anymore data from the server. I'm not sure if it's a server issue (we're just connecting to the Google jabber servers) where they don't close down their SSL connections correctly, or if I'm just doing something silly. So anyway, I could solve this by overriding a bunch of stuff and ensuring that self.transport.loseConnection() gets called no matter what, even when the server doesn't respond with the second part of the close handshake, but I figured I'd ask here first. So, any TLS experts out there? Thanks for any help or information - I really appreciate it, Mike

2 2

[Twisted-Python] Lore and generating reStructuredText (Lore2Sphinx)
by Kevin Horn 27 Mar '13

27 Mar '13

tl;dr A Lore plugin won't work for generating Sphinx source files, at least not by itself. Itamar posted some notes from the Twisted BoF session that was held at PyCon last weekend, and one of the things in it was the following line: - lore output plugin that generates ReST via docutils parse tree objects, then write code to run sphinx on this output I wasn't there, so I don't know the exact context that this was referring to, but let me try to explain a little bit about why this won't work (at least not as written). reStructuredText, as some of you may know, creates it's output by first creating an intermediate representation of a document called a "node tree", which is a tree of "nodes" which represents the various elements in a document (text, paragraphs, lists, list items, etc.). reStructuredText also has a construct called a "directive", which is some markup which tells the docutils reST parser to create a bunch of these nodes. Directives are awesome and are a big reason why reStructuredText is so much more powerful than other lightweight markup languages like markdown, textile, etc. They serve as extension point and allow users to create their own markup constructs without changing the actual parser. The key thing is that a directive is not itself a type of node. Rather it 's just a markup construct. This means that once a reStructuredText document goes through the docutils parser, the information about the directives is lost, because they have been transformed into a bunch of nodes. For example there's a container directive, which looks like this: Title ===== .. container:: I'm a content paragraph! Yay! When processed this creates a nodetree that looks something like this (in docutils "pseudoxml" representation: <document ids="title" names="title" source="test.rst" title="Title"> <title> Title <container> <paragraph> I'm a content paragraph! Yay! It is entirely coincidental that the container directive and the <container> node are named the same thing. Don't let this confuse you. The point is that the directive goes away and is replaced by a bunch of nodes (more specifically, the node tree is transformed in some way...I suppose a directive could remove nodes, but I don't think I've ever seen that done). We can see this using another example: Here's some markup: Title ===== .. warning:: I'm a content paragraph! Yay! and here's the pseudoxml representation of the nodetree: <document ids="title" names="title" source="test.rst" title="Title"> <title> Title <container> <paragraph> I'm a content paragraph! Yay! Notice that the node trees look exactly the same. Now this is not quite true, as there's probably some attributes on the actual Python nodes that might be used to distinguish them when writing output which aren't displayed here...they certainly get rendered into HTML differently. But the point is that the directive itself is GONE and you have no real way of recreating it from the node tree. I think this problem also happens with custom text roles, which is another extension mechanism in reST, but I haven't looked too deeply into that. Since you really, really want to have directives in your output (in fact you have to have them if you want to use Sphinx, which makes heavy use of them), you can't really generate Sphinx-capable source files using _only_ the nodetree representation. I suppose you might be able to do something where you try to detect where the directive _should_ go and try to insert it during the rendering step, but such a thing would be an egregious kludge, would take a lot of effort, and I can't imagine it would work very well, if at all. Another option would be to fork the distutils parser and change it so that it could create "directive nodes" or something, but I certainly would not recommend such a course. (If you think maintaining Lore is a pain, you ain't seen nothin' yet. And one thing this project has driven home to me is that no software only needs to be maintained "for a little while".) I'm not saying that the proposed plugin for lore is a bad idea...I think it would be pretty cool. You'd be able to send lore out to all of the various formats supported by docutils, and who doesn't want to write their next s5 presentaton in Lore, right? :) But it won't do the job that it was being put forward for in the note Itamar mentioned. So what about building some software that generates some other representation of the source document, and then renders that as reStructuredText? Well this is the best idea I've come up with (or heard) and is in fact exactly what lore2sphinx-ng_ (which is not intended to be a separate thing, it's just an experimental fork of lore2sphinx) and rstgen_ do. lore2spinx-ng creates the representation from lore sources (which is also a tree of "nodes", though they aren't called that), and rstgen defines the nodes, and renders them into reStructuredText source. The only problem is that these aren't done yet, though the work done so far looks very promising (in terms of actually being able to do the job reliably someday). If anyone has bothered to read this far and is interested in helping out, please feel free to fork the repos and lend a hand. Also feel free to contact me either on this list or directly if you have any questions. I apologize in advance for the current state of the code, which is a bit messy (especially lore2sphinx-ng, which still has a bunch of cruft from the "old"/"current" version that I haven't gotten around to removing yet). .. _lore2sphinx-ng: https://bitbucket.org/khorn/lore2sphinx-ng .. _rstgen:https://bitbucket.org/khorn/rstgen -- Kevin Horn

3 9

[Twisted-Python] GitHub Twisted organization/repo
by Adi Roiban 26 Mar '13

26 Mar '13

Hi, Are there any plans with the Twisted repo from GitHub? https://github.com/twisted/twisted Looks like it was not updated in a while. Garret Heaton is already doing a 15min mirroring of master/trunk https://github.com/powdahound/twisted Both repos have a couple of forks and it would be nice if all forks are from the same repo. I prefer and use GitHub code browsing since it is much faster than Twisted's official Trac code browsing. --------------- I now see that Tom Prince has already created git branches on GitHub for the ticket he is reviewing. Many thanks! I have already started creating git branches on GitHub for all my patched for Twisted. I found it much pleasant to send a "live patch" URL rather than a static attached diff file. Ex live diff: https://github.com/chevah/twisted/compare/chevah:master...chevah:5411-ftp-l… Append .diff to the url and you have a patch file: https://github.com/chevah/twisted/compare/chevah:master...chevah:5411-ftp-l… As a non-commiter for SVN this makes my life much more easier. Manually creating diff and attaching them to a slow Trac can cause a bit of frustration... but I always find to hard to follow a SVN branch that was created for at ticket and manually apply diffs on local copies. I find it very hard to work in parallel on more than 4 tickets and tracking diffs outside of SVN/GIT is not fun. I understand that moving from SVN to GIT is not easy and if core developers can create and share SVN branches there is no big inconvenience. For me, as a non-commiter, contributing to Twisted is not fun. Sorry for bothering you with my frustrations. I hope that you find the feedback useful, otherwise feel free to ignore it :) -- Adi Roiban

4 6

[Twisted-Python] Twisted dinner pictures
by Hynek Schlawack 26 Mar '13

26 Mar '13

Hi, As you may have noticed, I'm building a little page to show off that PyCon wasn't about donglegate. http://thisispycon.com I would love to post about our little dinner and would be delighted if anyone having pics AND quotes could forward them to me. Same goes for sprints etc, flood me! Cheers, Hynek Sent from my phone.

3 4

[Twisted-Python] Notes from Twisted BoF at PyCon 2013
by Itamar Turner-Trauring 26 Mar '13

26 Mar '13

5 4

[Twisted-Python] Testing Twisted code without trial
by Adi Roiban 26 Mar '13

26 Mar '13

Hi, I start my pet project using Python Nose as test runner. At the start, Twisted not used at all. Later I discovered Twisted and start using it... but testing Twisted requires trial. I resist migrating to Trial and ended up with a custom code for running Twisted tests together with Nose: https://github.com/chevah/chevah-rhinoplasty I though I can share this code on the list, in case there is someone else wrestling with deferres testing outside of Trial. Any feedback or comments is much appreciated. Cheers, -- Adi Roiban

4 10

[Twisted-Python] So... Python 3.4 is getting its own async I/O system
by Tim Allen 25 Mar '13

25 Mar '13

In Guido's keynote at PyCon 2013, apparently he talked about adding an async I/O module to Python 3.4. It looks like his slides can be viewed here: https://www.dropbox.com/s/xknbe58zcvjhzhv/PyCon2013.pptx ...while this is the PEP he's talking about: http://www.python.org/dev/peps/pep-3156/ At first glance, the proposed reactor API looks very much like Twisted's (or, to be fair, GTK's, or possibly any number of other async event loop I'm less familiar with) but rather than Deferreds and callbacks, the API will be based around Futures (similar, but not identical, to Python 3.2's concurrent.futures.Future class), and an inlineCallbacks-style decorator for generators. I know Deferreds are awesome, and I don't know much about Futures (and I know Twisted core developers have given negative reviews of other Deferred/Promise/Future implementations in, say, JavaScript libraries before), and inlineCallbacks seems to have a negative reputation among experienced Twisted users. Is there anybody on this list who knows more about this new PEP (maybe somebody who's at PyCon and saw the talk in person) who can give us an informed comparison with the current state of Twisted?

8 12