Mailman 3 April 2015 - Twisted

[Twisted-Python] Announcing Eliot 0.7: Logging for complex and distributed systems (with nice Twisted support)
by Itamar Turner-Trauring 28 Apr '15

28 Apr '15

Imagine you have a client that sends a request to a server and gets back a 500 error. What caused it? If you're using Eliot you can trace the logs across processes: $ cat server.log client.log | python eliottree.py e076ca50-9abc-44b2-95d8-85cf6956bc33 +-- main@1/started |-- process: client `-- timestamp: 2015-04-28 16:50:00.379033 +-- eliot:remote_task@2,2,1/started |-- process: server `-- timestamp: 2015-04-28 16:50:00.389533 +-- http_request@2,1/started |-- process: client |-- timestamp: 2015-04-28 16:50:00.379096 |-- x: 8 `-- y: 0 +-- divide@2,2,2,1/started |-- process: server |-- timestamp: 2015-04-28 16:50:00.389674 |-- x: 8 `-- y: 0 +-- divide@2,2,2,2/failed |-- exception: exceptions.ZeroDivisionError |-- process: server |-- reason: integer division or modulo by zero `-- timestamp: 2015-04-28 16:50:00.389724 +-- eliot:remote_task@2,2,3/failed |-- exception: exceptions.ZeroDivisionError |-- process: server |-- reason: integer division or modulo by zero `-- timestamp: 2015-04-28 16:50:00.389750 +-- http_request@2,3/failed |-- exception: requests.exceptions.HTTPError |-- process: client |-- reason: 500 Server Error: INTERNAL SERVER ERROR `-- timestamp: 2015-04-28 16:50:00.396112 +-- main@3/failed |-- exception: requests.exceptions.HTTPError |-- process: client |-- reason: 500 Server Error: INTERNAL SERVER ERROR `-- timestamp: 2015-04-28 16:50:00.396181 Intrigued? Learn more at https://eliot.readthedocs.org [1]. eliottree.py is an add-on utility for formatting Eliot logs, and can be found at https://github.com/jonathanj/eliottree [2] Links: ------ [1] https://eliot.readthedocs.org [2] https://github.com/jonathanj/eliottree

2 2

[Twisted-Python] attribute error when I am trying to make one factory to send data to another
by Jessica Tsui 28 Apr '15

28 Apr '15

I am working on this messaging app and I would like to pass data from one factory to another. I have been referencing to other answers provided in other similar questions by trying to add a variable storing the data from one factory then initiate another factory and write the data to it. Unfortunately there is an error when i call the other class, and I cannot figure out why. Is there anything I can do to fix it? I have been stuck in here for quite a while this is the error: receivermessage = self.factory.app.handle_message2(data) exceptions.AttributeError: MultiEcho instance has no attribute 'factory' Here is the code: import kivyfrom kivy.app import Appfrom kivy.uix.label import Labelfrom kivy.uix.scatter import Scatterfrom kivy.uix.boxlayout import BoxLayoutfrom kivy.uix.scrollview import ScrollViewfrom kivy.uix.button import Buttonfrom kivy.graphics.vertex_instructions import Rectanglefrom kivy.graphics.context_instructions import Colorfrom kivy.graphics.instructions import Instructionfrom kivy.base import runTouchAppfrom kivy.lang import Builderimport socketfrom kivy.core.window import Windowimport pygameimport randomfrom kivy.support import install_twisted_reactor install_twisted_reactor()from twisted.internet import reactor, protocol Window.size = (550, 400) # monitoring wordlistwith open("wordlist.txt") as word_file: wordlist = list(word.strip().lower() for word in word_file) # protocols for senderclass EchoProtocol(protocol.Protocol): """This is just about the simplest possible protocol""" def dataReceived(self, data): "As soon as any data is received, write it back." sendermessage = self.factory.app.handle_message(data) if sendermessage: self.transport.write(data) #this line here is the trouble maker that caused the error MultiEcho().dataReceived(sendermessage) class EchoFactory(protocol.Factory): protocol = EchoProtocol def __init__(self, app): self.app = app # protocols for receiver class MultiEcho(protocol.Protocol): def connectionMade(self): self.factory.echoers.append(self) def dataReceived(self, data): receivermessage = self.factory.app.handle_message2(data) if receivermessage: self.transport.write(data) #this line here is the trouble maker that caused the error EchoProtocol().dataReceived(receivermessage) class MultiEchoFactory(protocol.Factory): protocol = MultiEcho def __init__(self, app): self.echoers = [] self.app = app class ServerApp(App): def build(self): self.layout = BoxLayout(orientation='vertical', spacing=10) self.label = Button(text='Censoring process begin\nBeware of keyword "umbrella"\n ', color=[1.0, 1.0, 1.0, 1.0]) self.label.color = [0.9, 0.2, 0.2, 1.0] self.upperscroll = Button(pos_hint={'x': 0, 'center_y': .5}, size_hint=(None, None)) self.scatter = Scatter() self.displaybox = Label() self.displaybox.color = [0.4, 0.9, 0.4, 1.0] reactor.listenTCP(8000, EchoFactory(self)) # for sender reactor.listenTCP(8001, MultiEchoFactory(self)) # for receiver self.layout.add_widget(self.label) self.layout.add_widget(self.scatter) self.scatter.add_widget(self.displaybox) return self.layout def handle_message(self, msg): if any(word in msg.lower() for word in wordlist): self.displaybox.color = [0.9, 0.4, 0.4, 1.0] self.displaybox.text = "content blocked" self.label.text += "Alert! Sender posts %s \n" % msg else: self.label.text += "Safe - sender posts %s \n" % msg self.displaybox.color = [0.4, 0.9, 0.4, 1.0] self.displaybox.text = "%s" % msg msg = msg return msg def handle_message2(self, msg): if any(word in msg.lower() for word in wordlist): self.label.color = [0.8, 0.8, 0.5, 1.0] self.label.text += "Alert! Receiver got %s \n" % msg else: self.label.color = [0.2, 0.2, 1.0, 1.0] self.label.text += "Safe - receiver sends %s \n" % msg msg = msg return msg if __name__ == '__main__': ServerApp().run()

4 9

[Twisted-Python] Looking for some advice debugging issue with transport.write calls
by Brian Costlow 28 Apr '15

28 Apr '15

Summary: I am having an issue writing data from an object based on twisted's LineReceiver. Calling self.transport.write from the protocol, some data makes it through, some does not. Using tcpdump, I am not seeing the missing data cross the network interface. Ubuntu LTS 14.04 Python 2.7.6 Twisted 15.0.0 The app recieves a bunch of messages from various upstream servers (not mine) and either discards—or bends, folds, spindles and muilates and sends the modified message to a downstream server. The message format is text-based, proprietary and not mine. The way it works: 1. The app puts messages, consisting of a tuple of two strings: an id and the message data, on a queue. The queue is actually a deque on the factory. 2. There's one client protocol instance for sending, it holds a connection to downstream server. The factory is a reconnecting one, but only if the connection drops. (I'm not creating a new protocol instance/connection per message) 3. There's a method on the protocol that checks for messages in the factory's queue, and sends them, one at a time. It waits for a short ack text message back from the receiving end, or times out (my app's timeout, not a network stack one) before sending the next message, if any. Code for the sending piece is below. 4. The downstream end (an app not under my control) is not getting all the messages. If my timeout check runs, and there's not been a returned ack message yet, invariably the sysadmins for the downstream system report they never got that message. def check_for_send(self): if not self.in_send_message: try: self.in_send_message = True msg = self.factory.queue.popleft() try: self.waiting_id = msg[0] log.msg("Sending Message: {0}".format(msg[0])) raw_msg = "".join([self.MSG_START, msg[1], self.MSG_END, self.delimiter]) threads.deferToThread(self._dump_raw_message, msg[0], raw_msg) self.transport.write(raw_msg) reactor.callLater(30, self.check_for_timeout, msg[0]) except: log.msg("Error sending Message: {0}".format(msg[0])) log.err() self.factory.add_message(msg) self.waiting_id = "" self.in_send_message = False except IndexError: self.in_send_message = False finally: reactor.callLater(0, self.check_for_send) else: reactor.callLater(1, self.check_for_send) Mostly looking for some advice on further debugging this, as whenever I've called self.transport.write on a lineReceiver over tcp before, it "just worked." Observations. 1. The event loop doesn't appear to be getting stuck. Messages that don't arrive downstream are interleaved with those that do. And the other part of the app that is receiving messages from upstream keeps right on trucking. 2. I am now writing copies of the messages to disk, that call is right before the transport.write call, and all the "missing" messages do end up on disk. 3. I also fire the callLater for the timeout function on each message. The disk write and the callLater both happen for missing messages. 4. I ran tcpdump on the interface for about 30 minutes and matched up with log statements. If I see my protocol timed out waiting for an ack message in my app's log, I can't find the outgoing message on the interface. 5. I am not seeing any errors in my logs.

3 6

[Twisted-Python] Weekly Bug Summary
by trac＠twistedmatrix.com 26 Apr '15

26 Apr '15

1 0

[Twisted-Python] Deprecating subproject packaging
by HawkOwl 25 Apr '15

25 Apr '15

Hi, I think that Twisted should deprecate subproject packaging -- they aren't uploaded to pip, don't support Python 3, I don't think we have docs for using them anywhere, and when included in downstream they usually only end up in confusion as various imports fail but some succeed due to only some subprojects being installed. Eventuall removing them means we have less code (and only one setup.py + setup3.py), less packages, and simplifies things. It does mean that you can't *just* get Twisted Web, but Twisted Core is so big (internet, trial, amp, python, logger, cred, protocols -- even positioning is in there) that the size difference is marginal. For example, uncompressed Twisted (without the docs) is 14MB, while Core alone is 7MB. Assuming that you want to use Twisted Web, you'll need Twisted-Web and Twisted-Names (because not using Twisted Names in web.http.Request is deprecated), which is ~9MB. The difference in tarball size is even less -- Twisted without docs being 2.2MB and Twisted Core + Web + Names being 1.4MB. Plus, I am not sure that there is "subprojects" really, anymore -- it's all just Twisted. So, I think we should: - Deprecate the subproject setup.pys starting with 15.2 and stop packaging subprojects separately as of 16.0 (15.2 in May, 15.3 in July, 15.4 in September, 15.5 in November, 16.0 in January 2016 -- so ~4 releases + 1 major release notice). - Remove the support code for subprojects in twisted.python._release + the release scripts in the 16.0 release. - Migrate Twisted's source dist to being done by "python setup.py sdist" instead of manually assembling the tarball. This means we can generate the requisite PKG-INFO metadata automatically. This should be done for the main tarball (which is uploaded to PyPI) in maybe 15.2, but not the subproject tarballs (Twisted Core + rest) which will remain the same until they are removed in 16.0. - Replace manually uploading these distributions to PyPI and use twine to automate it for 15.2, when Twisted's tarball has the needed PKG-INFO (from being generated by sdist). Twine doesn't support uploading .msis, but as we now have Python wheels on PyPI, I think we can do without uploading MSIs, and just have them on the website. Keeping the projects topfiles as they are I think is good -- although I am not sure if having a NEWS in each subproject is useful. Maybe we should just have the one NEWS file once 16.0 lands, since the other ones wouldn't be packaged, and are therefore less useful. As for downstream: - Apple includes all of Twisted, so I don't think this will affect them. - Ubuntu and Debian package the subprojects separately. I guess they would have to make dummy packages that depend on the monolithic python-twisted instead. - Fedora (so Red Hat too?) have obsoleted the subprojects and point them at python-twisted instead, so they don't package the subprojects. - FreeBSD packages the Twisted subprojects separately, so they would have to do the same as Debian to migrate. Overall, I don't think this adversely affects downstream in the long term -- it does what Red Hat did as of Fedora 21, and it'll end up being less packages overall. I think this is a good idea and am willing to put the work in to implement this. Does anyone object to this? Is there a use case for keeping the subprojects packagable that I don't know about? - Hawkie

7 12

[Twisted-Python] Using six for Python3 porting
by Adi Roiban 24 Apr '15

24 Apr '15

Hi, I feel that twisted.python.compat is slowly duplicating / reinventing an important part of the six code. Code which does not import from twisted.python.compat will have all kind of ad-hoc / in module code to handle differences between py2 and py3 and many if _PY3 conditional blocks. What do you say if we add six as dependency for twisted ? In case you don't want six as a dependency, it can be copied in twisted code base and make it available from twisted.python.six or a similar package. -- Adi Roiban

4 4

[Twisted-Python] Add support for Next Protocol Negotiation and Application Layer Protocol Negotiation
by Adi Roiban 21 Apr '15

21 Apr '15

Hi, I just want to let you know about this patch which is in work : https://twistedmatrix.com/trac/ticket/7860 I did an initial review but it would help to get more feedback for this new API. Also it depends on a new pyOpenSSL 0.15 version. Do we have a buildslave with this version? What is the process of testing / reviewing this kind of patches? Many thanks! -- Adi Roiban

3 2

[Twisted-Python] Weekly Bug Summary
by trac＠twistedmatrix.com 19 Apr '15

19 Apr '15

1 0

Re: [Twisted-Python] [Cryptography-dev] [ANN] pyOpenSSL 0.15
by Glyph 16 Apr '15

16 Apr '15

Thank you very much to Jean-Paul and Hynek for getting out this most recent release! (And thanks to Hynek for my opportunity to contribute my first patch to pyOpenSSL ;-)). -glyph > On Apr 15, 2015, at 14:10, Alex Gaynor <alex.gaynor(a)gmail.com> wrote: > > Thank you for your years of maintenance of pyOpenSSL! > > Alex > > On Wed, Apr 15, 2015 at 2:02 PM, Jean-Paul Calderone <jean-paul(a)clusterhq.com <mailto:jean-paul@clusterhq.com>> wrote: > On Tue, Apr 14, 2015 at 12:54 PM, Hynek Schlawack <hs(a)ox.cx <mailto:hs@ox.cx>> wrote: > Greetings fellow Pythoneers, > > I'm happy to announce that pyOpenSSL 0.15 is now available. > > > Congrats on getting the release out, Hynek. Thanks once again for stepping in to take over the lead role on the pyOpenSSL project. Thanks also to all of the PyCA folks at the PyCon sprints on Monday to prepare for this release. > > Jean-Paul > > > _______________________________________________ > Cryptography-dev mailing list > Cryptography-dev(a)python.org <mailto:Cryptography-dev@python.org> > https://mail.python.org/mailman/listinfo/cryptography-dev <https://mail.python.org/mailman/listinfo/cryptography-dev> > > > > > -- > "I disapprove of what you say, but I will defend to the death your right to say it." -- Evelyn Beatrice Hall (summarizing Voltaire) > "The people's good is the highest law." -- Cicero > GPG Key fingerprint: 125F 5C67 DFE9 4084 > _______________________________________________ > Cryptography-dev mailing list > Cryptography-dev(a)python.org > https://mail.python.org/mailman/listinfo/cryptography-dev

1 0

[Twisted-Python] Methods to upload and download file to a sftp server using twisted conch
by halidou SANFO 15 Apr '15

15 Apr '15

Hi everybody,I am student and need help of any one.I am working on internship and I have to implement a SFTP client usign twisted conch module.I have program thoe code to establish th ssh tunnel using password authentication. I arrive to create directory with "MakeDirectory" method of FileTransfertClient class.But my objective is to upload and download files and the FileTransfertClient class has no method [or as I konw ] to do that. Can you help me with way to do the uploading and the downloding with twisted conch? Here is my program.<code>'''Created on 8 avr. 2015 @author: hsanfo''' from sys import stdout from twisted.python.log import startLogging, err from twisted.internet import reactorfrom twisted.internet.defer import Deferred from twisted.conch.ssh.common import NSimport twisted.conch.scripts.cftp.ClientOptionsfrom twisted.conch.scripts.cftp import ClientOptionsfrom twisted.conch.ssh.filetransfer import FileTransferClientfrom twisted.conch.client.connect import connectfrom twisted.conch.client.default import SSHUserAuthClient, verifyHostKeyfrom twisted.conch.ssh.connection import SSHConnectionfrom twisted.conch.ssh.channel import SSHChannel from twisted.conch.ssh import keys, userauthfrom twisted.internet import defer class SFTPSession(SSHChannel): name = 'session' def channelOpen(self, whatever): d = self.conn.sendRequest( self, 'subsystem', NS('sftp'), wantReply=True) d.addCallbacks(self._cbSFTP) def _cbSFTP(self, result): client = FileTransferClient() client.makeConnection(self) self.dataReceived = client.dataReceived self.conn._sftp.callback(client) class SFTPConnection(SSHConnection): def serviceStarted(self): self.openChannel(SFTPSession()) class ClientUserAuth(userauth.SSHUserAuthClient): def getPassword(self, prompt = None): #normal password authentication print "PASSWORD AUTH" return defer.succeed('test') # <-- YOUR PASSWORD def getGenericAnswers(self, name, instruction, prompts): #interactive password authentication print "INTERACTIVE AUTH" response = ['']*len(prompts) for i, p in enumerate(prompts): try: if('password' in p[0].lower()): response[i] = 'test' # <-- YOUR PASSWORD except: pass #The response is always a sequence, and the length of it is always #identical to the length of prompts return defer.succeed(response) def sftp(user, host, port): options = ClientOptions() options['host'] = host options['port'] = port conn = SFTPConnection() conn._sftp = Deferred() auth = SSHUserAuthClient(user, options, conn) #auth = ClientUserAuth(user, options, conn) #connect(host, port, options, verifyHostKey, auth) connect(host, port, options, None, auth) return conn._sftp def transfer(client): d = client.makeDirectory('foobarbaz', {}) def cbDir(ignored): print 'Made directory' d.addCallback(cbDir) return d def main(): startLogging(stdout) user = 'test' host = '192.168.29.129' port = 22 d = sftp(user, host, port) d.addCallback(transfer) d.addErrback(err, "Problem with SFTP transfer") d.addCallback(lambda ignored: reactor.stop()) reactor.run() if __name__ == '__main__': main()<code> Best regards SANFO

3 4