Twisted June 2014

twisted@python.org

31 participants
29 discussions

[Twisted-Python] Effects of calling transport.writeSomeData() ?
by Mark Montague 25 Jun '14

25 Jun '14

We're writing a Twisted 14.0.0 application (on Python 2.7.7, Mac OS 10.9.3) that uses Conch as an SSH client; this is working fine. However, we have the requirement that in an advanced mode of operation for power users that the application take advantage of OpenSSH connection multiplexing over an already-established-by-the-user OpenSSH ControlMaster session (via an OpenSSH ControlPath socket) instead of using Conch. OpenSSH requires its new session command and forwarded file descriptors to be sent over the socket in a very particular way: the command must be sent first, followed by message with a '\0' byte with each forwarded file descriptor. OpenSSH ignores the '\0' for each file descriptor, extracting the file descriptors themselves from the message's ancillary data. The following will not work because none of the calls to write() send their data until control is returned to the reactor, while sendFileDescriptor() queues up the descriptors such that they get sent them with the very next data that is sent -- which wind up being the first three bytes of the command rather than the three '\0' bytes. class OpenSSHMuxProtocol( protocol.Protocol ): # built via reactor.connectUNIX() def sendCommand( self, command ): # Does not work: self.transport.write( command ) self.transport.sendFileDescriptor( sys.stdin.fileno() ) self.transport.write( '\0' ) # payload for the stdin file descriptor self.transport.sendFileDescriptor( sys.stdout.fileno() ) self.transport.write( '\0' ) # payload for the stdout file descriptor self.transport.sendFileDescriptor( sys.stdout.fileno() ) self.transport.write( '\0' ) # payload for the stderr file descriptor # ^^^ Does not work But this next solution /does/ work: from socket import SOL_SOCKET from twisted.python.sendmsg import SCM_RIGHTS, send1msg class OpenSSHMuxProtocol( protocol.Protocol ): # built via reactor.connectUNIX() def sendCommand( self, command ): self.transport.writeSomeData( command ) # data is sent over the socket immediately send1msg( self.transport.socket.fileno(), "\0", 0, [ ( SOL_SOCKET, SCM_RIGHTS, pack( 'i', sys.stdin.fileno() ) ) ] ) send1msg( self.transport.socket.fileno(), "\0", 0, [ ( SOL_SOCKET, SCM_RIGHTS, pack( 'i', sys.stdout.fileno() ) ) ] ) send1msg( self.transport.socket.fileno(), "\0", 0, [ ( SOL_SOCKET, SCM_RIGHTS, pack( 'i', sys.stderr.fileno() ) ) ] ) My questions are: Is it bad to bypass the reactor and send data directly/immediately this way using writeSomeData() and send1msg()? Note that sendCommand() actually gets called in response to a OpenSSHMuxProtocol.dataReceived() event. If bypassing the reactor this way is bad, how bad is it and what are the consequences or effects? Is there a better way to get a working solution? I think I'd need some way to guarantee that the write of the command was actually sent to the OpenSSH server before the file descriptors are forwarded -- for example, if a Deferred was used whose first callback wrote the command and whose second callback forwarded the descriptors, would a call to the reactor to actually sent the command be guaranteed between the two callbacks? Any information and/or advice is appreciated! -- Mark Montague mark(a)catseye.org

3 4

[Twisted-Python] Trac downtime
by Glyph Lefkowitz 24 Jun '14

24 Jun '14

Hello Twisted users, David Reid and I are attempting to upgrade our production Trac instance to version 1.0.1 today, in order to address the spam and many other issues. While we will attempt to minimize the downtime, knowing Trac, you should expect at least 6 hours of downtime beginning as soon as 20 minutes (18:00 UTC). -glyph

7 14

[Twisted-Python] twistedcat - twisted endpoint concatenator
by David Stainton 22 Jun '14

22 Jun '14

Greetings, I was inspired by the "socat" CLI utility to write something similar called twistedcat. https://github.com/david415/twistedcat https://github.com/david415/twistedcat/blob/master/twistedcat/twistedcat.py Obviously it should have the "cat" word in the name... so what do you suggest? What do people think of this? Do you have ascii art drawings of cats can use or suggestions for me to improve the code quality? Right now I'm cleaning the code out of main() and creating an EndpointCombiner class to encapsulate the endpoint pipe logic (c2c, s2s, c2s). The reason I am very interested in improving the code to assist in pipe-lining endpoints is because it can be used by more than just this commandline tool. Any Twisted app wishing to create custom protocols can do so by pipe-lining endpoints. I think the API for this could be very easy to use. socat has an enormous feature set that I am not sufficiently motivated to duplicate here. http://www.dest-unreach.org/socat/doc/socat.html#ADDRESS_TYPES It was really interesting for me to read about these socat address types... there are many of them. Twisted has some of the same types... and in some cases they are even spelled the same. I am interested in implementing some of these socat address types in Twisted, including a TUN endpoint and endpoint parser plugin. I've played around with the new twisted pair TUN stuff... but I was using producers and consumers to make a VPN... and as it turns out this task should be greatly simplified by using pipelined Twisted endpoints. I've also been toying with the idea of writing an obfsproxy Twisted endpoint which would work similar to the txtorcon endpoint in that it would launch an external command... and then perform a tcp connect. obfsproxy can be launched in external mode and has several obfuscation protocols to choose from... some of which are very sophisticated. If someone actually had a good use case for obfuscating pipelined endpoint protocols then I would love to hear about it (besides the obvious VPN use-case)! https://www.torproject.org/projects/obfsproxy Twisted endpoint descriptor strings are just like socat address specifications... except that in socat the client and server endpoint types are in the same name space. Therefore twistedcat needs to know if the endpoint string is a server or client endpoint. It defaults to client unless -s or --server specifies it as a server endpoint string. Or do you dare suggest that it would be better to try clientFromString and then if that fails try serverFromString? We've already created a Tor client endpoint/parser plugin (https://github.com/david415/txsocksx) and a Tor Hidden Service endpoint/parser plugin (https://github.com/meejah/txtorcon)... Obviously twistedcat doesn't know anything about Tor... because it's completely endpoint agnostic. Here's my silly twistedcat Tor experimentations: $ ./twistedcat.py -s stdio: -s onion:80:hiddenServiceDir=/home/human/hs After running this one liner... I checked the contents of /home/human/hs/ and found my new onion address in the "hostname" file. I used my Tor browser to send a request... and here we see twistedcat printed it to stdout via the stdio endpoint: GET / HTTP/1.1 Host: ibce3yvjdb3mzcrq.onion User-Agent: Mozilla/5.0 (Windows NT 6.1; rv:24.0) Gecko/20100101 Firefox/24.0 Accept: text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8 Accept-Language: en-us,en;q=0.5 Accept-Encoding: gzip, deflate Connection: keep-alive And then there is this: $ ./twistedcat.py tor:timaq4ygg2iegci7.onion:80 -s onion:80:hiddenServiceDir=/home/human/hs Which of course proxies requests from my new onion to timaq4ygg2iegci7.onion... the onion address of the txtordoc documentation website. But this is crazy... um how many hops is that exactly? 14? 15? Cheers, David

2 1

Re: [Twisted-Python] Status of trac upgrade
by Glyph 20 Jun '14

20 Jun '14

On May 30, 2014, at 9:28 AM, Amar Takhar <verm(a)darkbeer.org> wrote: > On 2014-05-29 09:56 -0700, Glyph wrote: >> >> Thanks for this input, Dustin. It is actually _super_ useful, for me at least, >> to learn that there is in fact a light at the end of this tunnel :). > > I'm happy to give any help that's required if you need a hand upgrading trac and > migrating to git. I've done it quite a few times. > > Our spam problem is now completely under control thankfully. We have some > custom plugins to help with this. Hi Amar, We've upgraded Trac, finally, and now that that blocker is resolved, I think it might be time to start talking about migrating to Git. Would you mind joining us in #twisted-admin on Freenode to discuss this? The relevant folks tend to be around there during the intersection between EST and PST work hours :). -glyph

1 0

[Twisted-Python] Status of trac upgrade
by exarkun＠twistedmatrix.com 20 Jun '14

20 Jun '14

Hello, I was just wondering what the current status of the effort to upgrade trac on twistedmatrix.com is. Thanks, Jean-Paul

8 10

Re: [Twisted-Python] Known bug with twistd log rotation
by exarkun＠twistedmatrix.com 19 Jun '14

19 Jun '14

On 17 Jun, 03:56 pm, max(a)riehl.io wrote: >Hi, > >I finally found the time to do some work on the ticket but have been >unable to find out how to attach a patch so far. >I caught some discussion about a trac change that requires new users to >do "something" before they are able to make changes to tickets. > >Maybe you can brief me about the required steps? I'm also willing to >wait if some documentation is on the way. > > >I also noted a few problems while trying to contribute (wrong >instructions, hanging tests, etc) and would be happy to help make this >better, but I'm unsure where to start. Hi Max, Please keep discussion on the mailing list. I didn't even see your email until this morning. The required steps are simple: contact a trac admin (mail the list or join #twisted-dev on freenode), convince them you aren't a spammer, and tell them your trac username. I think you only need to complete the third step at this point. I'm waiting for someone to volunteer to update some trac pages to include this information. I looked at some templates and decided I didn't have enough time to figure out how they work. Jean-Paul

1 0

[Twisted-Python] ANN: txtorcon 0.10.0
by meejah 15 Jun '14

15 Jun '14

-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1 I'm excited to announce txtorcon 0.10.0 which adds support for Twisted's endpoint strings. This means that ANY Twisted program that uses endpoints can accept "onion:" strings to bring up a hidden services easily (by launching a new Tor instance). Typically, no code changes to the application should be needed (just "pip install txtorcon"). "twistd" supports endpoints, so for example to serve some Web content from ~/public_html as a hidden-serivce, we can do this (with txtorcon installed): twistd web --port onion:80 --path ~/public_html Some examples of other valid "onion:" endpoint strings: onion:80:hiddenServiceDir=/dev/shm/hidsrv onion:80:controlPort=9050:hiddenServiceDir=/srv/hidden The first allows specifying existing hidden service keys and the second says to connect to an already-running Tor instance. Thanks to David Stainton (dawuud) for the initial pull-request (and continued collaboration) that made this happen. There is a complete demonstration of the power of this Fully Operational endpoint-station here: https://txtorcon.readthedocs.org/en/latest/howtos.html#endpoints-enable-tor… You can download the release from PyPI or GitHub (or of course use "pip install txtorcon"): https://pypi.python.org/pypi/txtorcon/0.10.0 https://github.com/meejah/txtorcon/releases/tag/v0.10.0 sha256sum reports: 910ff3216035de0a779cfc167c0545266ff1f26687b163fc4655f298aca52d74 txtorcon-0.10.0-py2-none-any.whl c93f3d0f21d53c6b4c1521fc8d9dc2c9aff4a9f60497becea207d1738fa78279 txtorcon-0.10.0.tar.gz thanks, meejah -----BEGIN PGP SIGNATURE----- Version: GnuPG v1.4.12 (GNU/Linux) iQEcBAEBAgAGBQJTnTM5AAoJEMJgKAMSgGmnB5EIAIJACHhd7PEWYfCZmB8lL1pr J4rrhUuXD1iI7zJJL/rmS/SZLL+34JoKke6iQsBRGYzU0GRK2JQkFAeMq++AMdMu QvoTg745OhnMSbZyCyS6buN/NhAVcmD3GYI9h8TK60jfUPEFn7+sDstMG4OePdgq X0QQHBuJq9XtDwEmRWFHq2Aht//7J2DUliPPjtwT3C4FhPT5pXxasvfQA4jOYxsO jTja6UE4/4TnonV5gB/chijp60cKvVxFQi86mKuNnVaKTZu+QCZFJtXkr/DbRSxv CJ+z7jJ9s8xhfQ5Nkv9pkVAreH6w9bemc/iijKlZmQGYvLBTsvISPATyWGtn1f4= =wIVZ -----END PGP SIGNATURE-----

1 0

[Twisted-Python] twistd caching method calls?
by Jim Rainville 15 Jun '14

15 Jun '14

Hi - I have a very simple twisted service that reads a temperature sensor. A client can ask for the current temperature via JSON and as a result the service will read the temperature sensor and respond with the current temp. This works great if I run it as a python module from the command line but if I run it as a service with twistd it seems to read the temp on startup then the temp seems to be cached as it never changes. I have print calls in the getTemp method right before it calls into the hardware library, they are getting called in both cases - but the return from the call never changes in the twistd case. Any ideas? Also, I have another service (not twisted) that runs in the background that periodically reads the temp and responds to temperature changes. This is working fine so I’m pretty sure the problem is isolated to twistd. Thanks. Jim

2 1

[Twisted-Python] Weekly Bug Summary
by trac＠twistedmatrix.com 15 Jun '14

15 Jun '14

1 0

[Twisted-Python] adbapi.ConnectionPool - reconnecting to postgresql
by Jonathan Vanasco 13 Jun '14

13 Jun '14

I had to restart postgresql , and that triggered some endless errors in my logs. i don't have anything that can handle a suddenly dropped / resumed connection. does anyone have a recommendation for trying to reconnect or handle this in general from what I can tell from my logs... When I catch an error, my cleanup code tries to self._connection.rollback() which raises psycopg2.InterfaceError: connection already closed My immediate thoughts are: 1. catching the correct disconnect error 2. telling the connection/pool to reconnect

5 8

2022

2021

2020

2019

2018

2017

2016

2015

2014

2013

2012

2011

2010

2009

2008

2007

2006

2005

2004

2003

2002

2001

Twisted June 2014