Twisted June 2014

twisted@python.org

31 participants
28 discussions

by Glyph Lefkowitz

Hello Twisted users, David Reid and I are attempting to upgrade our production Trac instance to version 1.0.1 today, in order to address the spam and many other issues. While we will attempt to minimize the downtime, knowing Trac, you should expect at least 6 hours of downtime beginning as soon as 20 minutes (18:00 UTC). -glyph

7 years, 5 months

[Twisted-Python] twistedcat - twisted endpoint concatenator

by David Stainton

Greetings, I was inspired by the "socat" CLI utility to write something similar called twistedcat. https://github.com/david415/twistedcat https://github.com/david415/twistedcat/blob/master/twistedcat/twistedcat.py Obviously it should have the "cat" word in the name... so what do you suggest? What do people think of this? Do you have ascii art drawings of cats can use or suggestions for me to improve the code quality? Right now I'm cleaning the code out of main() and creating an EndpointCombiner class to encapsulate the endpoint pipe logic (c2c, s2s, c2s). The reason I am very interested in improving the code to assist in pipe-lining endpoints is because it can be used by more than just this commandline tool. Any Twisted app wishing to create custom protocols can do so by pipe-lining endpoints. I think the API for this could be very easy to use. socat has an enormous feature set that I am not sufficiently motivated to duplicate here. http://www.dest-unreach.org/socat/doc/socat.html#ADDRESS_TYPES It was really interesting for me to read about these socat address types... there are many of them. Twisted has some of the same types... and in some cases they are even spelled the same. I am interested in implementing some of these socat address types in Twisted, including a TUN endpoint and endpoint parser plugin. I've played around with the new twisted pair TUN stuff... but I was using producers and consumers to make a VPN... and as it turns out this task should be greatly simplified by using pipelined Twisted endpoints. I've also been toying with the idea of writing an obfsproxy Twisted endpoint which would work similar to the txtorcon endpoint in that it would launch an external command... and then perform a tcp connect. obfsproxy can be launched in external mode and has several obfuscation protocols to choose from... some of which are very sophisticated. If someone actually had a good use case for obfuscating pipelined endpoint protocols then I would love to hear about it (besides the obvious VPN use-case)! https://www.torproject.org/projects/obfsproxy Twisted endpoint descriptor strings are just like socat address specifications... except that in socat the client and server endpoint types are in the same name space. Therefore twistedcat needs to know if the endpoint string is a server or client endpoint. It defaults to client unless -s or --server specifies it as a server endpoint string. Or do you dare suggest that it would be better to try clientFromString and then if that fails try serverFromString? We've already created a Tor client endpoint/parser plugin (https://github.com/david415/txsocksx) and a Tor Hidden Service endpoint/parser plugin (https://github.com/meejah/txtorcon)... Obviously twistedcat doesn't know anything about Tor... because it's completely endpoint agnostic. Here's my silly twistedcat Tor experimentations: $ ./twistedcat.py -s stdio: -s onion:80:hiddenServiceDir=/home/human/hs After running this one liner... I checked the contents of /home/human/hs/ and found my new onion address in the "hostname" file. I used my Tor browser to send a request... and here we see twistedcat printed it to stdout via the stdio endpoint: GET / HTTP/1.1 Host: ibce3yvjdb3mzcrq.onion User-Agent: Mozilla/5.0 (Windows NT 6.1; rv:24.0) Gecko/20100101 Firefox/24.0 Accept: text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8 Accept-Language: en-us,en;q=0.5 Accept-Encoding: gzip, deflate Connection: keep-alive And then there is this: $ ./twistedcat.py tor:timaq4ygg2iegci7.onion:80 -s onion:80:hiddenServiceDir=/home/human/hs Which of course proxies requests from my new onion to timaq4ygg2iegci7.onion... the onion address of the txtordoc documentation website. But this is crazy... um how many hops is that exactly? 14? 15? Cheers, David

7 years, 5 months

Re: [Twisted-Python] Status of trac upgrade

by Glyph

On May 30, 2014, at 9:28 AM, Amar Takhar <verm(a)darkbeer.org> wrote: > On 2014-05-29 09:56 -0700, Glyph wrote: >> >> Thanks for this input, Dustin. It is actually _super_ useful, for me at least, >> to learn that there is in fact a light at the end of this tunnel :). > > I'm happy to give any help that's required if you need a hand upgrading trac and > migrating to git. I've done it quite a few times. > > Our spam problem is now completely under control thankfully. We have some > custom plugins to help with this. Hi Amar, We've upgraded Trac, finally, and now that that blocker is resolved, I think it might be time to start talking about migrating to Git. Would you mind joining us in #twisted-admin on Freenode to discuss this? The relevant folks tend to be around there during the intersection between EST and PST work hours :). -glyph

7 years, 5 months

[Twisted-Python] Status of trac upgrade

by exarkun＠twistedmatrix.com

Hello, I was just wondering what the current status of the effort to upgrade trac on twistedmatrix.com is. Thanks, Jean-Paul

7 years, 5 months

Re: [Twisted-Python] Known bug with twistd log rotation

by exarkun＠twistedmatrix.com

On 17 Jun, 03:56 pm, max(a)riehl.io wrote: >Hi, > >I finally found the time to do some work on the ticket but have been >unable to find out how to attach a patch so far. >I caught some discussion about a trac change that requires new users to >do "something" before they are able to make changes to tickets. > >Maybe you can brief me about the required steps? I'm also willing to >wait if some documentation is on the way. > > >I also noted a few problems while trying to contribute (wrong >instructions, hanging tests, etc) and would be happy to help make this >better, but I'm unsure where to start. Hi Max, Please keep discussion on the mailing list. I didn't even see your email until this morning. The required steps are simple: contact a trac admin (mail the list or join #twisted-dev on freenode), convince them you aren't a spammer, and tell them your trac username. I think you only need to complete the third step at this point. I'm waiting for someone to volunteer to update some trac pages to include this information. I looked at some templates and decided I didn't have enough time to figure out how they work. Jean-Paul

7 years, 5 months

[Twisted-Python] ANN: txtorcon 0.10.0

by meejah

-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1 I'm excited to announce txtorcon 0.10.0 which adds support for Twisted's endpoint strings. This means that ANY Twisted program that uses endpoints can accept "onion:" strings to bring up a hidden services easily (by launching a new Tor instance). Typically, no code changes to the application should be needed (just "pip install txtorcon"). "twistd" supports endpoints, so for example to serve some Web content from ~/public_html as a hidden-serivce, we can do this (with txtorcon installed): twistd web --port onion:80 --path ~/public_html Some examples of other valid "onion:" endpoint strings: onion:80:hiddenServiceDir=/dev/shm/hidsrv onion:80:controlPort=9050:hiddenServiceDir=/srv/hidden The first allows specifying existing hidden service keys and the second says to connect to an already-running Tor instance. Thanks to David Stainton (dawuud) for the initial pull-request (and continued collaboration) that made this happen. There is a complete demonstration of the power of this Fully Operational endpoint-station here: https://txtorcon.readthedocs.org/en/latest/howtos.html#endpoints-enable-tor… You can download the release from PyPI or GitHub (or of course use "pip install txtorcon"): https://pypi.python.org/pypi/txtorcon/0.10.0 https://github.com/meejah/txtorcon/releases/tag/v0.10.0 sha256sum reports: 910ff3216035de0a779cfc167c0545266ff1f26687b163fc4655f298aca52d74 txtorcon-0.10.0-py2-none-any.whl c93f3d0f21d53c6b4c1521fc8d9dc2c9aff4a9f60497becea207d1738fa78279 txtorcon-0.10.0.tar.gz thanks, meejah -----BEGIN PGP SIGNATURE----- Version: GnuPG v1.4.12 (GNU/Linux) iQEcBAEBAgAGBQJTnTM5AAoJEMJgKAMSgGmnB5EIAIJACHhd7PEWYfCZmB8lL1pr J4rrhUuXD1iI7zJJL/rmS/SZLL+34JoKke6iQsBRGYzU0GRK2JQkFAeMq++AMdMu QvoTg745OhnMSbZyCyS6buN/NhAVcmD3GYI9h8TK60jfUPEFn7+sDstMG4OePdgq X0QQHBuJq9XtDwEmRWFHq2Aht//7J2DUliPPjtwT3C4FhPT5pXxasvfQA4jOYxsO jTja6UE4/4TnonV5gB/chijp60cKvVxFQi86mKuNnVaKTZu+QCZFJtXkr/DbRSxv CJ+z7jJ9s8xhfQ5Nkv9pkVAreH6w9bemc/iijKlZmQGYvLBTsvISPATyWGtn1f4= =wIVZ -----END PGP SIGNATURE-----

7 years, 5 months

[Twisted-Python] twistd caching method calls?

by Jim Rainville

Hi - I have a very simple twisted service that reads a temperature sensor. A client can ask for the current temperature via JSON and as a result the service will read the temperature sensor and respond with the current temp. This works great if I run it as a python module from the command line but if I run it as a service with twistd it seems to read the temp on startup then the temp seems to be cached as it never changes. I have print calls in the getTemp method right before it calls into the hardware library, they are getting called in both cases - but the return from the call never changes in the twistd case. Any ideas? Also, I have another service (not twisted) that runs in the background that periodically reads the temp and responds to temperature changes. This is working fine so I’m pretty sure the problem is isolated to twistd. Thanks. Jim

7 years, 5 months

[Twisted-Python] Weekly Bug Summary

by trac＠twistedmatrix.com

7 years, 5 months

[Twisted-Python] adbapi.ConnectionPool - reconnecting to postgresql

by Jonathan Vanasco

I had to restart postgresql , and that triggered some endless errors in my logs. i don't have anything that can handle a suddenly dropped / resumed connection. does anyone have a recommendation for trying to reconnect or handle this in general from what I can tell from my logs... When I catch an error, my cleanup code tries to self._connection.rollback() which raises psycopg2.InterfaceError: connection already closed My immediate thoughts are: 1. catching the correct disconnect error 2. telling the connection/pool to reconnect

7 years, 5 months

[Twisted-Python] Announcing Crossbar.io

by Tobias Oberstein

Hi, this is an early announcement of Crossbar.io, a new application server based on Twisted and Autobahn. Crossbar.io is an open-source application router that allows to build distributed systems out of application components which are loosely coupled and communicate in (soft) real-time. README: https://github.com/crossbario/crossbar#crossbario At it's core, an application router provides generic routing of both calls (for remote procedure calls) and events (for publish & subscribe) which is realized via WAMP, an open WebSocket based communication protocol. Why WAMP?: http://wamp.ws/why/ Technically, Crossbar.io has a multi-process architecture with controller and worker processes, which lets use scale up on multi-core. We also prepared things to add scale-out on multi-node. We've got some early encouraging feedback from developers who think Crossbar.io might get the next Django, MeteorJS or Vert.x: A developer guest post: http://tavendo.com/blog/post/is-crossbar-the-future-of-python-web-apps/ Crossbar.io is an open-source project, and still in the very beginning. Rough edges, stuff breaks, moving fast. If you want to give it a try: https://github.com/crossbario/crossbar/wiki#quick-start We are looking for feedback, help and contributors. It's a big shot project, and to make it take off, we need core hackers and grow a developer community. I know, sounds bold. Sure, and it is. In any case, I'd love to see the Python community get even more ambitious. We can't let the J's of this world take over;) Cheers, /Tobias

7 years, 5 months

2021

2020

2019

2018

2017

2016

2015

2014

2013

2012

2011

2010

2009

2008

2007

2006

2005

2004

2003

2002

2001

Twisted June 2014