Hello Twisted users,
David Reid and I are attempting to upgrade our production Trac instance to version 1.0.1 today, in order to address the spam and many other issues.
While we will attempt to minimize the downtime, knowing Trac, you should expect at least 6 hours of downtime beginning as soon as 20 minutes (18:00 UTC).
I was inspired by the "socat" CLI utility to write something similar called twistedcat.
Obviously it should have the "cat" word in the name... so what do you suggest?
What do people think of this?
Do you have ascii art drawings of cats can use or suggestions for me to improve the code quality?
Right now I'm cleaning the code out of main() and creating an EndpointCombiner class
to encapsulate the endpoint pipe logic (c2c, s2s, c2s).
The reason I am very interested in improving the code to assist in pipe-lining endpoints is because it can be used by more than just this commandline tool. Any Twisted app wishing to create custom protocols can do so by pipe-lining endpoints. I think the API for this could be very easy to use.
socat has an enormous feature set that I am not sufficiently motivated to duplicate here.
It was really interesting for me to read about these socat address types... there are many of them.
Twisted has some of the same types... and in some cases they are even spelled the same.
I am interested in implementing some of these socat address types in Twisted, including a TUN endpoint and endpoint parser plugin.
I've played around with the new twisted pair TUN stuff... but I was using producers and consumers to make a VPN...
and as it turns out this task should be greatly simplified by using pipelined Twisted endpoints.
I've also been toying with the idea of writing an obfsproxy Twisted endpoint which would work similar to the txtorcon endpoint in that it would launch an external command... and then perform a tcp connect. obfsproxy can be launched in external mode and has several obfuscation protocols to choose from... some of which are very sophisticated. If someone actually had a good use case for obfuscating pipelined endpoint protocols then I would love to hear about it (besides the obvious VPN use-case)!
Twisted endpoint descriptor strings are just like socat address specifications... except that in socat the client and server endpoint types are in the same name space. Therefore twistedcat needs to know if the endpoint string is a server or client endpoint. It defaults to client unless -s or --server specifies it as a server endpoint string. Or do you dare suggest that it would be better to try clientFromString and then if that fails try serverFromString?
We've already created a Tor client endpoint/parser plugin (https://github.com/david415/txsocksx)
and a Tor Hidden Service endpoint/parser plugin (https://github.com/meejah/txtorcon)...
Obviously twistedcat doesn't know anything about Tor... because it's completely endpoint agnostic.
Here's my silly twistedcat Tor experimentations:
$ ./twistedcat.py -s stdio: -s onion:80:hiddenServiceDir=/home/human/hs
After running this one liner... I checked the contents of /home/human/hs/
and found my new onion address in the "hostname" file.
I used my Tor browser to send a request... and here we see twistedcat printed it to stdout via the stdio endpoint:
GET / HTTP/1.1
User-Agent: Mozilla/5.0 (Windows NT 6.1; rv:24.0) Gecko/20100101 Firefox/24.0
Accept-Encoding: gzip, deflate
And then there is this:
$ ./twistedcat.py tor:timaq4ygg2iegci7.onion:80 -s onion:80:hiddenServiceDir=/home/human/hs
Which of course proxies requests from my new onion to timaq4ygg2iegci7.onion... the onion address of the txtordoc documentation website.
But this is crazy... um how many hops is that exactly? 14? 15?
On May 30, 2014, at 9:28 AM, Amar Takhar <verm(a)darkbeer.org> wrote:
> On 2014-05-29 09:56 -0700, Glyph wrote:
>> Thanks for this input, Dustin. It is actually _super_ useful, for me at least,
>> to learn that there is in fact a light at the end of this tunnel :).
> I'm happy to give any help that's required if you need a hand upgrading trac and
> migrating to git. I've done it quite a few times.
> Our spam problem is now completely under control thankfully. We have some
> custom plugins to help with this.
We've upgraded Trac, finally, and now that that blocker is resolved, I think it might be time to start talking about migrating to Git.
Would you mind joining us in #twisted-admin on Freenode to discuss this? The relevant folks tend to be around there during the intersection between EST and PST work hours :).
On 17 Jun, 03:56 pm, max(a)riehl.io wrote:
>I finally found the time to do some work on the ticket but have been
>unable to find out how to attach a patch so far.
>I caught some discussion about a trac change that requires new users to
>do "something" before they are able to make changes to tickets.
>Maybe you can brief me about the required steps? I'm also willing to
>wait if some documentation is on the way.
>I also noted a few problems while trying to contribute (wrong
>instructions, hanging tests, etc) and would be happy to help make this
>better, but I'm unsure where to start.
Please keep discussion on the mailing list. I didn't even see your
email until this morning.
The required steps are simple: contact a trac admin (mail the list or
join #twisted-dev on freenode), convince them you aren't a spammer, and
tell them your trac username.
I think you only need to complete the third step at this point.
I'm waiting for someone to volunteer to update some trac pages to
include this information. I looked at some templates and decided I
didn't have enough time to figure out how they work.
-----BEGIN PGP SIGNED MESSAGE-----
I'm excited to announce txtorcon 0.10.0 which adds support for
Twisted's endpoint strings.
This means that ANY Twisted program that uses endpoints can accept
"onion:" strings to bring up a hidden services easily (by launching a
new Tor instance). Typically, no code changes to the application
should be needed (just "pip install txtorcon").
"twistd" supports endpoints, so for example to serve some Web content
from ~/public_html as a hidden-serivce, we can do this (with txtorcon
twistd web --port onion:80 --path ~/public_html
Some examples of other valid "onion:" endpoint strings:
The first allows specifying existing hidden service keys and the
second says to connect to an already-running Tor instance.
Thanks to David Stainton (dawuud) for the initial pull-request (and
continued collaboration) that made this happen. There is a complete
demonstration of the power of this Fully Operational endpoint-station
You can download the release from PyPI or GitHub (or of course use
"pip install txtorcon"):
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.12 (GNU/Linux)
-----END PGP SIGNATURE-----
I have a very simple twisted service that reads a temperature sensor. A client can ask for the current temperature via JSON and as a result the service will read the temperature sensor and respond with the current temp. This works great if I run it as a python module from the command line but if I run it as a service with twistd it seems to read the temp on startup then the temp seems to be cached as it never changes. I have print calls in the getTemp method right before it calls into the hardware library, they are getting called in both cases - but the return from the call never changes in the twistd case. Any ideas?
Also, I have another service (not twisted) that runs in the background that periodically reads the temp and responds to temperature changes. This is working fine so I’m pretty sure the problem is isolated to twistd.
I had to restart postgresql , and that triggered some endless errors in my logs.
i don't have anything that can handle a suddenly dropped / resumed connection.
does anyone have a recommendation for trying to reconnect or handle this in general
from what I can tell from my logs...
When I catch an error, my cleanup code tries to
psycopg2.InterfaceError: connection already closed
My immediate thoughts are:
1. catching the correct disconnect error
2. telling the connection/pool to reconnect
this is an early announcement of Crossbar.io, a new application server based on Twisted and Autobahn.
Crossbar.io is an open-source application router that allows to build distributed systems out of
application components which are loosely coupled and communicate in (soft) real-time.
At it's core, an application router provides generic routing of both calls (for remote procedure
calls) and events (for publish & subscribe) which is realized via WAMP, an open WebSocket
based communication protocol.
Why WAMP?: http://wamp.ws/why/
Technically, Crossbar.io has a multi-process architecture with controller and worker processes,
which lets use scale up on multi-core. We also prepared things to add scale-out on multi-node.
We've got some early encouraging feedback from developers who think Crossbar.io might
get the next Django, MeteorJS or Vert.x:
A developer guest post: http://tavendo.com/blog/post/is-crossbar-the-future-of-python-web-apps/
Crossbar.io is an open-source project, and still in the very beginning. Rough edges, stuff breaks,
If you want to give it a try: https://github.com/crossbario/crossbar/wiki#quick-start
We are looking for feedback, help and contributors. It's a big shot project, and to make it
take off, we need core hackers and grow a developer community. I know, sounds bold.
Sure, and it is. In any case, I'd love to see the Python community get even more ambitious.
We can't let the J's of this world take over;)