[Twisted-Python] tor hidden service endpoint parser returns a deferred
![](https://secure.gravatar.com/avatar/0c9380641256af855d659dce5ba9a41e.jpg?s=120&d=mm&r=g)
Greetings, Today I wrote an endpoint parser for Meejah's txtorcon Tor Hidden Service endpoint. Now that we have foolscap ported to twisted endpoints (branch needs review but I should probably rebase first) we'd like to be able to simply specify an "onion" endpoint descriptor in Tahoe-LAFS configuration... that way foolscap receives the descriptor and uses serverFromString to generate the endpoint object. This will work fine for the txsocksx tor client endpoint parser I wrote... However the txtorcon Tor Hidden Service endpoint setup requires a deferred to fire once the tor process is started... This means that the endpoint parser needs to return a deferred. But this breaks the interface!... meaning that foolscap or any other api using this onion endpoint parser will have to special case the situation where serverFromString returns a deferred. Terrible! Or do you think this is not so terrible? I am curious to know your thoughts and if any of you have alternate solutions... since I am obviously doing it wrong. Here's my txtorcon onion endpoint parser: https://github.com/david415/txtorcon/blob/endpoint_parser_plugin/examples/hi... And here's some example code that uses it: https://github.com/david415/txtorcon/blob/endpoint_parser_plugin/examples/la... By the way (just in case you don't know)... you can test this parser by placing it in the twisted/plugins directory... here's how I do it: cd projects/virtualenv-1.11.1/ ./virtualenv.py ~/virtenv-endpoints-test . ~/virtenv-endpoints-test/bin/activate usewithtor pip install twisted cd ~/projects git clone https://github.com/david415/txtorcon.git # defaults to my endpoint_parser_plugin branch cd txtorcon usewithtor python setup.py install cd examples # CWD is now /home/human/projects/txtorcon/examples cp hiddenservice_endpoint_parser.py /home/human/virtenv-endpoints-test/lib/python2.7/site-packages/twisted/plugins # and then run the example code that uses this endpoint parser python launch_tor_endpoint.py Cheers, David
![](https://secure.gravatar.com/avatar/869963bdf99b541c9f0bbfb04b0320f1.jpg?s=120&d=mm&r=g)
On 2 May 2014 15:59, David Stainton <dstainton415@gmail.com> wrote:
I think you're approaching this from the wrong angle; instead of starting the tor process during parsing of the endpoint, I think it would make more sense to start the tor process when the endpoint is started. -- mithrandi, i Ainil en-Balandor, a faer Ambar
![](https://secure.gravatar.com/avatar/0c9380641256af855d659dce5ba9a41e.jpg?s=120&d=mm&r=g)
I get what you are saying... but let's back up a second; foolscap (in my branch https://github.com/david415/foolscap/tree/endpoint_descriptors_server2) uses clientFromString and serverFromString to translate twisted endpoint descriptors into endpoint objects... AND foolscap doesn't know any details about how the endpoint wire protocols work... and we'd like to keep it that way. That's really the point of this exercise. The calling party using foolscap... in this case Tahoe-LAFS also doesn't know anything about any wire protocols. It simply receives twisted endpoint descriptors from the user and passes them to foolscap. On Fri, May 2, 2014 at 2:20 PM, Tristan Seligmann <mithrandi@mithrandi.net> wrote:
![](https://secure.gravatar.com/avatar/607cfd4a5b41fe6c886c978128b9c03e.jpg?s=120&d=mm&r=g)
On 02:53 pm, dstainton415@gmail.com wrote:
But Foolscap does know about *endpoints*. It knows to call `listen` or `connect` on them. Both of those methods return a `Deferred`. Any setup work you need to do you should be able to do in the implementation of those methods and however long it takes is how long it takes - just don't fire the `Deferred` until it is ready. Jean-Paul
![](https://secure.gravatar.com/avatar/0c9380641256af855d659dce5ba9a41e.jpg?s=120&d=mm&r=g)
Ah ha! Of course it is so obvious now that you point it out to me. The onion endpoint parser can do this setup work in it's `listen` method which returns a `Deferred`. Thanks for your observation/advice! Cheers! David On Fri, May 2, 2014 at 3:01 PM, <exarkun@twistedmatrix.com> wrote:
![](https://secure.gravatar.com/avatar/a93db92c60e9d5435d204407264457f2.jpg?s=120&d=mm&r=g)
David Stainton <dstainton415@gmail.com> writes:
Cool! Thanks for doing this. Can I pull it into txtorcon when it's finished? Also (more from the Tor end), someone specifying a service will also want to specify the keys for that service. Following the 'ssl' stuff, I guess something like "onion:80:hiddenServiceDir=/dev/shm/foo" might be appropriate? Or perhaps even a way to specify the private key directly? ...and to bikeshed a bit, what about "anontcp:" as a prefix? Then for client-side strings, it could look similar (e.g. "anontcp:encrypted.google.com:443") which could in worst-case launch a new tor instance, wait for it to bootstrap etc in connect(). Perhaps "looking similar" is a non-feature, though :) Another point for this is that "someday" tor will have UDP support... thanks, meejah
![](https://secure.gravatar.com/avatar/e1554622707bedd9202884900430b838.jpg?s=120&d=mm&r=g)
On May 2, 2014, at 1:02 PM, meejah <meejah@meejah.ca> wrote:
Another point for this is that "someday" tor will have UDP support...
There's a reason that the existing interfaces all have "stream" in their names :-). The main reason we didn't do "datagram" at the same time is that there's a pretty clear client/server division in all stream endpoints (one side has to initiate, the other side has to respond) but given the possibilities for multicast / fan-out / fan-in with datagram sockets, "just do what UDP does" didn't seem obviously correct. I'd love it if someone with an interest would make some datagram endpoint interfaces though. -glyph
![](https://secure.gravatar.com/avatar/0c9380641256af855d659dce5ba9a41e.jpg?s=120&d=mm&r=g)
Ah yeah... twisted endpoints that support udp... Perhaps when Yawning Angel finishes Lightweight Obfuscated Datagram Protocol (LODP) there will be interesting things we can do with this... and a more immediate perceived need for twisted datagram endpoint interfaces. On Fri, May 2, 2014 at 9:16 PM, Glyph Lefkowitz <glyph@twistedmatrix.com> wrote:
![](https://secure.gravatar.com/avatar/0c9380641256af855d659dce5ba9a41e.jpg?s=120&d=mm&r=g)
Yes of course! I have every intention of helping out the Tor project as much as I can... I want as many developers to benefit from this as possible so that many more applications can and will utilize the tor network... for anonymity, censorship resistance and great good.
Yes... precisely what I was thinking. For now specifying a directory containing keys might be good enough... but like you say... I should also provide a way to specify the private key directly. Sounds good!
...and to bikeshed a bit, what about "anontcp:" as a prefix? Then for
Yes I agree this is essentially in the bikeshed category of discussions. Let's try to reach consensus. Perhaps off the tahoe-lafs and twisted mailing lists... and perhaps on the tor-dev list. I look forward to collaborating with you (and anyone else who wants to help out).
Yeah... earlier today I also got the naive client side tor endpoint parser working. It uses txsocksx of course. I briefly chatted with the txsocksx author _habnabit on #twisted about possible designs for the client side tor endpoints. Basically we seem to all be on the same page.... in that there may well be a need for multiple types of client side endpoints; We can have an endpoint that has internal logic to retry the tor connection on various ports; 9050, 9150... and another tor client side endpoint that launches it's own tor process (using txtorcon of course) and then uses txsocksx to establish the socks connection to tor. Or maybe it'll be the same endpoint that behaves according to the arguments it's passed.
cheers, david
![](https://secure.gravatar.com/avatar/e1554622707bedd9202884900430b838.jpg?s=120&d=mm&r=g)
On May 2, 2014, at 6:59 AM, David Stainton <dstainton415@gmail.com> wrote:
Today I wrote an endpoint parser for Meejah's txtorcon Tor Hidden Service endpoint.
This is fantastic. Basically this is _exactly_ the use-case that we had in mind when designing the endpoint parser plugin APIs. Thanks so much for doing it. (I hope you'll contribute it to txtorcon when you're done...) -glyph
![](https://secure.gravatar.com/avatar/0c9380641256af855d659dce5ba9a41e.jpg?s=120&d=mm&r=g)
Thanks. Sure thing. Yes I have every intention of contributing endpoint parsers to txtorcon and txsocksx for use with tor. Both of these could make it much easier for developers to make their applications tor friendly. The endpoints/endpoint parsers api is brilliant! When the tor client/server endpoint parsers are complete then Tahoe-LAFS for instance can simply use these endpoints without any code changes at all. Of course I'm not saying that is all that is required for Tahoe-LAFS to be "tor friendly"... we have detailed other considerations in trac tickets: https://tahoe-lafs.org/trac/tahoe-lafs/ticket/517 http://foolscap.lothar.com/trac/ticket/203 I'm starting to think about the deployment issues... Perhaps someone on the list knows of the "correct" setup.py (distutils/setuputils?) procedure to deploy twisted plugins? I shall have to research this soon. For now my hacky development/testing flow works OK. On Fri, May 2, 2014 at 9:18 PM, Glyph Lefkowitz <glyph@twistedmatrix.com> wrote:
![](https://secure.gravatar.com/avatar/e1554622707bedd9202884900430b838.jpg?s=120&d=mm&r=g)
On May 2, 2014, at 5:45 PM, David Stainton <dstainton415@gmail.com> wrote:
This stack overflow question still hasn't gotten a great answer: <https://stackoverflow.com/questions/7275295/how-do-i-write-a-setup-py-for-a-...> If you figure something out, please post it there. However, assuming a post-deluvian version of pip, this simple setup appears to work OK for me: <https://github.com/glyph/txsni/blob/master/setup.py#L16> and I haven't found a setup that breaks it yet. -glyph
![](https://secure.gravatar.com/avatar/0c9380641256af855d659dce5ba9a41e.jpg?s=120&d=mm&r=g)
OK... I decided that txtorcon's endpoint constructor should only take string arguments besides the reactor arg... because serverFromString passes only str args to the endpoint constructor. I wrote the endpoint parser to be as simple as possible. The endpoint's `listen` method now handles the tor configuration and launch logic... and returns a deferred which will fire with an IListenPort. The pip method definitely works... my code works but prints traceback with this error: "Unexpected error while writing cache file" Do you as well? Here's my endpoint parser plugin: https://github.com/david415/txtorcon/blob/endpoint_parser_plugin-rewrite3/tw... Here's how I install it: cd ~/projects/virtualenv-1.11.5 ./virtualenv.py ~/virtenv-txtorcon . ~/virtenv-txtorcon/bin/activate usewithtor pip install twisted cd ~/projects git clone https://github.com/david415/txtorcon.git cd txtorcon git checkout endpoint_parser_plugin-rewrite3 python setup.py install Running the example code: cd examples python launch_tor_endpoint.py Unexpected error while writing cache file Traceback (most recent call last): File "/home/human/virtenv-txtorcon/local/lib/python2.7/site-packages/twisted/internet/endpoints.py", line 1465, in serverFromString return _serverFromStringLegacy(reactor, description, _NO_DEFAULT) File "/home/human/virtenv-txtorcon/local/lib/python2.7/site-packages/twisted/internet/endpoints.py", line 1393, in _serverFromStringLegacy nameOrPlugin, args, kw = _parseServer(description, None, default) File "/home/human/virtenv-txtorcon/local/lib/python2.7/site-packages/twisted/internet/endpoints.py", line 1380, in _parseServer for plugin in getPlugins(IStreamServerEndpointStringParser): File "/home/human/virtenv-txtorcon/local/lib/python2.7/site-packages/twisted/plugin.py", line 209, in getPlugins allDropins = getCache(package) --- <exception caught here> --- File "/home/human/virtenv-txtorcon/local/lib/python2.7/site-packages/twisted/plugin.py", line 181, in getCache dropinPath.setContent(pickle.dumps(dropinDotCache)) exceptions.AttributeError: 'ZipPath' object has no attribute 'setContent' I have set up a hidden service, advertised at: <<class 'twisted.internet.tcp.Port'> of twisted.web.server.Site on 55141> http://mqrcnytlnh4xynmh.onion:80 locally listening on IPv4Address(TCP, '127.0.0.1', 55141) and txtorcon's setup.py output: python setup.py install WARNING: not using PyPi over SSH! running install running bdist_egg running egg_info writing requirements to txtorcon.egg-info/requires.txt writing txtorcon.egg-info/PKG-INFO writing top-level names to txtorcon.egg-info/top_level.txt writing dependency_links to txtorcon.egg-info/dependency_links.txt package init file 'twisted/plugins/__init__.py' not found (or not a regular file) reading manifest file 'txtorcon.egg-info/SOURCES.txt' reading manifest template 'MANIFEST.in' warning: no files found matching 'scripts/*.py' warning: no previously-included files found matching 'docs/_static/*~' warning: no previously-included files found matching 'examples/*~' writing manifest file 'txtorcon.egg-info/SOURCES.txt' installing library code to build/bdist.linux-x86_64/egg running install_lib running build_py creating build/bdist.linux-x86_64/egg creating build/bdist.linux-x86_64/egg/twisted creating build/bdist.linux-x86_64/egg/twisted/plugins copying build/lib.linux-x86_64-2.7/twisted/plugins/txtorcon_endpoint_parser.py -> build/bdist.linux-x86_64/egg/twisted/plugins creating build/bdist.linux-x86_64/egg/txtorcon copying build/lib.linux-x86_64-2.7/txtorcon/stream.py -> build/bdist.linux-x86_64/egg/txtorcon copying build/lib.linux-x86_64-2.7/txtorcon/log.py -> build/bdist.linux-x86_64/egg/txtorcon copying build/lib.linux-x86_64-2.7/txtorcon/torconfig.py -> build/bdist.linux-x86_64/egg/txtorcon copying build/lib.linux-x86_64-2.7/txtorcon/router.py -> build/bdist.linux-x86_64/egg/txtorcon copying build/lib.linux-x86_64-2.7/txtorcon/torcontrolprotocol.py -> build/bdist.linux-x86_64/egg/txtorcon copying build/lib.linux-x86_64-2.7/txtorcon/circuit.py -> build/bdist.linux-x86_64/egg/txtorcon copying build/lib.linux-x86_64-2.7/txtorcon/addrmap.py -> build/bdist.linux-x86_64/egg/txtorcon copying build/lib.linux-x86_64-2.7/txtorcon/interface.py -> build/bdist.linux-x86_64/egg/txtorcon copying build/lib.linux-x86_64-2.7/txtorcon/spaghetti.py -> build/bdist.linux-x86_64/egg/txtorcon copying build/lib.linux-x86_64-2.7/txtorcon/torstate.py -> build/bdist.linux-x86_64/egg/txtorcon copying build/lib.linux-x86_64-2.7/txtorcon/torinfo.py -> build/bdist.linux-x86_64/egg/txtorcon copying build/lib.linux-x86_64-2.7/txtorcon/__init__.py -> build/bdist.linux-x86_64/egg/txtorcon copying build/lib.linux-x86_64-2.7/txtorcon/util.py -> build/bdist.linux-x86_64/egg/txtorcon byte-compiling build/bdist.linux-x86_64/egg/twisted/plugins/txtorcon_endpoint_parser.py to txtorcon_endpoint_parser.pyc byte-compiling build/bdist.linux-x86_64/egg/txtorcon/stream.py to stream.pyc byte-compiling build/bdist.linux-x86_64/egg/txtorcon/log.py to log.pyc byte-compiling build/bdist.linux-x86_64/egg/txtorcon/torconfig.py to torconfig.pyc byte-compiling build/bdist.linux-x86_64/egg/txtorcon/router.py to router.pyc byte-compiling build/bdist.linux-x86_64/egg/txtorcon/torcontrolprotocol.py to torcontrolprotocol.pyc byte-compiling build/bdist.linux-x86_64/egg/txtorcon/circuit.py to circuit.pyc byte-compiling build/bdist.linux-x86_64/egg/txtorcon/addrmap.py to addrmap.pyc byte-compiling build/bdist.linux-x86_64/egg/txtorcon/interface.py to interface.pyc byte-compiling build/bdist.linux-x86_64/egg/txtorcon/spaghetti.py to spaghetti.pyc byte-compiling build/bdist.linux-x86_64/egg/txtorcon/torstate.py to torstate.pyc byte-compiling build/bdist.linux-x86_64/egg/txtorcon/torinfo.py to torinfo.pyc byte-compiling build/bdist.linux-x86_64/egg/txtorcon/__init__.py to __init__.pyc byte-compiling build/bdist.linux-x86_64/egg/txtorcon/util.py to util.pyc installing package data to build/bdist.linux-x86_64/egg running install_data creating build/bdist.linux-x86_64/egg/share creating build/bdist.linux-x86_64/egg/share/txtorcon copying INSTALL -> build/bdist.linux-x86_64/egg/share/txtorcon copying README.rst -> build/bdist.linux-x86_64/egg/share/txtorcon copying TODO -> build/bdist.linux-x86_64/egg/share/txtorcon copying meejah.asc -> build/bdist.linux-x86_64/egg/share/txtorcon copying docs/apilinks_sphinxext.py -> build/bdist.linux-x86_64/egg/share/txtorcon copying docs/conf.py -> build/bdist.linux-x86_64/egg/share/txtorcon copying docs/Makefile -> build/bdist.linux-x86_64/egg/share/txtorcon copying docs/txtorcon-interface.rst -> build/bdist.linux-x86_64/egg/share/txtorcon copying docs/txtorcon-launching.rst -> build/bdist.linux-x86_64/egg/share/txtorcon copying docs/txtorcon-state.rst -> build/bdist.linux-x86_64/egg/share/txtorcon copying docs/txtorcon.rst -> build/bdist.linux-x86_64/egg/share/txtorcon copying docs/txtorcon-util.rst -> build/bdist.linux-x86_64/egg/share/txtorcon copying docs/examples.rst -> build/bdist.linux-x86_64/egg/share/txtorcon copying docs/releases.rst -> build/bdist.linux-x86_64/egg/share/txtorcon copying docs/walkthrough.rst -> build/bdist.linux-x86_64/egg/share/txtorcon copying docs/introduction.rst -> build/bdist.linux-x86_64/egg/share/txtorcon copying docs/txtorcon-config.rst -> build/bdist.linux-x86_64/egg/share/txtorcon copying docs/index.rst -> build/bdist.linux-x86_64/egg/share/txtorcon copying docs/txtorcon-protocol.rst -> build/bdist.linux-x86_64/egg/share/txtorcon copying docs/_static/logo.png -> build/bdist.linux-x86_64/egg/share/txtorcon copying docs/_static/haiku.css -> build/bdist.linux-x86_64/egg/share/txtorcon copying docs/_static/avatar.png -> build/bdist.linux-x86_64/egg/share/txtorcon creating build/bdist.linux-x86_64/egg/share/txtorcon/examples copying examples/monitor.py -> build/bdist.linux-x86_64/egg/share/txtorcon/examples copying examples/launch_tor.py -> build/bdist.linux-x86_64/egg/share/txtorcon/examples copying examples/webui_server.py -> build/bdist.linux-x86_64/egg/share/txtorcon/examples copying examples/stream_circuit_logger.py -> build/bdist.linux-x86_64/egg/share/txtorcon/examples copying examples/launch_tor_with_simplehttpd.py -> build/bdist.linux-x86_64/egg/share/txtorcon/examples copying examples/launch_tor_endpoint.py -> build/bdist.linux-x86_64/egg/share/txtorcon/examples copying examples/circuit_for_next_stream.py -> build/bdist.linux-x86_64/egg/share/txtorcon/examples copying examples/dump_config.py -> build/bdist.linux-x86_64/egg/share/txtorcon/examples copying examples/schedule_bandwidth.py -> build/bdist.linux-x86_64/egg/share/txtorcon/examples copying examples/disallow_streams_by_port.py -> build/bdist.linux-x86_64/egg/share/txtorcon/examples copying examples/attach_streams_by_country.py -> build/bdist.linux-x86_64/egg/share/txtorcon/examples copying examples/launch_tor_with_hiddenservice.py -> build/bdist.linux-x86_64/egg/share/txtorcon/examples copying examples/circuit_failure_rates.py -> build/bdist.linux-x86_64/egg/share/txtorcon/examples copying examples/tor_info.py -> build/bdist.linux-x86_64/egg/share/txtorcon/examples creating build/bdist.linux-x86_64/egg/EGG-INFO copying txtorcon.egg-info/PKG-INFO -> build/bdist.linux-x86_64/egg/EGG-INFO copying txtorcon.egg-info/SOURCES.txt -> build/bdist.linux-x86_64/egg/EGG-INFO copying txtorcon.egg-info/dependency_links.txt -> build/bdist.linux-x86_64/egg/EGG-INFO copying txtorcon.egg-info/requires.txt -> build/bdist.linux-x86_64/egg/EGG-INFO copying txtorcon.egg-info/top_level.txt -> build/bdist.linux-x86_64/egg/EGG-INFO zip_safe flag not set; analyzing archive contents... creating 'dist/txtorcon-0.9.2-py2.7.egg' and adding 'build/bdist.linux-x86_64/egg' to it removing 'build/bdist.linux-x86_64/egg' (and everything under it) Processing txtorcon-0.9.2-py2.7.egg Copying txtorcon-0.9.2-py2.7.egg to /home/human/virtenv-txtorcon/lib/python2.7/site-packages Adding txtorcon 0.9.2 to easy-install.pth file Installed /home/human/virtenv-txtorcon/lib/python2.7/site-packages/txtorcon-0.9.2-py2.7.egg Processing dependencies for txtorcon==0.9.2 Searching for zope.interface==4.1.1 Best match: zope.interface 4.1.1 Adding zope.interface 4.1.1 to easy-install.pth file Using /home/human/virtenv-txtorcon/lib/python2.7/site-packages Searching for Twisted==13.2.0 Best match: Twisted 13.2.0 Adding Twisted 13.2.0 to easy-install.pth file Using /home/human/virtenv-txtorcon/lib/python2.7/site-packages Searching for setuptools==3.4.4 Best match: setuptools 3.4.4 Adding setuptools 3.4.4 to easy-install.pth file Installing easy_install-3.4 script to /home/human/virtenv-txtorcon/bin Installing easy_install script to /home/human/virtenv-txtorcon/bin Using /home/human/virtenv-txtorcon/lib/python2.7/site-packages Finished processing dependencies for txtorcon==0.9.2
![](https://secure.gravatar.com/avatar/e1554622707bedd9202884900430b838.jpg?s=120&d=mm&r=g)
On May 5, 2014, at 6:52 PM, David Stainton <dstainton415@gmail.com> wrote:
I strongly suggest that this is wrong. The endpoint constructor should do _construction_. The endpoint parser should do _parsing_. The task of parsing is of taking strings and producing meaningful values. You may want to have the endpoint parser actually live in a separate method in txtorcon rather than putting it into the plugin itself, and leave the plugin simply the task of plugging in, but the parsing method should not be called __init__. (Also, don't use asserts. This is probably indicative of a weakness in the parsing API we've provided, but you should likely raise something like UsageError.)
Great.
What exactly is 'usewithtor'?
usewithtor pip install twisted
This traceback is actually harmless, if annoying. It's trying to generate the plugin cache, which is explained here: <https://twistedmatrix.com/documents/current/core/howto/plugin.html#auto3>
running bdist_egg ^ This right here is your problem.
There are a couple of solutions, in order of increasing complexity: Don't build eggs. Why are you building eggs? Eggs are unfortunately a huge problem magnet. I don't think 'pip install' builds eggs; I don't manually run setup.py anywhere any more because I assume pip understands these concerns better than I do. Set zip_safe in your setup.py to False, which will turn your egg into a directory rather than a directory. Figure out a way to hook the _build_ step specifically so you can run getPlugins() to generate the dropin cache before zipping up your zip file. Twisted plugins actually do work within zip files, but nobody has bothered working out the setuptools incantation to get them to generate at the appropriate time for building an egg. This is extra tricky because you only want to generate this dropin.cache file if the dropin is going to live inside an egg; otherwise, it would conflict with the communal dropin.cache file generated at install time. This option will be the most confusing and frustrating for you, but I hope you choose it, because it would be GREAT to have a Right Way™ to address this problem in a Twisted plugin project's setup.py :).
zip_safe flag not set; analyzing archive contents...
It's probably best to set zip_safe one way or the other though, even if you want to set it to 'true'. Letting setuptools guess like this will, unsurprisingly, get you unpredictable results. -g
![](https://secure.gravatar.com/avatar/0c9380641256af855d659dce5ba9a41e.jpg?s=120&d=mm&r=g)
Ah yeah I see what you mean. OK... I changed it: https://github.com/david415/txtorcon/commit/1e96d550c40bef1be1b45c3c975c0da0...
Done.
OK... I changed that part too.
What exactly is 'usewithtor'?
usewithtor pip install twisted
It's torsocks. It performs an LD_PRELOAD hack to force programs to use tor. Here's some torsocks info: https://trac.torproject.org/projects/tor/wiki/doc/torsocks https://gitweb.torproject.org/torsocks.git https://lists.torproject.org/pipermail/tor-dev/2014-April/006649.html Torsocks is useful... but a terrible hack... and this reminds me: I'm hoping that the effort to make Tahoe-LAFS more Tor-friendly will result in well documented solutions. I would hope that this would make it easier for other developers to make their twisted apps more Tor-friendly as well. I do not think we have all the solution details worked out yet... We'd like to stop using torsocks with Tahoe-LAFS and instead use twisted endpoints... which is why I ported the Foolscap client side to use twisted endpoints. (str4d already ported the Foolscap server side to use endpoints) https://tahoe-lafs.org/trac/tahoe-lafs/ticket/517 This trac ticket discusses various ideas to make Tahoe more Tor friendly... including this endpoint related idea that I've been thinking about lately in trac comment 16 (https://tahoe-lafs.org/trac/tahoe-lafs/ticket/517#comment:16) by leif : """My current thought is that Tahoe should have an "always_use_tor" option which causes all furls to be rewritten as tor: endpoints before being passed to foolscap. If this option is not enabled, tor connections will be made if a tor endpoint is used, but not otherwise. So, users of mixed tor/non-tor grids can leave it disabled if they prefer faster connections to non-tor nodes.""" This is necessary for a tahoe client wanting to use a public grid over tor... because the grid's introducer node is going to announce tcp endpoints not tor endpoints. However I was thinking that the "only_tor" mode for tahoe (mentioned here https://tahoe-lafs.org/trac/tahoe-lafs/ticket/517#comment:15 ) could be implemented in tahoe... it could filter endpoint descriptors by calling clientFromString or serverFromString and then checking the object type.... or is this a terrible way to do it? I'm not sure. The thing is... tahoe wouldn't normally know anything about endpoints because it uses Foolscap which could use endpoints (it does in my branch https://github.com/david415/foolscap/tree/endpoint_descriptors_server2).
Ah OK yeah it works just fine when installed with pip.
OK... I'll try to get it working that way... but for now a pip install seems to work fine.
![](https://secure.gravatar.com/avatar/a93db92c60e9d5435d204407264457f2.jpg?s=120&d=mm&r=g)
As a followup to all this, I have just merged a branch with a refactored endpoint parser. tomprince (I believe) on #twisted suggested some SSH endpoint code to look at, which inspired the @classmethod style way of getting a TCPHiddenServiceEndpoint instance configured differently. Nearly all the code is in endpoints.py if anyone is interested, here: https://github.com/meejah/txtorcon/blob/master/txtorcon/endpoints.py#L117 If you do look, I'd certainly appreciate any comments. (The tests are in test/test_endpoints.py) Anyway, one really cool thing now that txtorcon supports endpoints is that you can install it, and then easily make Twisted Web serve some content as a hidden service (launching a Tor instance first, etc): twistd -n web --port "onion:80" --path ~/public_html ...or of course make any service at all available as a hidden service, as long as it uses endpoints. -- meejah
![](https://secure.gravatar.com/avatar/869963bdf99b541c9f0bbfb04b0320f1.jpg?s=120&d=mm&r=g)
On 2 May 2014 15:59, David Stainton <dstainton415@gmail.com> wrote:
I think you're approaching this from the wrong angle; instead of starting the tor process during parsing of the endpoint, I think it would make more sense to start the tor process when the endpoint is started. -- mithrandi, i Ainil en-Balandor, a faer Ambar
![](https://secure.gravatar.com/avatar/0c9380641256af855d659dce5ba9a41e.jpg?s=120&d=mm&r=g)
I get what you are saying... but let's back up a second; foolscap (in my branch https://github.com/david415/foolscap/tree/endpoint_descriptors_server2) uses clientFromString and serverFromString to translate twisted endpoint descriptors into endpoint objects... AND foolscap doesn't know any details about how the endpoint wire protocols work... and we'd like to keep it that way. That's really the point of this exercise. The calling party using foolscap... in this case Tahoe-LAFS also doesn't know anything about any wire protocols. It simply receives twisted endpoint descriptors from the user and passes them to foolscap. On Fri, May 2, 2014 at 2:20 PM, Tristan Seligmann <mithrandi@mithrandi.net> wrote:
![](https://secure.gravatar.com/avatar/607cfd4a5b41fe6c886c978128b9c03e.jpg?s=120&d=mm&r=g)
On 02:53 pm, dstainton415@gmail.com wrote:
But Foolscap does know about *endpoints*. It knows to call `listen` or `connect` on them. Both of those methods return a `Deferred`. Any setup work you need to do you should be able to do in the implementation of those methods and however long it takes is how long it takes - just don't fire the `Deferred` until it is ready. Jean-Paul
![](https://secure.gravatar.com/avatar/0c9380641256af855d659dce5ba9a41e.jpg?s=120&d=mm&r=g)
Ah ha! Of course it is so obvious now that you point it out to me. The onion endpoint parser can do this setup work in it's `listen` method which returns a `Deferred`. Thanks for your observation/advice! Cheers! David On Fri, May 2, 2014 at 3:01 PM, <exarkun@twistedmatrix.com> wrote:
![](https://secure.gravatar.com/avatar/a93db92c60e9d5435d204407264457f2.jpg?s=120&d=mm&r=g)
David Stainton <dstainton415@gmail.com> writes:
Cool! Thanks for doing this. Can I pull it into txtorcon when it's finished? Also (more from the Tor end), someone specifying a service will also want to specify the keys for that service. Following the 'ssl' stuff, I guess something like "onion:80:hiddenServiceDir=/dev/shm/foo" might be appropriate? Or perhaps even a way to specify the private key directly? ...and to bikeshed a bit, what about "anontcp:" as a prefix? Then for client-side strings, it could look similar (e.g. "anontcp:encrypted.google.com:443") which could in worst-case launch a new tor instance, wait for it to bootstrap etc in connect(). Perhaps "looking similar" is a non-feature, though :) Another point for this is that "someday" tor will have UDP support... thanks, meejah
![](https://secure.gravatar.com/avatar/e1554622707bedd9202884900430b838.jpg?s=120&d=mm&r=g)
On May 2, 2014, at 1:02 PM, meejah <meejah@meejah.ca> wrote:
Another point for this is that "someday" tor will have UDP support...
There's a reason that the existing interfaces all have "stream" in their names :-). The main reason we didn't do "datagram" at the same time is that there's a pretty clear client/server division in all stream endpoints (one side has to initiate, the other side has to respond) but given the possibilities for multicast / fan-out / fan-in with datagram sockets, "just do what UDP does" didn't seem obviously correct. I'd love it if someone with an interest would make some datagram endpoint interfaces though. -glyph
![](https://secure.gravatar.com/avatar/0c9380641256af855d659dce5ba9a41e.jpg?s=120&d=mm&r=g)
Ah yeah... twisted endpoints that support udp... Perhaps when Yawning Angel finishes Lightweight Obfuscated Datagram Protocol (LODP) there will be interesting things we can do with this... and a more immediate perceived need for twisted datagram endpoint interfaces. On Fri, May 2, 2014 at 9:16 PM, Glyph Lefkowitz <glyph@twistedmatrix.com> wrote:
![](https://secure.gravatar.com/avatar/0c9380641256af855d659dce5ba9a41e.jpg?s=120&d=mm&r=g)
Yes of course! I have every intention of helping out the Tor project as much as I can... I want as many developers to benefit from this as possible so that many more applications can and will utilize the tor network... for anonymity, censorship resistance and great good.
Yes... precisely what I was thinking. For now specifying a directory containing keys might be good enough... but like you say... I should also provide a way to specify the private key directly. Sounds good!
...and to bikeshed a bit, what about "anontcp:" as a prefix? Then for
Yes I agree this is essentially in the bikeshed category of discussions. Let's try to reach consensus. Perhaps off the tahoe-lafs and twisted mailing lists... and perhaps on the tor-dev list. I look forward to collaborating with you (and anyone else who wants to help out).
Yeah... earlier today I also got the naive client side tor endpoint parser working. It uses txsocksx of course. I briefly chatted with the txsocksx author _habnabit on #twisted about possible designs for the client side tor endpoints. Basically we seem to all be on the same page.... in that there may well be a need for multiple types of client side endpoints; We can have an endpoint that has internal logic to retry the tor connection on various ports; 9050, 9150... and another tor client side endpoint that launches it's own tor process (using txtorcon of course) and then uses txsocksx to establish the socks connection to tor. Or maybe it'll be the same endpoint that behaves according to the arguments it's passed.
cheers, david
![](https://secure.gravatar.com/avatar/e1554622707bedd9202884900430b838.jpg?s=120&d=mm&r=g)
On May 2, 2014, at 6:59 AM, David Stainton <dstainton415@gmail.com> wrote:
Today I wrote an endpoint parser for Meejah's txtorcon Tor Hidden Service endpoint.
This is fantastic. Basically this is _exactly_ the use-case that we had in mind when designing the endpoint parser plugin APIs. Thanks so much for doing it. (I hope you'll contribute it to txtorcon when you're done...) -glyph
![](https://secure.gravatar.com/avatar/0c9380641256af855d659dce5ba9a41e.jpg?s=120&d=mm&r=g)
Thanks. Sure thing. Yes I have every intention of contributing endpoint parsers to txtorcon and txsocksx for use with tor. Both of these could make it much easier for developers to make their applications tor friendly. The endpoints/endpoint parsers api is brilliant! When the tor client/server endpoint parsers are complete then Tahoe-LAFS for instance can simply use these endpoints without any code changes at all. Of course I'm not saying that is all that is required for Tahoe-LAFS to be "tor friendly"... we have detailed other considerations in trac tickets: https://tahoe-lafs.org/trac/tahoe-lafs/ticket/517 http://foolscap.lothar.com/trac/ticket/203 I'm starting to think about the deployment issues... Perhaps someone on the list knows of the "correct" setup.py (distutils/setuputils?) procedure to deploy twisted plugins? I shall have to research this soon. For now my hacky development/testing flow works OK. On Fri, May 2, 2014 at 9:18 PM, Glyph Lefkowitz <glyph@twistedmatrix.com> wrote:
![](https://secure.gravatar.com/avatar/e1554622707bedd9202884900430b838.jpg?s=120&d=mm&r=g)
On May 2, 2014, at 5:45 PM, David Stainton <dstainton415@gmail.com> wrote:
This stack overflow question still hasn't gotten a great answer: <https://stackoverflow.com/questions/7275295/how-do-i-write-a-setup-py-for-a-...> If you figure something out, please post it there. However, assuming a post-deluvian version of pip, this simple setup appears to work OK for me: <https://github.com/glyph/txsni/blob/master/setup.py#L16> and I haven't found a setup that breaks it yet. -glyph
![](https://secure.gravatar.com/avatar/0c9380641256af855d659dce5ba9a41e.jpg?s=120&d=mm&r=g)
OK... I decided that txtorcon's endpoint constructor should only take string arguments besides the reactor arg... because serverFromString passes only str args to the endpoint constructor. I wrote the endpoint parser to be as simple as possible. The endpoint's `listen` method now handles the tor configuration and launch logic... and returns a deferred which will fire with an IListenPort. The pip method definitely works... my code works but prints traceback with this error: "Unexpected error while writing cache file" Do you as well? Here's my endpoint parser plugin: https://github.com/david415/txtorcon/blob/endpoint_parser_plugin-rewrite3/tw... Here's how I install it: cd ~/projects/virtualenv-1.11.5 ./virtualenv.py ~/virtenv-txtorcon . ~/virtenv-txtorcon/bin/activate usewithtor pip install twisted cd ~/projects git clone https://github.com/david415/txtorcon.git cd txtorcon git checkout endpoint_parser_plugin-rewrite3 python setup.py install Running the example code: cd examples python launch_tor_endpoint.py Unexpected error while writing cache file Traceback (most recent call last): File "/home/human/virtenv-txtorcon/local/lib/python2.7/site-packages/twisted/internet/endpoints.py", line 1465, in serverFromString return _serverFromStringLegacy(reactor, description, _NO_DEFAULT) File "/home/human/virtenv-txtorcon/local/lib/python2.7/site-packages/twisted/internet/endpoints.py", line 1393, in _serverFromStringLegacy nameOrPlugin, args, kw = _parseServer(description, None, default) File "/home/human/virtenv-txtorcon/local/lib/python2.7/site-packages/twisted/internet/endpoints.py", line 1380, in _parseServer for plugin in getPlugins(IStreamServerEndpointStringParser): File "/home/human/virtenv-txtorcon/local/lib/python2.7/site-packages/twisted/plugin.py", line 209, in getPlugins allDropins = getCache(package) --- <exception caught here> --- File "/home/human/virtenv-txtorcon/local/lib/python2.7/site-packages/twisted/plugin.py", line 181, in getCache dropinPath.setContent(pickle.dumps(dropinDotCache)) exceptions.AttributeError: 'ZipPath' object has no attribute 'setContent' I have set up a hidden service, advertised at: <<class 'twisted.internet.tcp.Port'> of twisted.web.server.Site on 55141> http://mqrcnytlnh4xynmh.onion:80 locally listening on IPv4Address(TCP, '127.0.0.1', 55141) and txtorcon's setup.py output: python setup.py install WARNING: not using PyPi over SSH! running install running bdist_egg running egg_info writing requirements to txtorcon.egg-info/requires.txt writing txtorcon.egg-info/PKG-INFO writing top-level names to txtorcon.egg-info/top_level.txt writing dependency_links to txtorcon.egg-info/dependency_links.txt package init file 'twisted/plugins/__init__.py' not found (or not a regular file) reading manifest file 'txtorcon.egg-info/SOURCES.txt' reading manifest template 'MANIFEST.in' warning: no files found matching 'scripts/*.py' warning: no previously-included files found matching 'docs/_static/*~' warning: no previously-included files found matching 'examples/*~' writing manifest file 'txtorcon.egg-info/SOURCES.txt' installing library code to build/bdist.linux-x86_64/egg running install_lib running build_py creating build/bdist.linux-x86_64/egg creating build/bdist.linux-x86_64/egg/twisted creating build/bdist.linux-x86_64/egg/twisted/plugins copying build/lib.linux-x86_64-2.7/twisted/plugins/txtorcon_endpoint_parser.py -> build/bdist.linux-x86_64/egg/twisted/plugins creating build/bdist.linux-x86_64/egg/txtorcon copying build/lib.linux-x86_64-2.7/txtorcon/stream.py -> build/bdist.linux-x86_64/egg/txtorcon copying build/lib.linux-x86_64-2.7/txtorcon/log.py -> build/bdist.linux-x86_64/egg/txtorcon copying build/lib.linux-x86_64-2.7/txtorcon/torconfig.py -> build/bdist.linux-x86_64/egg/txtorcon copying build/lib.linux-x86_64-2.7/txtorcon/router.py -> build/bdist.linux-x86_64/egg/txtorcon copying build/lib.linux-x86_64-2.7/txtorcon/torcontrolprotocol.py -> build/bdist.linux-x86_64/egg/txtorcon copying build/lib.linux-x86_64-2.7/txtorcon/circuit.py -> build/bdist.linux-x86_64/egg/txtorcon copying build/lib.linux-x86_64-2.7/txtorcon/addrmap.py -> build/bdist.linux-x86_64/egg/txtorcon copying build/lib.linux-x86_64-2.7/txtorcon/interface.py -> build/bdist.linux-x86_64/egg/txtorcon copying build/lib.linux-x86_64-2.7/txtorcon/spaghetti.py -> build/bdist.linux-x86_64/egg/txtorcon copying build/lib.linux-x86_64-2.7/txtorcon/torstate.py -> build/bdist.linux-x86_64/egg/txtorcon copying build/lib.linux-x86_64-2.7/txtorcon/torinfo.py -> build/bdist.linux-x86_64/egg/txtorcon copying build/lib.linux-x86_64-2.7/txtorcon/__init__.py -> build/bdist.linux-x86_64/egg/txtorcon copying build/lib.linux-x86_64-2.7/txtorcon/util.py -> build/bdist.linux-x86_64/egg/txtorcon byte-compiling build/bdist.linux-x86_64/egg/twisted/plugins/txtorcon_endpoint_parser.py to txtorcon_endpoint_parser.pyc byte-compiling build/bdist.linux-x86_64/egg/txtorcon/stream.py to stream.pyc byte-compiling build/bdist.linux-x86_64/egg/txtorcon/log.py to log.pyc byte-compiling build/bdist.linux-x86_64/egg/txtorcon/torconfig.py to torconfig.pyc byte-compiling build/bdist.linux-x86_64/egg/txtorcon/router.py to router.pyc byte-compiling build/bdist.linux-x86_64/egg/txtorcon/torcontrolprotocol.py to torcontrolprotocol.pyc byte-compiling build/bdist.linux-x86_64/egg/txtorcon/circuit.py to circuit.pyc byte-compiling build/bdist.linux-x86_64/egg/txtorcon/addrmap.py to addrmap.pyc byte-compiling build/bdist.linux-x86_64/egg/txtorcon/interface.py to interface.pyc byte-compiling build/bdist.linux-x86_64/egg/txtorcon/spaghetti.py to spaghetti.pyc byte-compiling build/bdist.linux-x86_64/egg/txtorcon/torstate.py to torstate.pyc byte-compiling build/bdist.linux-x86_64/egg/txtorcon/torinfo.py to torinfo.pyc byte-compiling build/bdist.linux-x86_64/egg/txtorcon/__init__.py to __init__.pyc byte-compiling build/bdist.linux-x86_64/egg/txtorcon/util.py to util.pyc installing package data to build/bdist.linux-x86_64/egg running install_data creating build/bdist.linux-x86_64/egg/share creating build/bdist.linux-x86_64/egg/share/txtorcon copying INSTALL -> build/bdist.linux-x86_64/egg/share/txtorcon copying README.rst -> build/bdist.linux-x86_64/egg/share/txtorcon copying TODO -> build/bdist.linux-x86_64/egg/share/txtorcon copying meejah.asc -> build/bdist.linux-x86_64/egg/share/txtorcon copying docs/apilinks_sphinxext.py -> build/bdist.linux-x86_64/egg/share/txtorcon copying docs/conf.py -> build/bdist.linux-x86_64/egg/share/txtorcon copying docs/Makefile -> build/bdist.linux-x86_64/egg/share/txtorcon copying docs/txtorcon-interface.rst -> build/bdist.linux-x86_64/egg/share/txtorcon copying docs/txtorcon-launching.rst -> build/bdist.linux-x86_64/egg/share/txtorcon copying docs/txtorcon-state.rst -> build/bdist.linux-x86_64/egg/share/txtorcon copying docs/txtorcon.rst -> build/bdist.linux-x86_64/egg/share/txtorcon copying docs/txtorcon-util.rst -> build/bdist.linux-x86_64/egg/share/txtorcon copying docs/examples.rst -> build/bdist.linux-x86_64/egg/share/txtorcon copying docs/releases.rst -> build/bdist.linux-x86_64/egg/share/txtorcon copying docs/walkthrough.rst -> build/bdist.linux-x86_64/egg/share/txtorcon copying docs/introduction.rst -> build/bdist.linux-x86_64/egg/share/txtorcon copying docs/txtorcon-config.rst -> build/bdist.linux-x86_64/egg/share/txtorcon copying docs/index.rst -> build/bdist.linux-x86_64/egg/share/txtorcon copying docs/txtorcon-protocol.rst -> build/bdist.linux-x86_64/egg/share/txtorcon copying docs/_static/logo.png -> build/bdist.linux-x86_64/egg/share/txtorcon copying docs/_static/haiku.css -> build/bdist.linux-x86_64/egg/share/txtorcon copying docs/_static/avatar.png -> build/bdist.linux-x86_64/egg/share/txtorcon creating build/bdist.linux-x86_64/egg/share/txtorcon/examples copying examples/monitor.py -> build/bdist.linux-x86_64/egg/share/txtorcon/examples copying examples/launch_tor.py -> build/bdist.linux-x86_64/egg/share/txtorcon/examples copying examples/webui_server.py -> build/bdist.linux-x86_64/egg/share/txtorcon/examples copying examples/stream_circuit_logger.py -> build/bdist.linux-x86_64/egg/share/txtorcon/examples copying examples/launch_tor_with_simplehttpd.py -> build/bdist.linux-x86_64/egg/share/txtorcon/examples copying examples/launch_tor_endpoint.py -> build/bdist.linux-x86_64/egg/share/txtorcon/examples copying examples/circuit_for_next_stream.py -> build/bdist.linux-x86_64/egg/share/txtorcon/examples copying examples/dump_config.py -> build/bdist.linux-x86_64/egg/share/txtorcon/examples copying examples/schedule_bandwidth.py -> build/bdist.linux-x86_64/egg/share/txtorcon/examples copying examples/disallow_streams_by_port.py -> build/bdist.linux-x86_64/egg/share/txtorcon/examples copying examples/attach_streams_by_country.py -> build/bdist.linux-x86_64/egg/share/txtorcon/examples copying examples/launch_tor_with_hiddenservice.py -> build/bdist.linux-x86_64/egg/share/txtorcon/examples copying examples/circuit_failure_rates.py -> build/bdist.linux-x86_64/egg/share/txtorcon/examples copying examples/tor_info.py -> build/bdist.linux-x86_64/egg/share/txtorcon/examples creating build/bdist.linux-x86_64/egg/EGG-INFO copying txtorcon.egg-info/PKG-INFO -> build/bdist.linux-x86_64/egg/EGG-INFO copying txtorcon.egg-info/SOURCES.txt -> build/bdist.linux-x86_64/egg/EGG-INFO copying txtorcon.egg-info/dependency_links.txt -> build/bdist.linux-x86_64/egg/EGG-INFO copying txtorcon.egg-info/requires.txt -> build/bdist.linux-x86_64/egg/EGG-INFO copying txtorcon.egg-info/top_level.txt -> build/bdist.linux-x86_64/egg/EGG-INFO zip_safe flag not set; analyzing archive contents... creating 'dist/txtorcon-0.9.2-py2.7.egg' and adding 'build/bdist.linux-x86_64/egg' to it removing 'build/bdist.linux-x86_64/egg' (and everything under it) Processing txtorcon-0.9.2-py2.7.egg Copying txtorcon-0.9.2-py2.7.egg to /home/human/virtenv-txtorcon/lib/python2.7/site-packages Adding txtorcon 0.9.2 to easy-install.pth file Installed /home/human/virtenv-txtorcon/lib/python2.7/site-packages/txtorcon-0.9.2-py2.7.egg Processing dependencies for txtorcon==0.9.2 Searching for zope.interface==4.1.1 Best match: zope.interface 4.1.1 Adding zope.interface 4.1.1 to easy-install.pth file Using /home/human/virtenv-txtorcon/lib/python2.7/site-packages Searching for Twisted==13.2.0 Best match: Twisted 13.2.0 Adding Twisted 13.2.0 to easy-install.pth file Using /home/human/virtenv-txtorcon/lib/python2.7/site-packages Searching for setuptools==3.4.4 Best match: setuptools 3.4.4 Adding setuptools 3.4.4 to easy-install.pth file Installing easy_install-3.4 script to /home/human/virtenv-txtorcon/bin Installing easy_install script to /home/human/virtenv-txtorcon/bin Using /home/human/virtenv-txtorcon/lib/python2.7/site-packages Finished processing dependencies for txtorcon==0.9.2
![](https://secure.gravatar.com/avatar/e1554622707bedd9202884900430b838.jpg?s=120&d=mm&r=g)
On May 5, 2014, at 6:52 PM, David Stainton <dstainton415@gmail.com> wrote:
I strongly suggest that this is wrong. The endpoint constructor should do _construction_. The endpoint parser should do _parsing_. The task of parsing is of taking strings and producing meaningful values. You may want to have the endpoint parser actually live in a separate method in txtorcon rather than putting it into the plugin itself, and leave the plugin simply the task of plugging in, but the parsing method should not be called __init__. (Also, don't use asserts. This is probably indicative of a weakness in the parsing API we've provided, but you should likely raise something like UsageError.)
Great.
What exactly is 'usewithtor'?
usewithtor pip install twisted
This traceback is actually harmless, if annoying. It's trying to generate the plugin cache, which is explained here: <https://twistedmatrix.com/documents/current/core/howto/plugin.html#auto3>
running bdist_egg ^ This right here is your problem.
There are a couple of solutions, in order of increasing complexity: Don't build eggs. Why are you building eggs? Eggs are unfortunately a huge problem magnet. I don't think 'pip install' builds eggs; I don't manually run setup.py anywhere any more because I assume pip understands these concerns better than I do. Set zip_safe in your setup.py to False, which will turn your egg into a directory rather than a directory. Figure out a way to hook the _build_ step specifically so you can run getPlugins() to generate the dropin cache before zipping up your zip file. Twisted plugins actually do work within zip files, but nobody has bothered working out the setuptools incantation to get them to generate at the appropriate time for building an egg. This is extra tricky because you only want to generate this dropin.cache file if the dropin is going to live inside an egg; otherwise, it would conflict with the communal dropin.cache file generated at install time. This option will be the most confusing and frustrating for you, but I hope you choose it, because it would be GREAT to have a Right Way™ to address this problem in a Twisted plugin project's setup.py :).
zip_safe flag not set; analyzing archive contents...
It's probably best to set zip_safe one way or the other though, even if you want to set it to 'true'. Letting setuptools guess like this will, unsurprisingly, get you unpredictable results. -g
![](https://secure.gravatar.com/avatar/0c9380641256af855d659dce5ba9a41e.jpg?s=120&d=mm&r=g)
Ah yeah I see what you mean. OK... I changed it: https://github.com/david415/txtorcon/commit/1e96d550c40bef1be1b45c3c975c0da0...
Done.
OK... I changed that part too.
What exactly is 'usewithtor'?
usewithtor pip install twisted
It's torsocks. It performs an LD_PRELOAD hack to force programs to use tor. Here's some torsocks info: https://trac.torproject.org/projects/tor/wiki/doc/torsocks https://gitweb.torproject.org/torsocks.git https://lists.torproject.org/pipermail/tor-dev/2014-April/006649.html Torsocks is useful... but a terrible hack... and this reminds me: I'm hoping that the effort to make Tahoe-LAFS more Tor-friendly will result in well documented solutions. I would hope that this would make it easier for other developers to make their twisted apps more Tor-friendly as well. I do not think we have all the solution details worked out yet... We'd like to stop using torsocks with Tahoe-LAFS and instead use twisted endpoints... which is why I ported the Foolscap client side to use twisted endpoints. (str4d already ported the Foolscap server side to use endpoints) https://tahoe-lafs.org/trac/tahoe-lafs/ticket/517 This trac ticket discusses various ideas to make Tahoe more Tor friendly... including this endpoint related idea that I've been thinking about lately in trac comment 16 (https://tahoe-lafs.org/trac/tahoe-lafs/ticket/517#comment:16) by leif : """My current thought is that Tahoe should have an "always_use_tor" option which causes all furls to be rewritten as tor: endpoints before being passed to foolscap. If this option is not enabled, tor connections will be made if a tor endpoint is used, but not otherwise. So, users of mixed tor/non-tor grids can leave it disabled if they prefer faster connections to non-tor nodes.""" This is necessary for a tahoe client wanting to use a public grid over tor... because the grid's introducer node is going to announce tcp endpoints not tor endpoints. However I was thinking that the "only_tor" mode for tahoe (mentioned here https://tahoe-lafs.org/trac/tahoe-lafs/ticket/517#comment:15 ) could be implemented in tahoe... it could filter endpoint descriptors by calling clientFromString or serverFromString and then checking the object type.... or is this a terrible way to do it? I'm not sure. The thing is... tahoe wouldn't normally know anything about endpoints because it uses Foolscap which could use endpoints (it does in my branch https://github.com/david415/foolscap/tree/endpoint_descriptors_server2).
Ah OK yeah it works just fine when installed with pip.
OK... I'll try to get it working that way... but for now a pip install seems to work fine.
![](https://secure.gravatar.com/avatar/a93db92c60e9d5435d204407264457f2.jpg?s=120&d=mm&r=g)
As a followup to all this, I have just merged a branch with a refactored endpoint parser. tomprince (I believe) on #twisted suggested some SSH endpoint code to look at, which inspired the @classmethod style way of getting a TCPHiddenServiceEndpoint instance configured differently. Nearly all the code is in endpoints.py if anyone is interested, here: https://github.com/meejah/txtorcon/blob/master/txtorcon/endpoints.py#L117 If you do look, I'd certainly appreciate any comments. (The tests are in test/test_endpoints.py) Anyway, one really cool thing now that txtorcon supports endpoints is that you can install it, and then easily make Twisted Web serve some content as a hidden service (launching a Tor instance first, etc): twistd -n web --port "onion:80" --path ~/public_html ...or of course make any service at all available as a hidden service, as long as it uses endpoints. -- meejah
participants (6)
-
David Stainton
-
exarkun@twistedmatrix.com
-
Glyph
-
Glyph Lefkowitz
-
meejah
-
Tristan Seligmann