[Twisted-Python] adding endpoint plugins
![](https://secure.gravatar.com/avatar/0c9380641256af855d659dce5ba9a41e.jpg?s=120&d=mm&r=g)
Greetings, I wrote a simple IStreamClientEndpointStringParser for using the SOCKS5ClientEndpoint with Tor: https://gist.github.com/david415/26a4ed59078d2e27376f I modified a txsocksx code example for testing the new endpoint descriptor: https://gist.github.com/david415/7c6040117319cc3b0230 ...but I am not sure how to "register" it such that calls to clientFromString can return this endpoint when the descriptor prefix matches. This doc: https://twistedmatrix.com/documents/13.2.0/api/twisted.internet.interfaces.I... mentions placing the plugin in the twisted.plugins package... however this doc: http://twistedsphinx.funsize.net/projects/core/howto/plugin.html suggests that I can also set the PYTHONPATH to a directory: """if a directory which has been added to sys.path (typically by adding it to the PYTHONPATH environment variable) contains a directory named twisted/plugins/ , each .py file in that directory will be loaded as a source of plugins.""" I'm using twisted 13.2.0 in a virtualenv... and I've tried setting PYTHONPATH to a "twisted/plugins" directory no avail. I must be doing something wrong here but after scouring the docs and twisted source code... I'm just not sure. This effort is part of solving for Tahoe-LAFS trac ticket 517: "make tahoe Tor- and I2P-friendly" https://tahoe-lafs.org/trac/tahoe-lafs/ticket/517 Thanks to Glyph and JP's help I was able to write a foolscap client side patch to use twisted endpoints. See foolscap trac ticket 203: http://foolscap.lothar.com/trac/ticket/203#comment:36 str4d has already written the foolscap server side endpoint patch... Therefore it seems to me the next steps are to create parsers for endpoints we are interested in supporting... and get Tahoe-LAFS to register them so that serverFromString and clientFromString will return the expected endpoint objects. Cheers! David
![](https://secure.gravatar.com/avatar/607cfd4a5b41fe6c886c978128b9c03e.jpg?s=120&d=mm&r=g)
On 01:42 pm, dstainton415@gmail.com wrote:
Greetings,
http://twistedsphinx.funsize.net/projects/core/howto/plugin.html
Note that's a random version of the documentation from who knows how long ago. If you want a random version of the documentation then I suggest using http://twisted.readthedocs.org/en/latest/ instead.
suggests that I can also set the PYTHONPATH to a directory:
Sure. As a development environment trick. Putting the module defining your plugin into *the* twisted/plugins/ directory will reduce your runtime cost. This is purely a deployment decision, though, so if you'd rather expand PYTHONPATH then you can.
I don't think there's enough information here to debug your problem. Try coming up with an http://sscce.org/ or at least providing all of the details about your setup - the exact value of PYTHONPATH, the command you use to launch the program, the working directory you use, the complete, exact filesystem hierarchy starting at the top-most directory containing any code you wrote or expect to be in use, etc. Some random suggestions that have helped other people and might help you but who knows: * delete all the dropin.cache files you can find and try again * delete all the .pyc files you can find and try again * put absolute paths into PYTHONPATH, not relative paths * don't put "/foo/bar/twisted/plugins" into PYTHONPATH, put "/foo/bar" * Make sure your plugin *provides* the plugin interface and IPlugin. This is not the same *implementing* those interfaces. Good luck, Jean-Paul
![](https://secure.gravatar.com/avatar/0c9380641256af855d659dce5ba9a41e.jpg?s=120&d=mm&r=g)
Hi Jean-Paul, Thanks for the suggestions. OK... here's my SSCCE: cd projects/virtualenv-1.11.1/ ./virtualenv.py ~/virtenv-endpoints-test . ~/virtenv-endpoints-test/bin/activate usewithtor pip install twisted cd ~/projects git clone https://github.com/david415/txsocksx.git cd txsocksx git checkout endpoint_parsers usewithtor python setup.py install cd examples # CWD is now /home/human/projects/txsocksx/examples cp tor_client_endpoint_parser.py /home/human/virtenv-endpoints-test/lib/python2.7/site-packages/twisted/plugins find /home/human/virtenv-endpoints-test/ -name "*.pyc" | xargs rm find /home/human/virtenv-endpoints-test/ -name "dropin.cache" | xargs rm python tor-http.py Traceback (most recent call last): File "tor-http.py", line 32, in <module> torEndpoint = clientFromString(reactor, "tor:host=timaq4ygg2iegci7.onion:port=80") File "/home/human/virtenv-endpoints-test/local/lib/python2.7/site-packages/twisted/internet/endpoints.py", line 1737, in clientFromString raise ValueError("Unknown endpoint type: %r" % (aname,)) ValueError: Unknown endpoint type: 'tor' I also tried the PYTHONPATH trick: mkdir -p fu/twisted/plugins cp tor_client_endpoint_parser.py fu/twisted/plugins PYTHONPATH=/home/projects/txsocks/examples/fu python tor-http.py Traceback (most recent call last): File "tor-http.py", line 32, in <module> torEndpoint = clientFromString(reactor, "tor:host=timaq4ygg2iegci7.onion:port=80") File "/home/human/virtenv-endpoints-test/local/lib/python2.7/site-packages/twisted/internet/endpoints.py", line 1737, in clientFromString raise ValueError("Unknown endpoint type: %r" % (aname,)) ValueError: Unknown endpoint type: 'tor' I wonder if I am having a deployment problem here or if my code has got problems. https://github.com/david415/txsocksx/blob/endpoint_parsers/examples/tor_clie... https://github.com/david415/txsocksx/blob/endpoint_parsers/examples/tor-http... Cheers! David On Wed, Apr 30, 2014 at 2:14 PM, <exarkun@twistedmatrix.com> wrote:
![](https://secure.gravatar.com/avatar/e1554622707bedd9202884900430b838.jpg?s=120&d=mm&r=g)
On Apr 30, 2014, at 12:09 PM, Donald Stufft <donald@stufft.io> wrote:
There is a drop down for default release in the RTD admin panel.
The drop down isn't super helpful: But I noticed that page also has a huuuuuge list of branches on it, so I checked off the most recent 14.0 branch and now it's toiling away. We should add this to the release process. Hawkowl? -g
![](https://secure.gravatar.com/avatar/607cfd4a5b41fe6c886c978128b9c03e.jpg?s=120&d=mm&r=g)
On 01:42 pm, dstainton415@gmail.com wrote:
Greetings,
http://twistedsphinx.funsize.net/projects/core/howto/plugin.html
Note that's a random version of the documentation from who knows how long ago. If you want a random version of the documentation then I suggest using http://twisted.readthedocs.org/en/latest/ instead.
suggests that I can also set the PYTHONPATH to a directory:
Sure. As a development environment trick. Putting the module defining your plugin into *the* twisted/plugins/ directory will reduce your runtime cost. This is purely a deployment decision, though, so if you'd rather expand PYTHONPATH then you can.
I don't think there's enough information here to debug your problem. Try coming up with an http://sscce.org/ or at least providing all of the details about your setup - the exact value of PYTHONPATH, the command you use to launch the program, the working directory you use, the complete, exact filesystem hierarchy starting at the top-most directory containing any code you wrote or expect to be in use, etc. Some random suggestions that have helped other people and might help you but who knows: * delete all the dropin.cache files you can find and try again * delete all the .pyc files you can find and try again * put absolute paths into PYTHONPATH, not relative paths * don't put "/foo/bar/twisted/plugins" into PYTHONPATH, put "/foo/bar" * Make sure your plugin *provides* the plugin interface and IPlugin. This is not the same *implementing* those interfaces. Good luck, Jean-Paul
![](https://secure.gravatar.com/avatar/0c9380641256af855d659dce5ba9a41e.jpg?s=120&d=mm&r=g)
Hi Jean-Paul, Thanks for the suggestions. OK... here's my SSCCE: cd projects/virtualenv-1.11.1/ ./virtualenv.py ~/virtenv-endpoints-test . ~/virtenv-endpoints-test/bin/activate usewithtor pip install twisted cd ~/projects git clone https://github.com/david415/txsocksx.git cd txsocksx git checkout endpoint_parsers usewithtor python setup.py install cd examples # CWD is now /home/human/projects/txsocksx/examples cp tor_client_endpoint_parser.py /home/human/virtenv-endpoints-test/lib/python2.7/site-packages/twisted/plugins find /home/human/virtenv-endpoints-test/ -name "*.pyc" | xargs rm find /home/human/virtenv-endpoints-test/ -name "dropin.cache" | xargs rm python tor-http.py Traceback (most recent call last): File "tor-http.py", line 32, in <module> torEndpoint = clientFromString(reactor, "tor:host=timaq4ygg2iegci7.onion:port=80") File "/home/human/virtenv-endpoints-test/local/lib/python2.7/site-packages/twisted/internet/endpoints.py", line 1737, in clientFromString raise ValueError("Unknown endpoint type: %r" % (aname,)) ValueError: Unknown endpoint type: 'tor' I also tried the PYTHONPATH trick: mkdir -p fu/twisted/plugins cp tor_client_endpoint_parser.py fu/twisted/plugins PYTHONPATH=/home/projects/txsocks/examples/fu python tor-http.py Traceback (most recent call last): File "tor-http.py", line 32, in <module> torEndpoint = clientFromString(reactor, "tor:host=timaq4ygg2iegci7.onion:port=80") File "/home/human/virtenv-endpoints-test/local/lib/python2.7/site-packages/twisted/internet/endpoints.py", line 1737, in clientFromString raise ValueError("Unknown endpoint type: %r" % (aname,)) ValueError: Unknown endpoint type: 'tor' I wonder if I am having a deployment problem here or if my code has got problems. https://github.com/david415/txsocksx/blob/endpoint_parsers/examples/tor_clie... https://github.com/david415/txsocksx/blob/endpoint_parsers/examples/tor-http... Cheers! David On Wed, Apr 30, 2014 at 2:14 PM, <exarkun@twistedmatrix.com> wrote:
![](https://secure.gravatar.com/avatar/e1554622707bedd9202884900430b838.jpg?s=120&d=mm&r=g)
On Apr 30, 2014, at 12:09 PM, Donald Stufft <donald@stufft.io> wrote:
There is a drop down for default release in the RTD admin panel.
The drop down isn't super helpful: But I noticed that page also has a huuuuuge list of branches on it, so I checked off the most recent 14.0 branch and now it's toiling away. We should add this to the release process. Hawkowl? -g
participants (6)
-
David Stainton
-
Donald Stufft
-
exarkun@twistedmatrix.com
-
Glyph Lefkowitz
-
HawkOwl
-
Kevin Horn