[Twisted-Python] adding endpoint plugins
Greetings, I wrote a simple IStreamClientEndpointStringParser for using the SOCKS5ClientEndpoint with Tor: https://gist.github.com/david415/26a4ed59078d2e27376f I modified a txsocksx code example for testing the new endpoint descriptor: https://gist.github.com/david415/7c6040117319cc3b0230 ...but I am not sure how to "register" it such that calls to clientFromString can return this endpoint when the descriptor prefix matches. This doc: https://twistedmatrix.com/documents/13.2.0/api/twisted.internet.interfaces.I... mentions placing the plugin in the twisted.plugins package... however this doc: http://twistedsphinx.funsize.net/projects/core/howto/plugin.html suggests that I can also set the PYTHONPATH to a directory: """if a directory which has been added to sys.path (typically by adding it to the PYTHONPATH environment variable) contains a directory named twisted/plugins/ , each .py file in that directory will be loaded as a source of plugins.""" I'm using twisted 13.2.0 in a virtualenv... and I've tried setting PYTHONPATH to a "twisted/plugins" directory no avail. I must be doing something wrong here but after scouring the docs and twisted source code... I'm just not sure. This effort is part of solving for Tahoe-LAFS trac ticket 517: "make tahoe Tor- and I2P-friendly" https://tahoe-lafs.org/trac/tahoe-lafs/ticket/517 Thanks to Glyph and JP's help I was able to write a foolscap client side patch to use twisted endpoints. See foolscap trac ticket 203: http://foolscap.lothar.com/trac/ticket/203#comment:36 str4d has already written the foolscap server side endpoint patch... Therefore it seems to me the next steps are to create parsers for endpoints we are interested in supporting... and get Tahoe-LAFS to register them so that serverFromString and clientFromString will return the expected endpoint objects. Cheers! David
On 01:42 pm, dstainton415@gmail.com wrote:
Greetings,
http://twistedsphinx.funsize.net/projects/core/howto/plugin.html
Note that's a random version of the documentation from who knows how long ago. If you want a random version of the documentation then I suggest using http://twisted.readthedocs.org/en/latest/ instead.
suggests that I can also set the PYTHONPATH to a directory:
Sure. As a development environment trick. Putting the module defining your plugin into *the* twisted/plugins/ directory will reduce your runtime cost. This is purely a deployment decision, though, so if you'd rather expand PYTHONPATH then you can.
"""if a directory which has been added to sys.path (typically by adding it to the PYTHONPATH environment variable) contains a directory named twisted/plugins/ , each .py file in that directory will be loaded as a source of plugins."""
I'm using twisted 13.2.0 in a virtualenv... and I've tried setting PYTHONPATH to a "twisted/plugins" directory no avail. I must be doing something wrong here but after scouring the docs and twisted source code... I'm just not sure.
I don't think there's enough information here to debug your problem. Try coming up with an http://sscce.org/ or at least providing all of the details about your setup - the exact value of PYTHONPATH, the command you use to launch the program, the working directory you use, the complete, exact filesystem hierarchy starting at the top-most directory containing any code you wrote or expect to be in use, etc. Some random suggestions that have helped other people and might help you but who knows: * delete all the dropin.cache files you can find and try again * delete all the .pyc files you can find and try again * put absolute paths into PYTHONPATH, not relative paths * don't put "/foo/bar/twisted/plugins" into PYTHONPATH, put "/foo/bar" * Make sure your plugin *provides* the plugin interface and IPlugin. This is not the same *implementing* those interfaces. Good luck, Jean-Paul
Hi Jean-Paul, Thanks for the suggestions. OK... here's my SSCCE: cd projects/virtualenv-1.11.1/ ./virtualenv.py ~/virtenv-endpoints-test . ~/virtenv-endpoints-test/bin/activate usewithtor pip install twisted cd ~/projects git clone https://github.com/david415/txsocksx.git cd txsocksx git checkout endpoint_parsers usewithtor python setup.py install cd examples # CWD is now /home/human/projects/txsocksx/examples cp tor_client_endpoint_parser.py /home/human/virtenv-endpoints-test/lib/python2.7/site-packages/twisted/plugins find /home/human/virtenv-endpoints-test/ -name "*.pyc" | xargs rm find /home/human/virtenv-endpoints-test/ -name "dropin.cache" | xargs rm python tor-http.py Traceback (most recent call last): File "tor-http.py", line 32, in <module> torEndpoint = clientFromString(reactor, "tor:host=timaq4ygg2iegci7.onion:port=80") File "/home/human/virtenv-endpoints-test/local/lib/python2.7/site-packages/twisted/internet/endpoints.py", line 1737, in clientFromString raise ValueError("Unknown endpoint type: %r" % (aname,)) ValueError: Unknown endpoint type: 'tor' I also tried the PYTHONPATH trick: mkdir -p fu/twisted/plugins cp tor_client_endpoint_parser.py fu/twisted/plugins PYTHONPATH=/home/projects/txsocks/examples/fu python tor-http.py Traceback (most recent call last): File "tor-http.py", line 32, in <module> torEndpoint = clientFromString(reactor, "tor:host=timaq4ygg2iegci7.onion:port=80") File "/home/human/virtenv-endpoints-test/local/lib/python2.7/site-packages/twisted/internet/endpoints.py", line 1737, in clientFromString raise ValueError("Unknown endpoint type: %r" % (aname,)) ValueError: Unknown endpoint type: 'tor' I wonder if I am having a deployment problem here or if my code has got problems. https://github.com/david415/txsocksx/blob/endpoint_parsers/examples/tor_clie... https://github.com/david415/txsocksx/blob/endpoint_parsers/examples/tor-http... Cheers! David On Wed, Apr 30, 2014 at 2:14 PM, <exarkun@twistedmatrix.com> wrote:
On 01:42 pm, dstainton415@gmail.com wrote:
Greetings,
http://twistedsphinx.funsize.net/projects/core/howto/plugin.html
Note that's a random version of the documentation from who knows how long ago. If you want a random version of the documentation then I suggest using http://twisted.readthedocs.org/en/latest/ instead.
suggests that I can also set the PYTHONPATH to a directory:
Sure. As a development environment trick. Putting the module defining your plugin into *the* twisted/plugins/ directory will reduce your runtime cost. This is purely a deployment decision, though, so if you'd rather expand PYTHONPATH then you can.
"""if a directory which has been added to sys.path (typically by adding it to the PYTHONPATH environment variable) contains a directory named twisted/plugins/ , each .py file in that directory will be loaded as a source of plugins."""
I'm using twisted 13.2.0 in a virtualenv... and I've tried setting PYTHONPATH to a "twisted/plugins" directory no avail. I must be doing something wrong here but after scouring the docs and twisted source code... I'm just not sure.
I don't think there's enough information here to debug your problem. Try coming up with an http://sscce.org/ or at least providing all of the details about your setup - the exact value of PYTHONPATH, the command you use to launch the program, the working directory you use, the complete, exact filesystem hierarchy starting at the top-most directory containing any code you wrote or expect to be in use, etc.
Some random suggestions that have helped other people and might help you but who knows:
* delete all the dropin.cache files you can find and try again * delete all the .pyc files you can find and try again * put absolute paths into PYTHONPATH, not relative paths * don't put "/foo/bar/twisted/plugins" into PYTHONPATH, put "/foo/bar" * Make sure your plugin *provides* the plugin interface and IPlugin. This is not the same *implementing* those interfaces.
Good luck, Jean-Paul
_______________________________________________ Twisted-Python mailing list Twisted-Python@twistedmatrix.com http://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python
On Wed, Apr 30, 2014 at 9:14 AM, <exarkun@twistedmatrix.com> wrote:
On 01:42 pm, dstainton415@gmail.com wrote:
Greetings,
http://twistedsphinx.funsize.net/projects/core/howto/plugin.html
Note that's a random version of the documentation from who knows how long ago. If you want a random version of the documentation then I suggest using http://twisted.readthedocs.org/en/latest/ instead.
Probably from 2011 or so... yup, looks like end of 2011.
Now that the Sphinx transition has actually happened, I suppose I should probably take that down and perhaps replace it with a pointer to the RTD version.
On Apr 30, 2014, at 9:53 AM, Kevin Horn <kevin.horn@gmail.com> wrote:
On Wed, Apr 30, 2014 at 9:14 AM, <exarkun@twistedmatrix.com> wrote: On 01:42 pm, dstainton415@gmail.com wrote: Greetings,
http://twistedsphinx.funsize.net/projects/core/howto/plugin.html
Note that's a random version of the documentation from who knows how long ago. If you want a random version of the documentation then I suggest using http://twisted.readthedocs.org/en/latest/ instead.
Probably from 2011 or so... yup, looks like end of 2011.
Now that the Sphinx transition has actually happened, I suppose I should probably take that down and perhaps replace it with a pointer to the RTD version.
Yes, that would be good. Speaking of which, does anyone know how to get the default RTD version to point to an actual release? Since 14.0 is imminent we should probably turn the appropriate knob there. -glyph
On Apr 30, 2014, at 2:56 PM, Glyph Lefkowitz <glyph@twistedmatrix.com> wrote:
On Apr 30, 2014, at 9:53 AM, Kevin Horn <kevin.horn@gmail.com> wrote:
On Wed, Apr 30, 2014 at 9:14 AM, <exarkun@twistedmatrix.com> wrote: On 01:42 pm, dstainton415@gmail.com wrote: Greetings,
http://twistedsphinx.funsize.net/projects/core/howto/plugin.html
Note that's a random version of the documentation from who knows how long ago. If you want a random version of the documentation then I suggest using http://twisted.readthedocs.org/en/latest/ instead.
Probably from 2011 or so... yup, looks like end of 2011.
Now that the Sphinx transition has actually happened, I suppose I should probably take that down and perhaps replace it with a pointer to the RTD version.
Yes, that would be good.
Speaking of which, does anyone know how to get the default RTD version to point to an actual release? Since 14.0 is imminent we should probably turn the appropriate knob there.
-glyph
_______________________________________________ Twisted-Python mailing list Twisted-Python@twistedmatrix.com http://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python
There is a drop down for default release in the RTD admin panel. ----------------- Donald Stufft PGP: 0x6E3CBCE93372DCFA // 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA
On Apr 30, 2014, at 12:09 PM, Donald Stufft <donald@stufft.io> wrote:
There is a drop down for default release in the RTD admin panel.
The drop down isn't super helpful: But I noticed that page also has a huuuuuge list of branches on it, so I checked off the most recent 14.0 branch and now it's toiling away. We should add this to the release process. Hawkowl? -g
Sounds like a good idea, I'll make a note to do it and write it in the release process once I've done it. - HawkOwl
On 2 May 2014, at 3:33, Glyph Lefkowitz <glyph@twistedmatrix.com> wrote:
On Apr 30, 2014, at 12:09 PM, Donald Stufft <donald@stufft.io> wrote:
There is a drop down for default release in the RTD admin panel.
The drop down isn't super helpful:
<PastedGraphic-1.png>
But I noticed that page also has a huuuuuge list of branches on it, so I checked off the most recent 14.0 branch and now it's toiling away.
We should add this to the release process. Hawkowl?
-g
_______________________________________________ Twisted-Python mailing list Twisted-Python@twistedmatrix.com http://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python
participants (6)
-
David Stainton -
Donald Stufft -
exarkun＠twistedmatrix.com -
Glyph Lefkowitz -
HawkOwl -
Kevin Horn