[Twisted-Python] Status of current proxy support (of various kinds)
Hello, I'm trying to determine the current status of proxy support in Twisted. I've googled quite a bit to find an answer but I couldn't get a crystal clear picture. What I mean by "proxy support" is the capability to: - act as a proxy server - use an external proxy server (i.e. act as a client) And by proxy, I'm talking in pratice of SOCKS v{4(a),5} and HTTP "CONNECT" standards.
Personnaly, what I'm interested in is the ability for Twisted to act as a proxy client and thus to connect to an external proxy server. I've seen bug reports on that topic, of which some are still open: Add a SOCKS client API http://twistedmatrix.com/trac/ticket/3508 Socks V5 functionality http://twistedmatrix.com/trac/ticket/1330 Add CONNECT support to HTTP client http://twistedmatrix.com/trac/ticket/4969 Regarding this last bug, I was wondering if there could a generic HTTP proxy mechanism, like with SOCKS, and not tied to the Web client. Because we might want to use an HTTP proxy for something else that Web traffic. What would be nice in Twisted is a generic, upper-protocol-agnostic proxy mechanism -- and one that would support indistinctly SOCKS and HTTP kinds of proxies. By the way, I'm totally new to Twisted, so forgive me if what I'm missing something obvious :-) Thanks, -- Damiano ALBANI
On Aug 5, 2011, at 4:14 AM, Damiano ALBANI wrote:
You've correctly identified the status of HTTP and SOCKS proxy support in Twisted. However, it doesn't really make sense to have a "generic" proxy layer - different protocols have wildly different rules about what it means to proxy a connection. HTTP and SOCKS can proxy the same traffic and get very different results about where that traffic goes. Similarly, it doesn't make sense to use an HTTP proxy for anything other than HTTP traffic. If you could clarify your request perhaps we could file a ticket? Otherwise it would be great if you could contribute a patch to advance one of those existing tickets. If you want a totally generic (i.e. TCP-level) proxy, Twisted does contain one: you can run 'twistd portforward --help' for information about how to run it. Good luck, -glyph
Myself and another are working on adding SOCKS client support to Twisted for use in Tor related projects. I'm also convinced there's an abstraction that would make it easier to implement proxy clients (or any transparent data/endpoint modifying protocol) in Twisted. I agree that it's difficult to generalise, but there's a class of protocol that does one or more of the following, transparently, and little else: - append data, before switching to application protocol (SSL handshake, SOCKS connect/bind) - prepend data, after losing connection on application protocol (SSL shutdown) - modify application data (SSL "recordification", encryption) - modify the endpoint (any proxy) Endpoints can do all these things - SSL4ClientEndpoint, for example, does the first three - but does it make sense to use Endpoints for this purpose? Endpoints aren't stackable, so no good if I want to run a SOCKS client over SSL. Also, with a proxied connection you have two Endpoints, one to the proxy server and one 'virtual Endpoint' to wherever. The Endpoints API doesn't accommodate this. Would it make sense to have Endpoint wrappers for this purpose? A sort of middleware (hate that word) to intercept transport events (including transport formation, i.e. Endpoint.connect()). I'm looking forward to being proven wrong on this as it will make my SOCKS work easier, or else if this turns out to be interesting then I have more ideas on how it should be implemented. Cheers, Peter.
On Aug 8, 2011, at 10:10 PM, Peter Le Bek wrote:
I think so.
Endpoints aren't stackable, so no good if I want to run a SOCKS client over SSL.
Sure they are. There's no SOCKS endpoint yet, but if you were to implement one, you could make it take another endpoint (or endpoint factory, as the case may be) as an argument to its constructor. The implementation of the various current SSL* enpdoints would set a better example if it weren't for the legacy connectSSL/listenSSL APIs. However, those APIs are being slowly phased out in favor of the endpoints themselves, as well as the internal implementation being moved to use a wrapping / delegation approach <http://twistedmatrix.com/trac/ticket/4854> between transports and protocols instead of OpenSSL's SSL-is-a-socket-except-when-it-isn't strategy.
How doesn't it? It's exactly this kind of API that endpoints were _designed_ to accommodate - the original impetus to create them was the Vertex project (now part of <https://launchpad.net/divmod.org/>), which proxies traffic in a variety of different ways.
It might make sense to have some utilities around this area. There are definitely parts of the process - especially producers and consumers - where you have to write a lot of boilerplate code to make sure everything's hooked up properly. But it's all certainly _possible_ now with the existing interfaces. You'd probably be interested in this ticket: <http://twistedmatrix.com/trac/ticket/3204>. Perhaps you could submit a patch? :)
It would help with that proof if you would be a bit more precise about where you think the current code falls short. -glyph
Also, anyone working on socks5 in twisted should be aware of this project: http://code.google.com/p/proxy65/ which implements a SOCK5 server. It's in the context of a Jabber/XMPP XEP-0065 proxy server, but a lot of it should be applicable, if just as an example. I make no guarantees about the quality and/or correctness of the code, as I've only skimmed it a couple of times. I do know it predates the endpoint API, and I doubt it has been updated. (getting SOCS5 support into twisted has been on my 'maybe someday" todo list, and I was planning on looking at this in more detail as a reference). It's MIT licensed. Kevin Horn On Mon, Aug 8, 2011 at 11:31 PM, Glyph Lefkowitz <glyph@twistedmatrix.com>wrote:
Pls, show me an example of HTTP authorization. Thanks all. __ http://itfreelancer.ru
On 06:50 am, lukas2k7@mail.ru wrote:
Pls, show me an example of HTTP authorization. Thanks all.
See the online documentation for Twisted Web: http://twistedmatrix.com/documents/current/web/howto/ in particular: http://twistedmatrix.com/documents/current/web/howto/web-in-60/http- auth.html Jean-Paul
On Aug 5, 2011, at 4:14 AM, Damiano ALBANI wrote:
You've correctly identified the status of HTTP and SOCKS proxy support in Twisted. However, it doesn't really make sense to have a "generic" proxy layer - different protocols have wildly different rules about what it means to proxy a connection. HTTP and SOCKS can proxy the same traffic and get very different results about where that traffic goes. Similarly, it doesn't make sense to use an HTTP proxy for anything other than HTTP traffic. If you could clarify your request perhaps we could file a ticket? Otherwise it would be great if you could contribute a patch to advance one of those existing tickets. If you want a totally generic (i.e. TCP-level) proxy, Twisted does contain one: you can run 'twistd portforward --help' for information about how to run it. Good luck, -glyph
Myself and another are working on adding SOCKS client support to Twisted for use in Tor related projects. I'm also convinced there's an abstraction that would make it easier to implement proxy clients (or any transparent data/endpoint modifying protocol) in Twisted. I agree that it's difficult to generalise, but there's a class of protocol that does one or more of the following, transparently, and little else: - append data, before switching to application protocol (SSL handshake, SOCKS connect/bind) - prepend data, after losing connection on application protocol (SSL shutdown) - modify application data (SSL "recordification", encryption) - modify the endpoint (any proxy) Endpoints can do all these things - SSL4ClientEndpoint, for example, does the first three - but does it make sense to use Endpoints for this purpose? Endpoints aren't stackable, so no good if I want to run a SOCKS client over SSL. Also, with a proxied connection you have two Endpoints, one to the proxy server and one 'virtual Endpoint' to wherever. The Endpoints API doesn't accommodate this. Would it make sense to have Endpoint wrappers for this purpose? A sort of middleware (hate that word) to intercept transport events (including transport formation, i.e. Endpoint.connect()). I'm looking forward to being proven wrong on this as it will make my SOCKS work easier, or else if this turns out to be interesting then I have more ideas on how it should be implemented. Cheers, Peter.
On Aug 8, 2011, at 10:10 PM, Peter Le Bek wrote:
I think so.
Endpoints aren't stackable, so no good if I want to run a SOCKS client over SSL.
Sure they are. There's no SOCKS endpoint yet, but if you were to implement one, you could make it take another endpoint (or endpoint factory, as the case may be) as an argument to its constructor. The implementation of the various current SSL* enpdoints would set a better example if it weren't for the legacy connectSSL/listenSSL APIs. However, those APIs are being slowly phased out in favor of the endpoints themselves, as well as the internal implementation being moved to use a wrapping / delegation approach <http://twistedmatrix.com/trac/ticket/4854> between transports and protocols instead of OpenSSL's SSL-is-a-socket-except-when-it-isn't strategy.
How doesn't it? It's exactly this kind of API that endpoints were _designed_ to accommodate - the original impetus to create them was the Vertex project (now part of <https://launchpad.net/divmod.org/>), which proxies traffic in a variety of different ways.
It might make sense to have some utilities around this area. There are definitely parts of the process - especially producers and consumers - where you have to write a lot of boilerplate code to make sure everything's hooked up properly. But it's all certainly _possible_ now with the existing interfaces. You'd probably be interested in this ticket: <http://twistedmatrix.com/trac/ticket/3204>. Perhaps you could submit a patch? :)
It would help with that proof if you would be a bit more precise about where you think the current code falls short. -glyph
Also, anyone working on socks5 in twisted should be aware of this project: http://code.google.com/p/proxy65/ which implements a SOCK5 server. It's in the context of a Jabber/XMPP XEP-0065 proxy server, but a lot of it should be applicable, if just as an example. I make no guarantees about the quality and/or correctness of the code, as I've only skimmed it a couple of times. I do know it predates the endpoint API, and I doubt it has been updated. (getting SOCS5 support into twisted has been on my 'maybe someday" todo list, and I was planning on looking at this in more detail as a reference). It's MIT licensed. Kevin Horn On Mon, Aug 8, 2011 at 11:31 PM, Glyph Lefkowitz <glyph@twistedmatrix.com>wrote:
Pls, show me an example of HTTP authorization. Thanks all. __ http://itfreelancer.ru
On 06:50 am, lukas2k7@mail.ru wrote:
Pls, show me an example of HTTP authorization. Thanks all.
See the online documentation for Twisted Web: http://twistedmatrix.com/documents/current/web/howto/ in particular: http://twistedmatrix.com/documents/current/web/howto/web-in-60/http- auth.html Jean-Paul
participants (7)
-
Damiano ALBANI -
exarkun＠twistedmatrix.com -
Glyph Lefkowitz -
Itamar Turner-Trauring -
Kevin Horn -
Lukichev Anton -
Peter Le Bek