[Twisted-Python] twisted and python 2.1.3
I'm using twisted and some other python libraries (zope) and would like to use them together. Of course zope is still back in the 2.1.3 days of python. If I were to deal with defining True and False where needed would there be other compatibility issues with 2.1.3 and twisted? -EAD
On Mon, 22 Sep 2003 11:13:09 -0400 Erik A.Dahl <edahl@mindspring.com> wrote:
I'm using twisted and some other python libraries (zope) and would like to use them together. Of course zope is still back in the 2.1.3 days of python. If I were to deal with defining True and False where needed would there be other compatibility issues with 2.1.3 and twisted?
Yes, we are using other 2.2 features as well. So, either : 1. switch to older version of Twisted (1.0.4 had 2.1 support). 2. upgrade Zope to use 2.2 (current versions while not officially supported do run decently on 2.2 - ask around). 3. Wait for Zope 2.7 which runs on python 2.2 IIRC. 4. Run Twisted and Zope in different processes and communicate via XML-RPC or something like that. Personally I'd try for #4 in most cases. -- Itamar Shtull-Trauring http://itamarst.org/ Available for Python & Twisted consulting
I tried to use twisted python to write a proxy following the given socksv4 example. The problem i discovered is that: when i try to forward two connections, their capacities may not match. In a blocking implementation, data = source.read(); dest.write(data); In a non-blocking implementation, what socksv4 did is something like: def dataReceived(data): otherConn.transport.write(data) However, if source is a much faster link than dest, would there be a problem? write always return even it should block so that read always happens, and the buffer of source tcp connection will never get full, to let tcp start flow control? Thanks. Yun
On Wednesday, Sep 24, 2003, at 00:08 America/New_York, Yun Mao wrote:
I tried to use twisted python to write a proxy following the given socksv4 example.
The problem i discovered is that: when i try to forward two connections, their capacities may not match.
In a blocking implementation, data = source.read(); dest.write(data);
In a non-blocking implementation, what socksv4 did is something like:
def dataReceived(data): otherConn.transport.write(data)
However, if source is a much faster link than dest, would there be a problem? write always return even it should block so that read always happens, and the buffer of source tcp connection will never get full, to let tcp start flow control?
With that code, you are right. Twisted will buffer everything for dest.. so you won't lose any packets, but you might start using a lot of memory and source may think that dest has a faster connection than dest actually does. The twisted.internet.policies module can throttle transports, however I don't think that any of them can actually determine how fast dest is actually consuming the data. That would require introspecting the underlying transport's buffer and I don't believe the transport interface has any way of doing that. You could break the rules and assume that there is a dataBuffer attribute (from twisted.internet.abstract.FileDescriptor) which would most likely work, until you try and use it on a transport that is not a FileDescriptor (wrapped by another policy, is a loopback for testing, using a custom reactor, etc.). In any case, yes, it's possible to solve this problem, but you will have to write your own throttling policy which must bend the interface rules in its implementation. I would suggest writing the throttling policy code, and then posting your code with a description of your use case to http://twistedmatrix.com/bugs/ so there can be a discussion about how to best change Twisted's interfaces around a bit so that this kind of throttling policy can be implemented in a future release of Twisted without digging into objects beyond their declared interfaces. I could be mistaken, though. I'm not an expert on twisted.internet, but I poked around it a bit when writing my own reactor. -bob
Is it true that the only way to see the success of write() is to check the FileDescriptor.dataBuffer? I'm thinking that suppose there is one way to check this, and I can somehow add a callback function to the success of each write(), then I can simply call source.stopReading() after write() and startReading() when I get callback. Yun On Wed, 24 Sep 2003, Bob Ippolito wrote:
On Wednesday, Sep 24, 2003, at 00:08 America/New_York, Yun Mao wrote:
I tried to use twisted python to write a proxy following the given socksv4 example.
The problem i discovered is that: when i try to forward two connections, their capacities may not match.
In a blocking implementation, data = source.read(); dest.write(data);
In a non-blocking implementation, what socksv4 did is something like:
def dataReceived(data): otherConn.transport.write(data)
However, if source is a much faster link than dest, would there be a problem? write always return even it should block so that read always happens, and the buffer of source tcp connection will never get full, to let tcp start flow control?
With that code, you are right. Twisted will buffer everything for dest.. so you won't lose any packets, but you might start using a lot of memory and source may think that dest has a faster connection than dest actually does.
The twisted.internet.policies module can throttle transports, however I don't think that any of them can actually determine how fast dest is actually consuming the data. That would require introspecting the underlying transport's buffer and I don't believe the transport interface has any way of doing that.
You could break the rules and assume that there is a dataBuffer attribute (from twisted.internet.abstract.FileDescriptor) which would most likely work, until you try and use it on a transport that is not a FileDescriptor (wrapped by another policy, is a loopback for testing, using a custom reactor, etc.).
In any case, yes, it's possible to solve this problem, but you will have to write your own throttling policy which must bend the interface rules in its implementation.
I would suggest writing the throttling policy code, and then posting your code with a description of your use case to http://twistedmatrix.com/bugs/ so there can be a discussion about how to best change Twisted's interfaces around a bit so that this kind of throttling policy can be implemented in a future release of Twisted without digging into objects beyond their declared interfaces.
I could be mistaken, though. I'm not an expert on twisted.internet, but I poked around it a bit when writing my own reactor.
-bob
_______________________________________________ Twisted-Python mailing list Twisted-Python@twistedmatrix.com http://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python
-- Yun Mao Ph.D Student, University of Pennsylvania http://www.cis.upenn.edu/~maoy/
On Wed, Sep 24, 2003 at 01:16:06AM -0400, Yun Mao wrote:
Is it true that the only way to see the success of write() is to check the FileDescriptor.dataBuffer?
There's another way -- if a FileDescriptor is consumer of a producer, it will call the producer's pauseProducing when its buffer grows too large (look at the abstract.FileDescriptor.write source).
I'm thinking that suppose there is one way to check this, and I can somehow add a callback function to the success of each write(), then I can simply call source.stopReading() after write() and startReading() when I get callback.
You really don't want to do this manually -- you're just going to reinvent Twisted's producers/consumers infrastructure. See my other reply. -Andrew.
Is there any doc/example about producer/consumer? What is that streaming argument for? Thanks. -Y On Wed, 24 Sep 2003, Andrew Bennetts wrote:
On Wed, Sep 24, 2003 at 01:16:06AM -0400, Yun Mao wrote:
Is it true that the only way to see the success of write() is to check the FileDescriptor.dataBuffer?
There's another way -- if a FileDescriptor is consumer of a producer, it will call the producer's pauseProducing when its buffer grows too large (look at the abstract.FileDescriptor.write source).
I'm thinking that suppose there is one way to check this, and I can somehow add a callback function to the success of each write(), then I can simply call source.stopReading() after write() and startReading() when I get callback.
You really don't want to do this manually -- you're just going to reinvent Twisted's producers/consumers infrastructure. See my other reply.
-Andrew.
_______________________________________________ Twisted-Python mailing list Twisted-Python@twistedmatrix.com http://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python
On Wed, 24 Sep 2003 13:35:32 -0400 (EDT) Yun Mao <maoy@cis.upenn.edu> wrote:
Is there any doc/example about producer/consumer? What is that streaming argument for?
Streaming means push, non-streaming means pull. The only real docs available is http://itamarst.org/writings/OSCON03/twisted_internet-108.html -- Itamar Shtull-Trauring http://itamarst.org/ Available for Python & Twisted consulting
You won't be able to register a callback to it.. on second thought, you may be able to proxy writeSomeData and look at the result to see how much data got written, but I think that the reactor may have the actual FileDescriptor registered so you'd have to do some more trickery. On Wednesday, Sep 24, 2003, at 01:16 America/New_York, Yun Mao wrote:
Is it true that the only way to see the success of write() is to check the FileDescriptor.dataBuffer?
I'm thinking that suppose there is one way to check this, and I can somehow add a callback function to the success of each write(), then I can simply call source.stopReading() after write() and startReading() when I get callback.
Yun
On Wed, 24 Sep 2003, Bob Ippolito wrote:
On Wednesday, Sep 24, 2003, at 00:08 America/New_York, Yun Mao wrote:
I tried to use twisted python to write a proxy following the given socksv4 example.
The problem i discovered is that: when i try to forward two connections, their capacities may not match.
In a blocking implementation, data = source.read(); dest.write(data);
In a non-blocking implementation, what socksv4 did is something like:
def dataReceived(data): otherConn.transport.write(data)
However, if source is a much faster link than dest, would there be a problem? write always return even it should block so that read always happens, and the buffer of source tcp connection will never get full, to let tcp start flow control?
With that code, you are right. Twisted will buffer everything for dest.. so you won't lose any packets, but you might start using a lot of memory and source may think that dest has a faster connection than dest actually does.
The twisted.internet.policies module can throttle transports, however I don't think that any of them can actually determine how fast dest is actually consuming the data. That would require introspecting the underlying transport's buffer and I don't believe the transport interface has any way of doing that.
You could break the rules and assume that there is a dataBuffer attribute (from twisted.internet.abstract.FileDescriptor) which would most likely work, until you try and use it on a transport that is not a FileDescriptor (wrapped by another policy, is a loopback for testing, using a custom reactor, etc.).
In any case, yes, it's possible to solve this problem, but you will have to write your own throttling policy which must bend the interface rules in its implementation.
I would suggest writing the throttling policy code, and then posting your code with a description of your use case to http://twistedmatrix.com/bugs/ so there can be a discussion about how to best change Twisted's interfaces around a bit so that this kind of throttling policy can be implemented in a future release of Twisted without digging into objects beyond their declared interfaces.
I could be mistaken, though. I'm not an expert on twisted.internet, but I poked around it a bit when writing my own reactor.
-bob
_______________________________________________ Twisted-Python mailing list Twisted-Python@twistedmatrix.com http://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python
-- Yun Mao Ph.D Student, University of Pennsylvania http://www.cis.upenn.edu/~maoy/
_______________________________________________ Twisted-Python mailing list Twisted-Python@twistedmatrix.com http://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python
On Wed, Sep 24, 2003 at 12:08:48AM -0400, Yun Mao wrote:
In a non-blocking implementation, what socksv4 did is something like:
def dataReceived(data): otherConn.transport.write(data)
However, if source is a much faster link than dest, would there be a problem? write always return even it should block so that read always happens, and the buffer of source tcp connection will never get full, to let tcp start flow control?
Twisted will merrily buffer as much data as necessary if the connection isn't ready for writing. Unless you are talking about very large amounts of data, I doubt this would be a serious problem. I'm guessing here though; real benchmarks are welcome :) Still, if it is a problem -- or you're just sufficiently concerned to care about it anyway :) -- you could use producers and consumers to effectively throttle the upstream connection (the producer) when the downstream connection's (the consumer) buffers are full. Probably something like twisted.protocols.pcp.ProducerConsumerProxy would help here. -Andrew.
OK. Here are some numbers: First, I tried the bandwidth to download kernel from kernel.org: around 150K/s reported from wget. Then, I wrote a little program to use twisted.protocols.socks: application = Application("socks4 proxy") application.listenTCP(1080, socks.SOCKSv4Factory("/tmp/socks.log")) if '__main__' == __name__: application.run() I start the program on my own machine, port 1080. After that I use a "sockify" program (tsocks) to use the "127.0.0.1:1080" proxy to download. wget told me the speed is 4.10K/s Ideally, a proxy on localhost should bring almost 0 overhead, right? I also observe 100% cpu usage. A faster CPU may get better results. But, still, what is going on here? I modified socks.py and tried the producer idea, but it doesn't work. Maybe because the wrong usage? Somebody can help me out? Thanks a lot!
Twisted will merrily buffer as much data as necessary if the connection isn't ready for writing. Unless you are talking about very large amounts of data, I doubt this would be a serious problem. I'm guessing here though; real benchmarks are welcome :)
Still, if it is a problem -- or you're just sufficiently concerned to care about it anyway :) -- you could use producers and consumers to effectively throttle the upstream connection (the producer) when the downstream connection's (the consumer) buffers are full. Probably something like twisted.protocols.pcp.ProducerConsumerProxy would help here.
-Andrew.
_______________________________________________ Twisted-Python mailing list Twisted-Python@twistedmatrix.com http://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python
-- Yun Mao Ph.D Student, University of Pennsylvania http://www.cis.upenn.edu/~maoy/
Yun Mao wrote:
Somebody can help me out? Thanks a lot!
Wow! These are some really bad numbers. I don't know what to say except it looks like you're using things correctly. Can you run this under the Python profiler and attach the output to an issue in the bugtracker? The most likely answer is just that the SOCKS proxy has seen no optimization and is doing something silly like accidentally creating a huge number of string copies for no reason.
I've put the profiling result to http://www.twistedmatrix.com/users/roundup.twistd/twisted/file87/bugreport.t... Forgot to mention the env: gentoo, python 2.2.3, twisted 1.0.7 Yun On Wed, 24 Sep 2003, Glyph Lefkowitz wrote:
Wow! These are some really bad numbers. I don't know what to say except it looks like you're using things correctly. Can you run this under the Python profiler and attach the output to an issue in the bugtracker?
The most likely answer is just that the SOCKS proxy has seen no optimization and is doing something silly like accidentally creating a huge number of string copies for no reason.
It looks like the log function in line 141 is doing the silly thing. I disabled the log and it works fine now. :) Yun On Wed, 24 Sep 2003, Yun Mao wrote:
I've put the profiling result to
http://www.twistedmatrix.com/users/roundup.twistd/twisted/file87/bugreport.t...
Forgot to mention the env: gentoo, python 2.2.3, twisted 1.0.7
Yun
yay for profiling :) oh wow that's definitely going to be slow: f.write(string.join(map(lambda x:'%02X'%ord(x),p),' ')+' ') It's just that one line of code killing all your performance :) That little loop could (probably) be done MUCH more efficiently by using something like: map('%02X'.__mod__, struct.unpack('B' * len(data), data)) instead of doing it 16 at a time with lambda and ord. Just out of curiosity, how does your performance and CPU utilization look now, without logging on? I don't hear much about Twisted and performance.. Maybe it's not bad enough to complain about, or perhaps nobody has deployed twisted apps that see excessive load, but I haven't heard much either way. -bob On Wednesday, Sep 24, 2003, at 17:59 America/New_York, Yun Mao wrote:
It looks like the log function in line 141 is doing the silly thing. I disabled the log and it works fine now. :)
On Wed, 24 Sep 2003, Yun Mao wrote:
I've put the profiling result to
http://www.twistedmatrix.com/users/roundup.twistd/twisted/file87/ bugreport.txt
Forgot to mention the env: gentoo, python 2.2.3, twisted 1.0.7
On Wed, 24 Sep 2003, Bob Ippolito wrote:
look now, without logging on? I don't hear much about Twisted and performance.. Maybe it's not bad enough to complain about, or perhaps nobody has deployed twisted apps that see excessive load, but I haven't heard much either way.
-bob
The two download speed have no major difference now. The CPU load varies from 2% to 10% from me watching top. You may here from performance report (or complain) from me in the near future. :) -y
On Wed, 24 Sep 2003 18:30:06 -0400 Bob Ippolito <bob@redivi.com> wrote:
look now, without logging on? I don't hear much about Twisted and performance.. Maybe it's not bad enough to complain about, or perhaps
nobody has deployed twisted apps that see excessive load, but I haven't heard much either way.
Strakt were having some performance issues when sending large amounts of data, but JP's buffer() changes (went in in version 1.0.5? 6?) solved them. -- Itamar Shtull-Trauring http://itamarst.org/ Available for Python & Twisted consulting
On Wed, Sep 24, 2003 at 06:30:06PM -0400, Bob Ippolito wrote:
yay for profiling :)
oh wow that's definitely going to be slow: f.write(string.join(map(lambda x:'%02X'%ord(x),p),' ')+' ')
It's just that one line of code killing all your performance :) That little loop could (probably) be done MUCH more efficiently by using something like: map('%02X'.__mod__, struct.unpack('B' * len(data), data))
instead of doing it 16 at a time with lambda and ord.
Probably just "data.encode('hex')" would be fine? -Andrew.
On Thu, Sep 25, 2003 at 02:10:56PM +1000, Andrew Bennetts wrote:
On Wed, Sep 24, 2003 at 06:30:06PM -0400, Bob Ippolito wrote:
yay for profiling :)
oh wow that's definitely going to be slow: f.write(string.join(map(lambda x:'%02X'%ord(x),p),' ')+' ')
It's just that one line of code killing all your performance :) That little loop could (probably) be done MUCH more efficiently by using something like: map('%02X'.__mod__, struct.unpack('B' * len(data), data))
instead of doing it 16 at a time with lambda and ord.
Probably just "data.encode('hex')" would be fine?
Especially since str.__mod__ is a pretty recent development (ie, not in 2.2). Or one could ask why exactly the data needs to be hex encoded at all... Jp -- "Pascal is Pascal is Pascal is dog meat." -- M. Devine and P. Larson, Computer Science 340
participants (7)
-
Andrew Bennetts -
Bob Ippolito -
Erik A.Dahl -
Glyph Lefkowitz -
Itamar Shtull-Trauring -
Jp Calderone -
Yun Mao