Patch to telnetlib.py
The Telnet module telnetlib.py can be very slow -- unusably slow -- for large automated data transfers. There are typically done in raw mode. The attached patch greatly increased the speed of telnet interactions in raw mode. I submitted this a couple of year ago, but it was for an older branch of python. There are 2 key things being done: 1) concatenations string with string.join instead of '+' (which is probably a minor issue) 2) wholesale appending the raw and processed buffers when the IAC character is not found. The should be examined carefully since I am not an expert in the Telnet protocol, but it seems to work very well giving me a 5x speedup. --- installed/telnetlib.py 2010-02-02 22:57:58.000000000 -0500 +++ telnetlib.py 2010-03-13 12:17:02.000000000 -0500 @@ -30,6 +30,7 @@ - timeout should be intrinsic to the connection object instead of an option on one of the read calls only +Modified by G. Dudek for greater efficiency. """ @@ -420,6 +421,14 @@ """ buf = ['', ''] try: + if self.rawq: + if not IAC in self.rawq: + # speed hack, no IAC just grab whole queue. --Dudek + buf[self.sb] = "".join((buf[self.sb] , self.rawq)) + self.cookedq = "".join((self.cookedq , buf[0] )) + self.sbdataq = "".join((self.sbdataq , buf[1] )) + self.rawq_flush() + return while self.rawq: c = self.rawq_getchar() if not self.iacseq: @@ -428,7 +437,7 @@ if c == "\021": continue if c != IAC: - buf[self.sb] = buf[self.sb] + c + buf[self.sb] = "".join((buf[self.sb] , c)) continue else: self.iacseq += c @@ -480,8 +489,14 @@ self.iacseq = '' # Reset on EOF self.sb = 0 pass - self.cookedq = self.cookedq + buf[0] - self.sbdataq = self.sbdataq + buf[1] + self.cookedq = "".join((self.cookedq , buf[0] )) + self.sbdataq = "".join((self.sbdataq , buf[1] )) + + + def rawq_flush(self): + """ Set the queue to empty status """ + self.rawq = '' + self.irawq = 0 def rawq_getchar(self): """Get next char from raw queue. @@ -516,7 +531,7 @@ buf = self.sock.recv(50) self.msg("recv %r", buf) self.eof = (not buf) - self.rawq = self.rawq + buf + self.rawq = "".join((self.rawq,buf)) def sock_avail(self): """Test whether data is available on the socket."""
On 3/13/2010 12:24 PM, gregory dudek wrote:
The Telnet module telnetlib.py can be very slow -- unusably slow -- for large automated data transfers. There are typically done in raw mode.
The attached patch
Please submit to the tracker. If there is an existing issue, attach it to that.
On Sat, Mar 13, 2010 at 9:24 AM, gregory dudek <dudek@cim.mcgill.ca> wrote:
The Telnet module telnetlib.py can be very slow -- unusably slow -- for large automated data transfers. There are typically done in raw mode.
The attached patch greatly increased the speed of telnet interactions in raw mode. I submitted this a couple of year ago, but it was for an older branch of python.
For which Python version is your new patch?
There are 2 key things being done: 1) concatenations string with string.join instead of '+' (which is probably a minor issue)
This looks like superstition in the patch below -- there is no reason why "".join((a, b)) should be any faster than a+b. (The join trick is important when joining *many* strings, but for two it doesn't make a difference, both strings have to be copied at least once no matter what you try.)
2) wholesale appending the raw and processed buffers when the IAC character is not found. The should be examined carefully since I am not an expert in the Telnet protocol, but it seems to work very well giving me a 5x speedup.
This makes sense and is likely the reason your patch is faster. You should really submit this to the bug tracker at bugs.python.org.
--- installed/telnetlib.py 2010-02-02 22:57:58.000000000 -0500 +++ telnetlib.py 2010-03-13 12:17:02.000000000 -0500 @@ -30,6 +30,7 @@ - timeout should be intrinsic to the connection object instead of an option on one of the read calls only
+Modified by G. Dudek for greater efficiency.
You're gonna need to submit a Python contributor agreement (http://www.python.org/psf/contrib/contrib-form/) if/when your patch gets accepted. Welcome to the club!
"""
@@ -420,6 +421,14 @@ """ buf = ['', ''] try: + if self.rawq: + if not IAC in self.rawq: + # speed hack, no IAC just grab whole queue. --Dudek + buf[self.sb] = "".join((buf[self.sb] , self.rawq)) + self.cookedq = "".join((self.cookedq , buf[0] )) + self.sbdataq = "".join((self.sbdataq , buf[1] )) + self.rawq_flush() + return while self.rawq: c = self.rawq_getchar() if not self.iacseq: @@ -428,7 +437,7 @@ if c == "\021": continue if c != IAC: - buf[self.sb] = buf[self.sb] + c + buf[self.sb] = "".join((buf[self.sb] , c)) continue else: self.iacseq += c @@ -480,8 +489,14 @@ self.iacseq = '' # Reset on EOF self.sb = 0 pass - self.cookedq = self.cookedq + buf[0] - self.sbdataq = self.sbdataq + buf[1] + self.cookedq = "".join((self.cookedq , buf[0] )) + self.sbdataq = "".join((self.sbdataq , buf[1] )) + + + def rawq_flush(self): + """ Set the queue to empty status """ + self.rawq = '' + self.irawq = 0
def rawq_getchar(self): """Get next char from raw queue. @@ -516,7 +531,7 @@ buf = self.sock.recv(50) self.msg("recv %r", buf) self.eof = (not buf) - self.rawq = self.rawq + buf + self.rawq = "".join((self.rawq,buf))
def sock_avail(self): """Test whether data is available on the socket."""
_______________________________________________ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/guido%40python.org
-- --Guido van Rossum (python.org/~guido)
Can you create an issue on the bug tracker? Otherwise this will get lost. Eric. On 3/13/2010 12:24 PM, gregory dudek wrote:
The Telnet module telnetlib.py can be very slow -- unusably slow -- for large automated data transfers. There are typically done in raw mode.
The attached patch greatly increased the speed of telnet interactions in raw mode. I submitted this a couple of year ago, but it was for an older branch of python.
There are 2 key things being done: 1) concatenations string with string.join instead of '+' (which is probably a minor issue) 2) wholesale appending the raw and processed buffers when the IAC character is not found. The should be examined carefully since I am not an expert in the Telnet protocol, but it seems to work very well giving me a 5x speedup.
--- installed/telnetlib.py 2010-02-02 22:57:58.000000000 -0500 +++ telnetlib.py 2010-03-13 12:17:02.000000000 -0500 @@ -30,6 +30,7 @@ - timeout should be intrinsic to the connection object instead of an option on one of the read calls only
+Modified by G. Dudek for greater efficiency. """
@@ -420,6 +421,14 @@ """ buf = ['', ''] try: + if self.rawq: + if not IAC in self.rawq: + # speed hack, no IAC just grab whole queue. --Dudek + buf[self.sb] = "".join((buf[self.sb] , self.rawq)) + self.cookedq = "".join((self.cookedq , buf[0] )) + self.sbdataq = "".join((self.sbdataq , buf[1] )) + self.rawq_flush() + return while self.rawq: c = self.rawq_getchar() if not self.iacseq: @@ -428,7 +437,7 @@ if c == "\021": continue if c != IAC: - buf[self.sb] = buf[self.sb] + c + buf[self.sb] = "".join((buf[self.sb] , c)) continue else: self.iacseq += c @@ -480,8 +489,14 @@ self.iacseq = '' # Reset on EOF self.sb = 0 pass - self.cookedq = self.cookedq + buf[0] - self.sbdataq = self.sbdataq + buf[1] + self.cookedq = "".join((self.cookedq , buf[0] )) + self.sbdataq = "".join((self.sbdataq , buf[1] )) + + + def rawq_flush(self): + """ Set the queue to empty status """ + self.rawq = '' + self.irawq = 0
def rawq_getchar(self): """Get next char from raw queue. @@ -516,7 +531,7 @@ buf = self.sock.recv(50) self.msg("recv %r", buf) self.eof = (not buf) - self.rawq = self.rawq + buf + self.rawq = "".join((self.rawq,buf))
def sock_avail(self): """Test whether data is available on the socket."""
_______________________________________________ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/eric%2Ba-python-dev%40true...
On Sat, Mar 13, 2010 at 12:24 PM, gregory dudek <dudek@cim.mcgill.ca> wrote:
The Telnet module telnetlib.py can be very slow -- unusably slow -- for large automated data transfers. There are typically done in raw mode.
The attached patch greatly increased the speed of telnet interactions in raw mode. I submitted this a couple of year ago, but it was for an older branch of python.
There are 2 key things being done: 1) concatenations string with string.join instead of '+' (which is probably a minor issue) 2) wholesale appending the raw and processed buffers when the IAC character is not found. The should be examined carefully since I am not an expert in the Telnet protocol, but it seems to work very well giving me a 5x speedup.
As others mentioned, please post the bug to the tracker. Also, please assign the patch to me ("jackdied") and mention the previous bug number - I thought I had reviewed all the telnetlib bugs including those that were closed WONTFIX. Thanks for the kick in the pants, I have a whole new inner loop for data processing but I haven't applied it. I've been adding unit tests to the module so I could be sure I wouldn't break anything but never finished the job. -Jack
participants (5)
-
Eric Smith -
gregory dudek -
Guido van Rossum -
Jack Diederich -
Terry Reedy