[Twisted-Python] Embedding manhole interpreter into insults widget.

Hello, I have been trying to create a widget that encloses manhole interpreter. Here is somewhat hacky implementation that I came up with at this moment: 1 from twisted.conch.insults import insults 2 from twisted.conch.insults import window 3 from twisted.conch.insults import helper 4 from twisted.conch import manhole 5 6 7 class TerminalBufferLastWrite(helper.TerminalBuffer): 8 9 lastWrite = '' 10 11 def write(self, bytes): 12 self.lastWrite = bytes 13 helper.TerminalBuffer.write(self, bytes) 14 15 for name, const in zip(insults._KEY_NAMES, insults.FUNCTION_KEYS): 16 setattr(TerminalBufferLastWrite, name, const) 17 18 19 class ManholeWidget(window.Widget): 20 21 def __init__(self, namespace, width, height): 22 self._buf = TerminalBufferLastWrite() 23 self._buf.width = width 24 self._buf.height = height 25 self._buf.connectionMade() 26 27 self.manholeProto = manhole.Manhole(namespace) 28 self.manholeProto.makeConnection(self._buf) 29 30 def keystrokeReceived(self, keyID, modifier): 31 super(ManholeWidget, self).keystrokeReceived(keyID, modifier) 32 self.manholeProto.keystrokeReceived(keyID, modifier) 33 self.repaint() 34 35 def render(self, width, height, terminal): 36 for y, line in enumerate(self._buf.lines[0:height]): 37 terminal.cursorPosition(0, y) 38 n = 0 39 for n, (ch, attr) in enumerate(line[0:width]): 40 if ch is self._buf.void: 41 ch = ' ' 42 else: 43 cursorRow = y 44 terminal.write(ch) 45 if n < width: 46 terminal.write(' ' * (width - n - 1)) 47 terminal.cursorPosition(self.manholeProto.lineBufferIndex + 4, 48 cursorRow) Basically, I substitute real terminal (`insults.ServerProtocol`) with slightly extended `TerminalBuffer`, which is used by manhole interpreter to write its output. `ManholeWidget.render` method is almost entirely reuses code from `window.Viewport`. This widget appears to work. However, here is the problem: if terminal size is large enough (say, 200x50), then there are some io lags (similar to ssh session over slow internet connection). The reason is that on each keystroke, the whole terminal buffer is redrawn. I wonder how I can optimize this. Currently I don't see a solution. Also I am wondering if I took right approach to embed manhole interpreter into a widget in the first place, but I don't see a solution, except using `TerminalBuffer` to capture manhole output. Thanks. -- Regards, Maxim

On Aug 6, 2014, at 11:14 AM, Maxim Lacrima <lacrima.maxim@gmail.com> wrote:
The reason is that on each keystroke, the whole terminal buffer is redrawn. I wonder how I can optimize this. Currently I don't see a solution. Also I am wondering if I took right approach to embed manhole interpreter into a widget in the first place, but I don't see a solution, except using `TerminalBuffer` to capture manhole output.
Optimizing this will almost certainly involve patching Twisted. Which you should definitely do :). The reason the entire screen is redrawn is that a general implementation of a widget, i.e. one that can be displayed anywhere on the terminal, you do just have to redraw the entire widget when certain things happen, like when the screen scrolls. You can see this in other software by observing the fairly significant performance difference between a vertical split and a horizontal split in tmux. Nothing to do with Twisted there, just a general limitation of terminals. The optimizations you could implement are one where inputting individual characters doesn't cause a full screen redraw, and avoiding a full screen redraw when scrolling if the edges of the widget touch the edges of the full terminal (setting up a scroll area for that case). I think that using a widget for this case does indeed make sense (just as I'm super glad that tmux has vertical splits, even if they're kinda slow), and it would be great to get these optimizations into Twisted. -glyph

On 6 August 2014 22:48, Glyph Lefkowitz <glyph@twistedmatrix.com> wrote:
Optimizing this will almost certainly involve patching Twisted. Which you should definitely do :).
The reason the entire screen is redrawn is that a *general* implementation of a widget, i.e. one that can be displayed anywhere on the terminal, you do just have to redraw the entire widget when certain things happen, like when the screen scrolls. You can see this in other software by observing the fairly significant performance difference between a vertical split and a horizontal split in tmux. Nothing to do with Twisted there, just a general limitation of terminals.
The optimizations you could implement are one where inputting individual characters doesn't cause a full screen redraw, and avoiding a full screen redraw when scrolling if the edges of the widget touch the edges of the full terminal (setting up a scroll area for that case).
When manhole is used directly, then it is very responsive, and the performance doesn't degrade with increase of screen size. It holds only what user entered and writes the input line and command output directly to its transport, without bothering itself with processing of the whole screen. When I embed manhole into the widget like above, then the performance suffers. I think this is due to `TerminalBuffer` having to hold whole widget area (columns * rows) in memory, even if only a small portion of it is actually occupied by user input and command output. Rendering the whole buffer on each keystroke is slow. After experimenting a little bit, I think optimizations that you described mostly apply to `ManholeWidget.render` method, which shouldn't traverse the whole buffer on each keystroke. Currently I don't see how this involves patching widget implementation in Twisted. What is more important to me is that `TerminalBuffer` implementation is slightly incomplete. I think it misses color support, because I couldn't make it work with `ColoredManhole`.
-glyph
_______________________________________________ Twisted-Python mailing list Twisted-Python@twistedmatrix.com http://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python
-- Regards, Maxim

On 6 Aug, 06:14 pm, lacrima.maxim@gmail.com wrote:
Hello,
I have been trying to create a widget that encloses manhole interpreter. Here is somewhat hacky implementation that I came up with at this moment:
Widgets and the manhole interpreter are implemented using two pretty distinct approaches. As you've figured out, these approaches can be integrated (at least in one direction) using a terminal emulator - `twisted.conch.insults.helper.TerminalBuffer`. However, `TerminalBuffer` is an extremely low quality terminal emulator. Among its many flaws is that it uses a number of incredibly inefficient implementations for common terminal operations (many of its other flaws are along the lines of "it has a bug in the implementation of X"). If I were going to create a manhole widget then I would do it "natively" - not by bridging the existing manhole protocol to the widget system with TerminalBuffer but by writing a new widget that does manhole stuff all by itself. I suggest that you might want to pursue this approach instead of trying to optimize and fix bugs in TerminalBuffer. Ultimately it would be nice if a widget-based manhole were *the* manhole and the current implementation went away. However, that said, as far as I know, no vt-style terminals support efficient redrawing for the "vertically scroll one portion of the terminal separated from another portion by a vertical divider". You will be able to avoid the redundant work of re-rendering the Python buffer representing the terminal state but you'll still have to re- transmit the bytes representing the entire terminal display to the terminal on each redraw - unless you avoid vertical dividers. Jean-Paul

Hi, Thanks for your explanation. Now I understand a lot better how things work. I will try to implement "native" manhole widget. I think this will be a good exercise for me to learn how terminals and `twisted.conch` work. Thanks. On 7 August 2014 14:53, <exarkun@twistedmatrix.com> wrote:
On 6 Aug, 06:14 pm, lacrima.maxim@gmail.com wrote:
Hello,
I have been trying to create a widget that encloses manhole interpreter. Here is somewhat hacky implementation that I came up with at this moment:
Widgets and the manhole interpreter are implemented using two pretty distinct approaches.
As you've figured out, these approaches can be integrated (at least in one direction) using a terminal emulator - `twisted.conch.insults.helper. TerminalBuffer`.
However, `TerminalBuffer` is an extremely low quality terminal emulator. Among its many flaws is that it uses a number of incredibly inefficient implementations for common terminal operations (many of its other flaws are along the lines of "it has a bug in the implementation of X").
If I were going to create a manhole widget then I would do it "natively" - not by bridging the existing manhole protocol to the widget system with TerminalBuffer but by writing a new widget that does manhole stuff all by itself.
I suggest that you might want to pursue this approach instead of trying to optimize and fix bugs in TerminalBuffer.
Ultimately it would be nice if a widget-based manhole were *the* manhole and the current implementation went away.
However, that said, as far as I know, no vt-style terminals support efficient redrawing for the "vertically scroll one portion of the terminal separated from another portion by a vertical divider".
You will be able to avoid the redundant work of re-rendering the Python buffer representing the terminal state but you'll still have to re- transmit the bytes representing the entire terminal display to the terminal on each redraw - unless you avoid vertical dividers.
Jean-Paul
_______________________________________________ Twisted-Python mailing list Twisted-Python@twistedmatrix.com http://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python
-- Regards, Maxim
participants (3)
-
exarkun@twistedmatrix.com
-
Glyph Lefkowitz
-
Maxim Lacrima