Sorry for hijacking the thread, but I actually was going to ask
something about this a few days ago, so let's use this reference from
Glyph as a starting point instead :)
On Fri, Aug 08, 2014 at 02:59:39PM -0700, Glyph Lefkowitz wrote:
> I've participated in this discussion several times:
> Hypothetical Amalgam of Median Interlocutors Speaking Here: "I'm using Tulip because I really like its style of coroutines."
> Glyph: "That's interesting. Did you know that Twisted has an equivalent style of coroutines, called inlineCallbacks, that's been around for years?"
> HAMISH: "I saw that, and I asked about that a while ago and I heard it was bad. It haven't heard that Tulip has the same problems, though."
> Glyph: "Really? What problems does inlineCallbacks have that Tulip's coroutines don't?"
> HAMISH: "When I asked about it everybody told me I have to use Deferreds instead, but Deferreds are really confusing and they make your code look all gross, so I didn't want to do that. With Tulip I don't have to!"
> Glyph: <facepalm>
> Of course the problems that we describe with inlineCallbacks are the
> exact same problems that you will have with Tulip-style coroutines,
> and in fact in one of the conversations that was averaged out to
> produce the above composite, my interlocutor specifically mentioned
> that they'd already had the kind of bug that explicit-yield coroutines
> can sometimes encourage (thoughtlessly putting in too many 'yield's
> and not considering their consequences) and were wondering how Twisted
> dealt with that sort of thing.
It's not the first time I'm hearing that there are problems/and or
limitations with inlineCallbacks, and I don't think I ever read
somewhere what they were exactly? And so, each time I'm seeing code
using inlineCallbacks, I'm frowning then realizing I have no idea why I
Is there some place where I could find more information about these
problems/limitations? That would be enlightening.
On Sun, Aug 10, 2014 at 03:31:11PM -0700, Tobias Oberstein wrote:
> Thanks a lot for those hints! I will read into this material.
Just a final note.. a single no-fds call to select with a 0 timeout
seems to take around 280ns on my Core 2. Presumably the better
interfaces (e.g. epoll, but not poll) will also take around the same
It's really hard to write even a single Python function that gets
anywhere below 1usec CPU time, and given how function-heavy Twisted is,
I'd be surprised considerations like this factored usefully into a
design at all :)
Adjusting timer coalescing to extreme settings might even worsen your
app's performance, since it'll cause interpreter time and syscalls to
all be compressed around the slack intervals, leaving the CPU idle more
often, rather than running evenly spaced over time. This might produce
less desirable app behavior overall (e.g. it has knock-on effects for
network interface queues, bursts of disk IO/SQL queries, network switch
buffer overruns, or whatever else).
I have a question regarding scalability of timers in Twisted.
Say I have a massive number of periodic timers (lets say each with period 1s, but all slightly time shifted to each other).
As far as I understand, timers are implemented ultimately by setting the timeout parameter when calling into OS select/poll/epoll/kqueue.
If this is true, then the number of timers scales linearly with the number of syscalls. This can get limiting (the total number of syscalls a Linux box can sustain is a couple of 100k's per second). As more and more timers are setup, the timeout essentially will approach 0. On the upside, timers will fire precisely.
However, say I am fine with a precision of 1ms.
Is there a way that limits the syscall rate to 1000/s (given no FD activity happens) _independently_ of the number of timers setup?
Timers that fall into a certain ms slice would all fire roughly at the same time (still ordered).
Is that possible?
I've found an example for UDP broadcasting:
However, it combines the sender and receiver in a way that I find
confusing. I figured out how output UDP broadcast packets, but not how
to make a client that receives to the packets.
Here is what we have, which doesn't work. Based on a comment on the
broadcast UDP ticket. I suspect the problem is specifying the broadcast
address using the interface argument to listenUDP, but I'm not sure. In
any case, no value for interface that I've tried works:
-"<broadcast>" (which is what my senders uses) results in
-"255.255.255.255" results in twisted.internet.error.CannotListenError:
Couldn't listen on 255.255.255.255:1235: [Errno 49] Can't assign
- omitting it results in no packets received.
I also tried listenMulticast, but it didn't work (and I didn't expect it
to, based on comments I saw on the ticket for implementing UDP broadcast
Any hints would be appreciated.
"""Attempt to listen to UDP broadcasts
from twisted.internet.protocol import DatagramProtocol
from twisted.internet import reactor
Port = 1235
def datagramReceived(self, datagram, address):
print "got a UDP broadcast packet"
def __init__(self, port):
self.port = port
self.broadcastClient = BroadcastUDPClient()
self.listener = None
if self.listener is None:
interface = "255.255.255.255"
self.listener = reactor.listenUDP(self.port,
if self.listener is not None:
self.listener = None
if __name__ == "__main__":
listener = UDPListener(Port)
I have been trying to create a widget that encloses manhole interpreter.
Here is somewhat hacky implementation that I came up with at this moment:
1 from twisted.conch.insults import insults
2 from twisted.conch.insults import window
3 from twisted.conch.insults import helper
4 from twisted.conch import manhole
7 class TerminalBufferLastWrite(helper.TerminalBuffer):
9 lastWrite = ''
11 def write(self, bytes):
12 self.lastWrite = bytes
13 helper.TerminalBuffer.write(self, bytes)
15 for name, const in zip(insults._KEY_NAMES, insults.FUNCTION_KEYS):
16 setattr(TerminalBufferLastWrite, name, const)
19 class ManholeWidget(window.Widget):
21 def __init__(self, namespace, width, height):
22 self._buf = TerminalBufferLastWrite()
23 self._buf.width = width
24 self._buf.height = height
27 self.manholeProto = manhole.Manhole(namespace)
30 def keystrokeReceived(self, keyID, modifier):
31 super(ManholeWidget, self).keystrokeReceived(keyID, modifier)
32 self.manholeProto.keystrokeReceived(keyID, modifier)
35 def render(self, width, height, terminal):
36 for y, line in enumerate(self._buf.lines[0:height]):
37 terminal.cursorPosition(0, y)
38 n = 0
39 for n, (ch, attr) in enumerate(line[0:width]):
40 if ch is self._buf.void:
41 ch = ' '
43 cursorRow = y
45 if n < width:
46 terminal.write(' ' * (width - n - 1))
47 terminal.cursorPosition(self.manholeProto.lineBufferIndex +
Basically, I substitute real terminal (`insults.ServerProtocol`) with
slightly extended `TerminalBuffer`, which is used by manhole interpreter to
write its output. `ManholeWidget.render` method is almost entirely reuses
code from `window.Viewport`.
This widget appears to work.
However, here is the problem: if terminal size is large enough (say,
200x50), then there are some io lags (similar to ssh session over slow
The reason is that on each keystroke, the whole terminal buffer is redrawn.
I wonder how I can optimize this. Currently I don't see a solution. Also I
am wondering if I took right approach to embed manhole interpreter into a
widget in the first place, but I don't see a solution, except using
`TerminalBuffer` to capture manhole output.
When size of terminal window of ssh client is changed, then twisted ssh
server throws an exception:
exceptions.AttributeError: TerminalSession instance has no attribute
`manhole_ssh.TerminalSession` doesn't implement this method, but according
to ISession interface it should.
Is this a bug?
I use insults to create some curses-like ui and I need widgets to be
redrawn when the size of window is changed.