Mailman 3 August 2014 - Twisted

[Twisted-Python] ANN: Flocker 0.1, a data volume manager and multi-host Docker cluster management tool
by Itamar Turner-Trauring 15 Aug '14

15 Aug '14

Flocker is an open source data volume manager and multi-host Docker cluster management tool. With it you can control your data using the same tools you use for your stateless applications. This means that you can run your databases, queues and key-value stores in Docker and move them around as easily as the rest of your app. It's written with Twisted (of course) and features the work of Twisted developers Jean-Paul Calderone, Tom Prince, Richard Wall and myself. Very much a preliminary release though, so there's a bunch of code that needs to be Twisted-ified :) Documentation: http://docs.clusterhq.com/en/0.1.0/ Code: https://github.com/ClusterHQ/flocker

3 2

[Twisted-Python] Who's doing the merges for reviewed and blessed pull requests at twisted/nevow?
by Werner Thie 12 Aug '14

12 Aug '14

Mahalo for enlightening me, Werner

2 1

[Twisted-Python] On problems with inlineCallbacks (was Re: Is there pb documentation somewhere?)
by Jonathan Ballet 12 Aug '14

12 Aug '14

Sorry for hijacking the thread, but I actually was going to ask something about this a few days ago, so let's use this reference from Glyph as a starting point instead :) On Fri, Aug 08, 2014 at 02:59:39PM -0700, Glyph Lefkowitz wrote: > I've participated in this discussion several times: > > Hypothetical Amalgam of Median Interlocutors Speaking Here: "I'm using Tulip because I really like its style of coroutines." > Glyph: "That's interesting. Did you know that Twisted has an equivalent style of coroutines, called inlineCallbacks, that's been around for years?" > HAMISH: "I saw that, and I asked about that a while ago and I heard it was bad. It haven't heard that Tulip has the same problems, though." > Glyph: "Really? What problems does inlineCallbacks have that Tulip's coroutines don't?" > HAMISH: "When I asked about it everybody told me I have to use Deferreds instead, but Deferreds are really confusing and they make your code look all gross, so I didn't want to do that. With Tulip I don't have to!" > Glyph: <facepalm> > > Of course the problems that we describe with inlineCallbacks are the > exact same problems that you will have with Tulip-style coroutines, > and in fact in one of the conversations that was averaged out to > produce the above composite, my interlocutor specifically mentioned > that they'd already had the kind of bug that explicit-yield coroutines > can sometimes encourage (thoughtlessly putting in too many 'yield's > and not considering their consequences) and were wondering how Twisted > dealt with that sort of thing. It's not the first time I'm hearing that there are problems/and or limitations with inlineCallbacks, and I don't think I ever read somewhere what they were exactly? And so, each time I'm seeing code using inlineCallbacks, I'm frowning then realizing I have no idea why I did... Is there some place where I could find more information about these problems/limitations? That would be enlightening. Jonathan

3 2

Re: [Twisted-Python] Scalability of timers
by dw+twisted-python＠hmmz.org 11 Aug '14

11 Aug '14

On Sun, Aug 10, 2014 at 03:31:11PM -0700, Tobias Oberstein wrote: > Thanks a lot for those hints! I will read into this material. Just a final note.. a single no-fds call to select with a 0 timeout seems to take around 280ns on my Core 2. Presumably the better interfaces (e.g. epoll, but not poll) will also take around the same time. It's really hard to write even a single Python function that gets anywhere below 1usec CPU time, and given how function-heavy Twisted is, I'd be surprised considerations like this factored usefully into a design at all :) Adjusting timer coalescing to extreme settings might even worsen your app's performance, since it'll cause interpreter time and syscalls to all be compressed around the slack intervals, leaving the CPU idle more often, rather than running evenly spaced over time. This might produce less desirable app behavior overall (e.g. it has knock-on effects for network interface queues, bursts of disk IO/SQL queries, network switch buffer overruns, or whatever else). David

2 1

[Twisted-Python] Scalability of timers
by Tobias Oberstein 11 Aug '14

11 Aug '14

Hi, I have a question regarding scalability of timers in Twisted. Say I have a massive number of periodic timers (lets say each with period 1s, but all slightly time shifted to each other). As far as I understand, timers are implemented ultimately by setting the timeout parameter when calling into OS select/poll/epoll/kqueue. If this is true, then the number of timers scales linearly with the number of syscalls. This can get limiting (the total number of syscalls a Linux box can sustain is a couple of 100k's per second). As more and more timers are setup, the timeout essentially will approach 0. On the upside, timers will fire precisely. However, say I am fine with a precision of 1ms. Is there a way that limits the syscall rate to 1000/s (given no FD activity happens) _independently_ of the number of timers setup? Timers that fall into a certain ms slice would all fire roughly at the same time (still ordered). Is that possible? Thanks, Tobias

4 10

[Twisted-Python] Weekly Bug Summary
by trac＠twistedmatrix.com 10 Aug '14

10 Aug '14

1 0

[Twisted-Python] help with listening to UDP broadcast?
by Russell E. Owen 09 Aug '14

09 Aug '14

I've found an example for UDP broadcasting: <https://twistedmatrix.com/trac/browser/trunk/docs/projects/core/examples /udpbroadcast.py?rev=41516> However, it combines the sender and receiver in a way that I find confusing. I figured out how output UDP broadcast packets, but not how to make a client that receives to the packets. Here is what we have, which doesn't work. Based on a comment on the broadcast UDP ticket. I suspect the problem is specifying the broadcast address using the interface argument to listenUDP, but I'm not sure. In any case, no value for interface that I've tried works: -"<broadcast>" (which is what my senders uses) results in twisted.internet.error.InvalidAddressError -"255.255.255.255" results in twisted.internet.error.CannotListenError: Couldn't listen on 255.255.255.255:1235: [Errno 49] Can't assign requested address. - omitting it results in no packets received. I also tried listenMulticast, but it didn't work (and I didn't expect it to, based on comments I saw on the ticket for implementing UDP broadcast support). Any hints would be appreciated. #!/usr/bin/env python2 """Attempt to listen to UDP broadcasts """ from twisted.internet.protocol import DatagramProtocol from twisted.internet import reactor Port = 1235 class BroadcastUDPClient(DatagramProtocol): def startProtocol(self): self.transport.setBroadcastAllowed(True) def datagramReceived(self, datagram, address): print "got a UDP broadcast packet" class UDPListener(object): def __init__(self, port): self.port = port self.broadcastClient = BroadcastUDPClient() self.listener = None self.startListening() def startListening(self): if self.listener is None: interface = "255.255.255.255" self.listener = reactor.listenUDP(self.port, self.broadcastClient, interface) def stopListening(self): if self.listener is not None: self.listener.stopListening() self.listener = None if __name__ == "__main__": listener = UDPListener(Port) reactor.run()

2 1

[Twisted-Python] Embedding manhole interpreter into insults widget.
by Maxim Lacrima 08 Aug '14

08 Aug '14

Hello, I have been trying to create a widget that encloses manhole interpreter. Here is somewhat hacky implementation that I came up with at this moment: 1 from twisted.conch.insults import insults 2 from twisted.conch.insults import window 3 from twisted.conch.insults import helper 4 from twisted.conch import manhole 5 6 7 class TerminalBufferLastWrite(helper.TerminalBuffer): 8 9 lastWrite = '' 10 11 def write(self, bytes): 12 self.lastWrite = bytes 13 helper.TerminalBuffer.write(self, bytes) 14 15 for name, const in zip(insults._KEY_NAMES, insults.FUNCTION_KEYS): 16 setattr(TerminalBufferLastWrite, name, const) 17 18 19 class ManholeWidget(window.Widget): 20 21 def __init__(self, namespace, width, height): 22 self._buf = TerminalBufferLastWrite() 23 self._buf.width = width 24 self._buf.height = height 25 self._buf.connectionMade() 26 27 self.manholeProto = manhole.Manhole(namespace) 28 self.manholeProto.makeConnection(self._buf) 29 30 def keystrokeReceived(self, keyID, modifier): 31 super(ManholeWidget, self).keystrokeReceived(keyID, modifier) 32 self.manholeProto.keystrokeReceived(keyID, modifier) 33 self.repaint() 34 35 def render(self, width, height, terminal): 36 for y, line in enumerate(self._buf.lines[0:height]): 37 terminal.cursorPosition(0, y) 38 n = 0 39 for n, (ch, attr) in enumerate(line[0:width]): 40 if ch is self._buf.void: 41 ch = ' ' 42 else: 43 cursorRow = y 44 terminal.write(ch) 45 if n < width: 46 terminal.write(' ' * (width - n - 1)) 47 terminal.cursorPosition(self.manholeProto.lineBufferIndex + 4, 48 cursorRow) Basically, I substitute real terminal (`insults.ServerProtocol`) with slightly extended `TerminalBuffer`, which is used by manhole interpreter to write its output. `ManholeWidget.render` method is almost entirely reuses code from `window.Viewport`. This widget appears to work. However, here is the problem: if terminal size is large enough (say, 200x50), then there are some io lags (similar to ssh session over slow internet connection). The reason is that on each keystroke, the whole terminal buffer is redrawn. I wonder how I can optimize this. Currently I don't see a solution. Also I am wondering if I took right approach to embed manhole interpreter into a widget in the first place, but I don't see a solution, except using `TerminalBuffer` to capture manhole output. Thanks. -- Regards, Maxim

3 4

[Twisted-Python] Weekly Bug Summary
by trac＠twistedmatrix.com 03 Aug '14

03 Aug '14

1 0

[Twisted-Python] TerminalSession instance has no attribute 'windowChanged'
by Maxim Lacrima 01 Aug '14

01 Aug '14

Hello! When size of terminal window of ssh client is changed, then twisted ssh server throws an exception: exceptions.AttributeError: TerminalSession instance has no attribute 'windowChanged' `manhole_ssh.TerminalSession` doesn't implement this method, but according to ISession interface it should. Is this a bug? I use insults to create some curses-like ui and I need widgets to be redrawn when the size of window is changed. -- Regards, Maxim

4 3