Mailman 3 September 2013 - Twisted

[Twisted-Python] (no subject)
by Daniel Sank Sept. 26, 2013

Sept. 26, 2013

Dear Twisted users, I recently found myself implementing a design pattern that I think twisted.pb was specifically designed to address. I think I'm not using pb correctly so I'd like advice. This is a somewhat longish post because I need to describe the problem I'm trying to solve. I have done internet searches on Stack Overflow and this list but have not found the answer to my question. If I've missed something kindly direct me to the appropriate reference. I want to implement something functionally equivalent to a network chess game. I first consider how I would do this on a single computer with no network (maybe this is bad thinking). Each piece in the game is represented by an instance of class Agent. Each agent has a .graphics attribute which is an instance of a class from a GUI toolkit library or equivalent. Whenever an agent in the game needs to do something there will be business logic executed by the game objects proper (ie the agents) which will invoke methods on the .graphics objects to update the screen. This sort of structure seems natural as it allows easy integration of drag/drop, mouse click detection etc. It also nicely separates the real business logic from the GUI. Now I want to run over the network. The question is how should I set up references between the client and server objects? Surely the server will maintain a set of objects representing the pieces in the game. It seems reasonable that each user's program will have a corresponding set of objects (with .graphics attributes). The issue is, what do we mean by "corresponding" and how do these objects talk to one another? Following is my idea so far: Each instance of AgentClient has a .server attribute which is a remote reference to an instance of AgentServer, and each instance of AgentServer has a .clients attribute which is a list of remote references to instances of AgentClient. class AgentServer(pb.referenceable): def remote_move(self, targetSquare): """Handle move request from client""" if self.thisMoveIsLegal(targetSquare): self.position = targetSquare for client in self.clients: client.callRemote("move", targetSquare) def thisMoveIsLegal(self, targetSquare): <check that this is a legal move> class AgentClient(pb.referenceable): def requestMove(self, targetSquare): """Tell server we'd like to move""" self.server.callRemote("move", targetSquare) def remote_move(self, targetSquare): """Server told us we moved""" self.position = targetSquare self.graphics.setNewPosition(targetSquare) This isn't THAT bad. The client's requestMove is thin and unecessary (I put it there for illustration). Still I need to have two separate classes with corresponding methods to handle moving the piece. This seems like the kind of thing I could twisted.pb to solve more cleanly if I only would look in the right place. This problem gets even worse when I think about how to birth new in-game objects. It would have to look like this: class PlayerServer(pb.referenceable): def newAgent(self, asker): """Client told us it wants a new Agent""" if self.thisIsLegal(): a = AgentServer() self.agents.append(a) for client in self.clients: d = client.callRemote("newAgent", a) d.addCallback(lambda obj: a.clients.append(obj)) class PlayerClient(bp.referenceable): def requestNewAgent(self): """Tell the server we want to spawn a new Agent""" self.server.callRemote("newAgent", self) def newAgent(self, serverObj): a = AgentClient() self.agents.append(a) a.server = serverObj return a This just looks wrong. Any advice? Thank you in advance for your help. Regards, Daniel Sank

1 0

[Twisted-Python] Running unit tests without trial?
by Russell E. Owen Sept. 25, 2013

Sept. 25, 2013

Is it possible to run twisted.trial unit tests using python instead of trial, i.e.: "python mytest.py" instead of "trial mytest.py"? I'm hoping there is some simple magic I can put into the unit test itself to make this work. Perhaps something in if __name__ == "__main__": ...? to make it start up the reactor and run it correctly. I ask because I'm using a build environment that runs all unit tests this way. It does the sniffing out and the saving of logs, but it insists on running .py tests using python. -- Russell P.S. being *able* to run trial unit tests using python seems like a feature to me in any case -- one I'd be willing to add extra code to support. Trial has some nice features, but I find it confusing that it is *required* to run a single test with a ".py" extension when visually it looks like a python file.

2 2

Re: [Twisted-Python] Running unit tests without trial?
by Julian Berman Sept. 25, 2013

Sept. 25, 2013

On Wed, 25 Sep 2013 18:45:44 -0000 exarkun(a)twistedmatrix.com wrote: > On 06:15 pm, rowen(a)uw.edu wrote: > >Is it possible to run twisted.trial unit tests using python instead of > >trial, i.e.: "python mytest.py" instead of "trial mytest.py"? > > Sure. Here's the worst case answer (contents of mytest.py): > > import os > os.system("trial myrealtests.py") There is also http://twistedmatrix.com/trac/ticket/4797 Cheers, Julian

1 0

[Twisted-Python] Weekly Bug Summary
by trac＠twistedmatrix.com Sept. 22, 2013

Sept. 22, 2013

1 0

[Twisted-Python] How to disconnect an HTTP connection and notify the far end
by Bob Novas Sept. 20, 2013

Sept. 20, 2013

I've got a question on how to explicitly disconnect a connection from the client side so that the server side knows that the connection has been disconnected. I'd be happy if this worked even with no guarantees, i.e., only when the client and server were in fact well-connected. It's just an optimization to allow the server to cleanup. I've got an HTTP client that uses HTTPConnectionPool to get an Agent, gets a Request from the Agent, and sends the Request to a WebServer on a linux box. On a Mac client, if I call closeCachedConnections() on the HTTPConnectionPool, the client immediately sends FIN on the port with the Request outstanding, the server delivers a callback registered on the Request via NotifyFinish() and the server knows the client has disconnected. On a Windows client, if I do the same, same code, the client waits until the outstanding Request finishes and then sends FIN. Nothing happens at the server - there's no Request outstanding on which to deliver a callback, twisted.internet.tcp.Port.connectionLost() doesn't get called, nada. Am I missing something? Is there a way to send a FIN on a connection on which there is an outstanding Request on Windows? Thanks in advance, Bob

2 1

[Twisted-Python] Cancelling a pb callRemote
by Phil Mayers Sept. 19, 2013

Sept. 19, 2013

Is there any support, or any interest in / way to add support for deferred cancellation in pb? Specifically I'd like do have something like: class blah(pb...): def remote_somelongtask(self, args): ... taskid = somefunc() d = defer.Deferred(lambda : canceltask(taskid)) return d ...and on the client: d = callRemote('somelongtask', args) ... # now we no longer need it d.cancel() ...and have the cancel the server-side task.

6 11

[Twisted-Python] Twisted at PyConUK 2013
by Richard Wall Sept. 19, 2013

Sept. 19, 2013

Hello everyone, I'm giving a talk about Twisted and Twisted Names at PyConUK this year. The schedule hasn't been fixed yet, but it looks like it'll be this Friday 20 September at 1530. If you're going to PyConUK I'd welcome your support, especially with fielding any tricky audience questions. ;-) * http://pyconuk.net/Schedule I'm also hoping to organise a Twisted Sprint, so if you're still around on Monday come and find me. * http://pyconuk.net/Sprints -RichardW. PS. I'd love to merge https://tm.tl/5675 before the conference, so if you're familiar with the DNS RFCs, I'd really appreciate a code review. I'm rwall on #twisted-dev and #twisted if you want to discuss it

4 6

[Twisted-Python] run python application with twistd -y ifinger.tac, error occurs. please help to check
by 吴志高 Sept. 18, 2013

Sept. 18, 2013

Experts: 您好！ my ifinger.tac file content like following: fromtwisted.applicationimportinternet, servicefromtwisted.internetimportprotocol, reactor, deferfromtwisted.protocolsimportbasicclassFingerProtocol(basic.LineReceiver): deflineReceived(self, user): d = self.factory.getUser(user) defonError(err): return'Internal error in server'd.addErrback(onError) defwriteResponse(message): self.transport.write(message + '\r\n') self.transport.loseConnection() d.addCallback(writeResponse) classFingerFactory(protocol.ServerFactory): protocol = FingerProtocoldef__init__(self, **kwargs): self.users = kwargsdefgetUser(self, user): returndefer.succeed(self.users.get(user, "No such user")) application = service.Application('finger', uid=1, gid=1) factory = FingerFactory(moshez='Happy and well') internet.TCPServer(79, factory).setServiceParent(service.IServiceCollection(application)) when running this script with command-line : twistd -y ifinger.tac, error occurs like : Traceback (most recent call last): File "/usr/local/bin/twistd", line 7, in <module> import _preamble ImportError: No module named '_preamble' During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/usr/local/bin/twistd", line 9, in <module> sys.exc_clear() AttributeError: 'module' object has no attribute 'exc_clear' And i run python3 in interactive mode to check if exc_clear method is defined in sys, command line like : >>>import sys >>>sys.__dict__["exc_clear"] Traceback (most recent call last): File "<stdin>", line 1, in <module> sys.exc_clear() KeyError: 'exc_clear' >>> Is there any configuration i should do ? please help to give me an answer. 谢谢！！ ---------------------------- 姓名 : 吴志高/Charles 电话 : (+086)13410859946 邮箱 : wuzhigaoem(a)163.com ================================================

3 5

[Twisted-Python] infrastructure hosting confusion / cleanup
by Glyph Sept. 18, 2013

Sept. 18, 2013

Pages like this one: <http://buildbot.twistedmatrix.com/builders/pyflakes/builds/1008> information about the source code being built in the form of a git revision. As long as we're still using Subversion for our upstream VCS, it would be desirable to also have a Subversion revision number. I wanted to request this, but even I have lost track, at this point, of where to file bugs for things like this. It looks like we're hosting our infrastructure projects on Github right now, which means <https://github.com/twisted-infra/twisted-buildbot-configuration> would be the place to go. But there is no 'issues' link there. Nor any link to any place where bugs might be filed. I guess <https://bugs.launchpad.net/twisted-buildbot-configuration> is still what is being used for that thing? But then that project provides a non-up-to-date view of the source code. Richard Wall's draft for a new "new ticket" page directs people to file issues against twisted-infra but that appears to be wrong in this particular case, since the buildbot configuration doesn't have a bugtracker there. <https://twistedmatrix.com/trac/wiki/users/rwall/DraftNewTicketPage> Plus, that suggestion links to a list of like a zillion infrastructure projects, half of which don't even have a description that explains what they are, let alone where to file bugs. Can someone please volunteer to update all of the READMEs on all of these projects, and all of the various locations where they've ever lived, to point at the actual, canonical place where infrastructure is developed? It would be good if users who want to report that the website is down or ask for an enhancement to the website, or Trac or Buildbot instances did not have to select between a dozen different projects that they might file an issue against. (Perhaps they should all link to <https://twistedmatrix.com/trac/wiki/Infrastructure> or some page beneath it, which can _remain_ the canonical location for everything and be updated once, since based on our history it seems likely that half of our infrastructure will randomly move to Bitbucket, or perhaps to Sourceforge, at some point in the future...) If someone who has not previously participated in infrastructure development would like to take point on creating such a resource, I will be happy to get you the appropriate access to any systems you need to. -glyph

1 0

[Twisted-Python] logging
by Glyph Sept. 18, 2013

Sept. 18, 2013

Hello everybody. Today I'd like to talk about everyone's favorite subsystem within Twisted: logging. There are a number of problems with Twisted's logging system. The general theme of these problems is that Twisted log system was designed with some intended use-cases in mind, but its implementation was stopped halfway through and none of those use-cases were really well satisfied. At various conferences, and in person at various times, we've gotten together to express frustration at the logging system, but I don't think that the problems with it have ever been really thoroughly written down. Wilfredo has been working on logging here: http://trac.calendarserver.org/browser/CalendarServer/trunk/twext/python/lo…. This began as an attempt to just address some issues with our own logging wrappers, but has been growing into a design that is more fully capable of replacing logging within Twisted wholesale. I hope one of us can contribute it to Twisted soon, but in the meanwhile, please feel free to provide feedback - hopefully if some people can provide a bit of informal feedback now, the formal review process can be shorter and more pleasant :). Now, back to some explanation of the current problems. The main intended use-case for Twisted's logging system is to avoid doing unnecessary work in the case where it is not required. One of the reasons we even still have our own logging system is that the stdlib logging module is notorious for doing too much work per log message, and doing it too prematurely. One aspect of this is that logging would just use the existing 'kwargs' dictionary as its message type, and not actually do any formatting; that is, string-copying, until the message was logged. On Python 1.5.2, that meant no need to allocate anything since the interpreter needed a dict for the kwargs anyway! Woohoo premature optimization! Obviously, at least on PyPy, that optimization is no longer relevant (and we don't have performance tests for logging anyway). But, I don't think we should change that because a dict is still a good object for the sort of extensibility that log messages need. But, we have never written any filtering logic, nor have we instrumented any of our existing log messages with formatting stuff. Every message with a format string just goes to the log. For a long time, there's been consensus among at least a few core Twisted developers that log messages ought to be specified in terms of a format message and several relevant keyword arguments, so that tools can extract the values of the keyword arguments without parsing ad-hoc messages or using regexes. In other words, like this: log.msg(format="Foo happened. Relevant information includes: " "bar=%(bar)s baz=%(baz)s boz=%(boz)s", bar=bar, baz=baz, boz=boz) Yet, nothing within Twisted itself does this. There's a bit of a chicken and egg problem: since we have no tools for filtering log messages based on their structured values, there's no obvious motivation to break the structured contents of a message out into those values. And since no messages break out their values into nice structured fields, there's no motivation to write filtering tools that inspect those fields. And, even if we did have filtering, we don't have a facility to easily defer work. You could, of course, just do something like this: class GenerateOnlyIfNeeded(object): def __str__(self): return doSomethingExpensiveToGenerateAString() log.msg(format="debugging: %(expensive)s", logLevel=debug, expensive=GenerateOnlyIfNeeded()) But that's an awful lot of thinking for a single log message, not to mention the temporary class and the method you needed to allocate. Such deferral of work should be easy. And, speaking of the things that one might filter on, log messages don't typically include information about what module and class they're from; or, for that matter, what object is logging them. This information should be included by default in idiomatic usage of the API, so that a filtering system can inspect them. Another reason that we - both Twisted itself and Twisted users - don't typically pass keyword arguments and a format string, is that the default argument to msg is, simply, a "message". (It also does a dumb thing where it tries to emulate the argument-passing convention of 'print' and stringify its arguments and join them with spaces; we should get rid of that.) Instead, that argument always ought to be a format string. We should also have a log writer within Twisted that preserves structure, and dumps the logs using JSON (or similar, perhaps more than one of these; AMP would be another good thing). It's also not possible to use format strings when you are writing an error log message in Twisted; instead, 'why' is treated completely differently than 'message'. The 'why' argument to 'err', which should be mandatory (what were you doing when you tried to log a message?) is optional, which means that error messages are even more inscrutable. The portions of the dictionary used by the logging system are not namespaced in any way, even by convention. Therefore it's easy for a user to conflict with the keys used by the system. If users of Twisted actually used the system as intended more pervasively, I'm pretty sure that we would end up seeing people conflict with 'time' and 'system' arguments often. Speaking of those keys: the 'time' key is unconditionally overwritten by log.msg, which means that re-publishing an event that happened previously is destructive; there's no way to track historical time. Messages logged before the logging system starts are just lost. It would be good for the initial log observer to keep a (limited-size, of course) buffer of log messages which could be dumped upon failing. This would make it easier, for example, to debug tap plugins that use the log system to report something about their state during makeService. (And, as the maintainer of a phenomenally complex makeService, I can tell you that this would definitely be helpful.) The naming of methods is bad and confusing. msg? err? Easy to type, but hard to understand. We could have equally short names that are not confusing. The contract between parts of the system is poorly specified. For example, trial -j has a hard time with log output that includes any non-bytes keys. I imagine this will only become worse when trial actually supports Python 3. And speaking of encoding, the log system is written in terms of bytes instead of in terms of unicode. Now, there's good reason to be particularly permissive of garbage data in the logging system (logging a message should never provoke an error), but at some point - probably before the data gets to user-written observers - everything should be normalized into a structure that has enough properties that the user can depend on it. On that note, when presented with unicode encoding errors, Twisted's logger will directly raise an exception back to application code, something that it will otherwise only do when presented with KeyboardInterrupt. That I just don't understand. We use old, semi-deprecated '%' formatting style. If we're going to introduce a new API, we should use PEP 3101 formatting; in addition to being new and improved instead of old and inferior, it's a lot easier to type {foo} correctly than to type %(foo)s without forgetting the s. The whole LogContext thing is poorly designed. Mea culpa; this was entirely my fault. One problem with its design was that it was somewhat poorly explained, and so its intent has not carried over years of maintenance. The idea was supposed to be that every event-source within Twisted would provide a useful system key, which would indicate what event-source object (TCP connection, listening port, timed event, UDP port) was being activated when the given message was logged. However, a lot of factors conspire to make this information less than useful. For one thing, especially in database-backed applications, or those that are careful about rate-limiting or flow-control, nothing ever happens except in response to a callFromThread or a callLater; this means that the system argument is always "-". I think that the feature this is trying to support needs to be fundamentally re-thought (and in the intervening decade I've had a lot of ideas about how it might work better). Maybe we can build on some of Chris Armstrong's recent work on Deferred debugging in tracking asynchronous context in Deferreds to help identify the real cause(s) of work being executed. OK, I think that's it for now. (NB: The implementation linked above doesn't address all of the issues I've enumerated in this message, and I don't think that it should address every single one of them before being added to Twisted; writing this one message took me long enough, but we should possibly also discuss which ones are fundamental issues with the logging system and which are features that should be considered separately.) -glyph

12 40