Mailman 3 September 2008 - Twisted

Re: [Twisted-Python] Question on default t.p.log output
by Terry Jones Sept. 30, 2008

Sept. 30, 2008

>>>>> "glyph" == glyph <glyph(a)divmod.com> writes: glyph> Thank you for at least noticing this ;-). The 1K char log lines were hard to miss :-) glyph> The logging code is quite ugly, but unfortunately, almost nobody glyph> cares. If you could dedicate any time at all to improving it, even glyph> a little, we'd all really appreciate it. You'd all appreciate it but OTOH almost nobody cares??? Hmmm. Hard to resist :-) glyph> The way the log prefix is calculated is based on what actual event is glyph> firing from the reactor. So, whatever object *generates* the callback glyph> is the one whose log prefix will be used. But I also see lines like this: 2008/09/29 23:56 +0200 [-] Starting factory <HTTPClientFactory: http://ec2-75-101-219-127.compute-1.amazonaws.com:8001/dummy> 2008/09/29 23:56 +0200 [Uninitialized] Stopping factory <HTTPClientFactory: http://ec2-75-101-219-127.compute-1.amazonaws.com:8001/dummy> And - and Uninitialized do not have any clear correspondence with reactor events. (BTW, I do see where these are coming from.) glyph> It would definitely be nice if the log could follow the asynchronous glyph> thread of execution. Patches accepted :). Yes, agreed that would be nice. From my memory of defer.py... (insert vague comment about how this might not be so hard - for you :-)). glyph> It sounds like you have several complaints here. glyph> 1.) You don't want logs to go to stdout. I do want that, that's why I used sys.stdout. glyph> 2.) Factories are too noisy by default. glyph> Knowing when factories start up and shut down can actually be incredibly glyph> useful. It's a pain when they're filling up your logs with spam, but a glyph> better solution would be some kind of structured log format with a log glyph> viewer application, not destroying the information so that viewing logs glyph> with 'less' is easier. Right, and so viewing them when running a script from the command line that wants to do some logging to stdout is also fine. glyph> 4.) The log prefix isn't always useful or correct. glyph> This may be true, but we'd need a ticket filed that includes a glyph> complete enumeration of exactly _how_ they aren't useful or correct, glyph> and each case would need to be fixed. OK, thinking about that. glyph> 5.) The log prefix for Deferred callbacks doesn't follow the glyph> asynchronous origin of the callback. glyph> Figuring out how to do this is really tricky, because in different cases glyph> you want different things. However, twisted.python.context provides a glyph> hook for you to potentially capture some state when Deferreds, at least glyph> certain Deferreds, are spun up. If you can come up with a really clear glyph> statement of requirements for a particular log prefix, you can probably glyph> get it. And this. glyph> 6.) The HTTP client is too noisy and should emit fewer log messages. glyph> Possibly - but then I recently discovered some very interesting glyph> information by watching a server which had a bunch of outstanding glyph> getPage connections spew out thousands of 'stopping factory' glyph> messages in their output - and seeing those URLs was an invaluable glyph> debugging aid. OK, but by your own description that's output that's useful for debugging (and above you call it spam). My only real question was why that output is on by default instead of off (especially seeing as it's often adding 1K of output to my log file for a single logged message). glyph> 7.) Twisted's log messages are generally ugly and difficult to glyph> understand. glyph> Touche But I didn't say that... I'm very happy to not have to write logging code. Sure, the [HTTPPageGetter,client] speaking from beyond the grave is a bit unsettling, but that's minor. glyph> Except, of course, to come up with some kind of standard for what glyph> makes up a useful / helpful log message; and that's probably not glyph> going to be the unstructured mess of strings that currently goes glyph> into the log. (Can you tell I'm pushing a structured log format glyph> that can be parsed?) Yes, agreed that would be good. glyph> Please feel free to file some tickets for areas of this behavior that glyph> you have noticed. I wish I had time to pour into the logging system, glyph> but neither I (nor any of our sponsors) _really_ needs it to be any glyph> better; it's just a constant annoyance. Understood, thanks. I'm mainly just trying to understand what's going on and why. I will have a bit more of a poke around. As I was going to say, from what I remember about the callback mechanism in defer.py it seems like one should be able to have some kind of data structure that's being maintained (pushed, popped) there that could be updating a log prefix in some way. Thanks! Terry

1 0

[Twisted-Python] Question on default t.p.log output
by Terry Jones Sept. 30, 2008

Sept. 30, 2008

Greetings Twisted superheroes! I have a simple question about logging. Here's a snippet of code: from twisted.python import log from twisted.web import client from twisted.internet import reactor import sys log.startLogging(sys.stdout) def stop(x): reactor.callLater(0, reactor.stop) client.getPage('http://www.divmod.com').addBoth(stop) reactor.run() This produces the output 2008/09/30 14:16 +0200 [-] Log opened. 2008/09/30 14:16 +0200 [-] Starting factory <HTTPClientFactory: http://www.divmod.com> 2008/09/30 14:16 +0200 [HTTPPageGetter,client] Stopping factory <HTTPClientFactory: http://divmod.com/?__session_just_started__=1> 2008/09/30 14:16 +0200 [-] Main loop terminated. I had a look around and can see where (most) of this is coming from. First off, the start/stop factory messages are surprising and feel ugly to me. I mean that's my stdout you're scribbling that stuff all over. The starting and stopping factory methods, despite appearing next to each other in the same base Factory class, don't give the same [...] component in their output. Sometimes I get [Uninitialized], sometimes [-], and sometimes [HTTPPageGetter,client] as above - with both starting and stopping messages. And then, and misleading IMO, is that in a callback attached to the deferred from getPage, the log output (to the uninitiated, a class I am arguably a prime member of) still appears to have something to do with HTTPPageGetter,client when in fact the page has been retrieved, and the callback fired. It's like HTTPPageGetter,client refuses to lie down and die quietly, and insists on sitting up in the casket to holler at its ungrateful descendant callbacks: Hey, I'm the one that kicked this off and don't you forget it! Ahem. I see that output is controlled by a var called noisy in the base Factory class. By why is it True by default if it really is just noise? Judging from the code (and the use of log.msg not log.err) it's not an error. In my actual use case, I see my log filling up with output like this: 2008/09/29 23:55 +0200 [HTTPPageGetter,client] Stopping factory <HTTPClientFactory: https://ec2.amazonaws.com:443/?Action=RunInstances&AWSAccessKeyId=923832926…> I guess the answer might be that I should make my own version of getPage that sets noisy to False after making its HTTPClientFactory, or maybe subclass FileLogObserver and override emit. Or I guess I could be calling functions with log.callWithContext. But those approached seem to be wheeling in a lot of machinery just to set to False a variable that (arguably?) should be False to begin with. See - told you it was simple. I'm at my very best nibbling away on the very edges of the Twist-o-verse. Terry

2 1

[Twisted-Python] jabber_client.py
by John Mangual Sept. 30, 2008

Sept. 30, 2008

I looking at all the code libraries for writing jabber clients and I found twistedwords. I found jabber_client.py from http://twistedmatrix.com/projects/words/documentation/examples/jabber_clien… i ran it on my command line but nothing happens. It's running, but nothing's happening. Please help. - john

3 2

[Twisted-Python] twisted reactor under win32 (test SerialPort)
by Zhe Hu Sept. 29, 2008

Sept. 29, 2008

Hi,I try to use twisted for asynchronous serial port read(). Here is my simple twisted code: ========================== from twisted.internet import reactor from twisted.internet.serialport import SerialPort from twisted.internet.protocol import Protocol class Client(Protocol): def connectionMade(self): print 'connected' def dataReceived(self, data): print data SerialPort(Client(), 'COM4', reactor) reactor.callLater(20, reactor.stop()) reactor.run() ========================== It runs on my Mac with this line changed to SerialPort(Client, 'dev/tty.usbserial-A4001mLL', reactor) which is my Arduino board device. But on Windows XP, it gave the error message: "AttributeError: 'module' object has no attribute 'addEvent' " I tried to used different reactor, such as selectreactor, win32eventreactor, but the error message became "AssertionError: reactor already installed" Please help. Thanks

1 0

[Twisted-Python] txThrift released
by Esteve Fernandez Sept. 29, 2008

Sept. 29, 2008

Hi all, I'm pleased to announce that Fluidinfo [1] is releasing support for Twisted in Apache/Facebook Thrift [2] Download from https://issues.apache.org/jira/browse/THRIFT-148 We hope to get it accepted into the Thrift core, but in the meantime you can apply the patch. There's also a simple example based on the Thrift calculator demo. To add support for Twisted in a Thrift service, simply run "thrift --gen py:twisted specfile.thrift" from the command line. For more details, you can follow the calculator example, as it covers Thrift extensively. Feedback and contributions will be greatly appreciated! :-) Cheers. PS: the JIRA web page lets you vote issues, so just show your support if you think this patch is useful :-) [1] - Fluidinfo - http://www.fluidinfo.com [2] - Thrift - http://incubator.apache.org/thrift/

1 0

[Twisted-Python] Weekly Bug Summary
by exarkun＠twistedmatrix.com Sept. 28, 2008

Sept. 28, 2008

1 0

[Twisted-Python] Ampoule - 0.0.2
by Valentino Volonghi Sept. 27, 2008

Sept. 27, 2008

I just released Ampoule 0.0.2 on pypi and launchpad http://pypi.python.org/pypi/ampoule/0.0.2 https://launchpad.net/ampoule/ampoule/0.0.2 Ampoule is a process pool implementation written on top of Twisted Matrix. Its name comes from the use of AMP as the default communication protocol between the pool and all its children. It's different from other alternative solutions because it provides an API very close to that of the Twisted ThreadPool. As an helper function it also provides a deferToAMPProcess function that creates the ProcessPool and submits jobs to it. The code is pretty well tested, in fact it contains more test lines than actual code lines, and it's pretty well documented. Speedwise on my macbook pro I was able to run about 1700 Ping calls per second without recycling processes and about 1500 with a recycling value of about 10000 calls per process. If anyone wants to contribute a C implementation of amp serializer/deserializer I'm pretty sure that Ampoule would benefit a lot from it but for my current tasks 1500 calls per second is more than fast enough, it's actually pretty fast. A usage example is the following: from ampoule import child, util from twisted.protocols import amp class Pid(amp.Command): response = [("pid", amp.Integer())] class MyChild(child.AMPChild): @Pid.responder def pid(self): import os return {"pid": os.getpid()} @util.mainpoint def main(args): import sys from twisted.internet import reactor, defer from twisted.python import log log.startLogging(sys.stdout) from ampoule import pool @defer.inlineCallbacks def _run(): pp = pool.ProcessPool(MyChild) yield pp.start() result = yield pp.doWork(Pid) print "The Child process PID is:", result['pid'] yield pp.stop() reactor.stop() reactor.callLater(1, _run) reactor.run() What's new in this version: Ampoule 0.0.2 (2008-09-26) ========================== Features -------- - Support process recycling after predefined number of calls. Changes ------- - ProcessPool argument max_idle is now maxIdle to comply with Twisted style guidelines. - removed ampoule.runner module because it was essentially useless, if you want to change subprocess startup behavior pass a string that contains the new code as an argument by overriding the default processFactory for example with a closure like this: from ampoule.main import startAMPProcess from ampoule.pool import ProcessPool pp = ProcessPool() def myProcessFactory(*args, **kwargs): kwargs['bootstrap'] = myBootstrapCode return startAMPProcess(*args, **kwargs) pp.processFactory = staticmethod(myProcessFactory) Have fun with ampoule! :) -- Valentino Volonghi aka Dialtone Now running MacOS X 10.5 Home Page: http://www.twisted.it http://www.adroll.com

1 0

[Twisted-Python] Twisted sprint next Sunday
by Itamar Shtull-Trauring Sept. 26, 2008

Sept. 26, 2008

Unless someone vetoes this, I am going to take glyph up on his offer and preemptively declare a Twisted sprint for next Sunday, September 28, to take place at the Divmod offices in Cambridge, MA. Unfortunately I don't actually know what the address of the Divmod office is, or what the earliest hour we can expect someone to be there, so I will leave these details up to someone else. Those who cannot make it to Boston are of course welcome to work in parallel elsewhere. Those who can't work on that day are of course welcome to work any day they feel like :)

2 1

[Twisted-Python] py2exe with Twisted and plugins, plugins not found
by Gabriel Rossetti Sept. 26, 2008

Sept. 26, 2008

Hello everybody, I am trying to use py2exe with twisted, but I'm running into a problem : one of my classes uses twisted's plugin mechanism and they aren't loaded. I tried the python version and it works correctly. I saw that to use sqlite with storm I have to explicitly import it somewhere like so : import storm.databases.sqlite because py2exe wasn't finding it since it's imported using __import__(), do I have to do something like that with twisted? Thank you, Gabriel

1 0

[Twisted-Python] leave this maillist!
by ross ross Sept. 26, 2008

Sept. 26, 2008

plaease delete my mail address from this maillist, thanks! Ross!

1 0