Mailman 3 December 2009 - Twisted

[Twisted-Python] HTML shoudl not be baked into twisted.web HTTP implementation
by Jared Gisin 10 Dec '09

10 Dec '09

I'm writing a HTTP server that exposes various resources as an API. Unless I'm missing something, twisted's HTTP protocol implementation is in twisted.web.http. The problem with this package is that it's inexplicably wrapped up in HTML. HTML has nothing to do with HTTP as a whole. Sure, HTML is often what HTTP requests return, but there's no reason why it should nor is there any RFC that says it should. HTTP request can return anything. The software I'm writing is a programmatic interface. One never uses a web browser, so things such as displaying tracebacks (twisted.web.util.formatFailure) in HTML format is completely wrong. When implementing HTTP, why assume the client always wants HTML. It seem completely wrong from these modules and libraries to be so full of HTML output. HTML output should be provided as a separate config or option for twisted.web. In this case, why not just dump the traceback directly to the HTTP entity-body? As a consumer of twisted.web, I should not have to battle with the hard-coded HTML output of this library. If I want the library to dump things in HTML output, I should have an option to tell it to do that (and I should be able to better customize the HTML), but I should not get HTML by default. Jared

8 10

Re: [Twisted-Python] A resizable cooperator class for queuing and dispatching jobs
by Terry Jones 10 Dec '09

10 Dec '09

>>>>> "Glyph" == Glyph Lefkowitz <glyph(a)twistedmatrix.com> writes: Glyph> On Dec 8, 2009, at 11:09 PM, Terry Jones wrote: >> I just wrote a fun class that lets you >> >> - submit jobs to be dispatched to a queue >> - manage how many tasks are in progress at once >> - dynamically adjust that number >> - shut down cleanly, including >> - recovering jobs that were queued but hadn't been dispatched Glyph> That does sound like fun! It *is* fun. >> # Later... send off some SMS messages. >> dispatcher.put((2127399921, 'Hello...')) >> dispatcher.put((5052929919, 'Test...')) Glyph> Not that this is really germane to the class in question, but, if Glyph> you're going to be sending SMS messages That was just an example. I even thought about using strings, but kept it looking simpler. >> # Later, bump up to 10 simultaneous jobs. >> dispatcher.width = 10 >> >> # Oops, turns out we're sending too fast, turn it down a little. >> dispatcher.narrow(3) Glyph> This seems somewhat asymmetric. Why are there 'widen' and 'narrow' Glyph> methods if I can just set the 'width' attribute directly? I could Glyph> always just do '.width +=' and '.width -=' if I have a relative Glyph> value. Agreed. I started out just implementing widen and narrow, and they (barely) survived when I later added the width property. I had been thinking it would be good to just be able to change the width up or down a notch, but then when I went to write my (a posteriori) tests, I realized that wasn't really so convenient. So I think you're right they could be simplified away. But the code Glyph, the code? Or have I graduated to the point where I no longer need a laconic garotting? I think the combo of a DeferredQueue and a task.Cooperator is pretty deadly. Terry

2 1

[Twisted-Python] A resizable cooperator class for queuing and dispatching jobs
by Terry Jones 10 Dec '09

10 Dec '09

I just wrote a fun class that lets you - submit jobs to be dispatched to a queue - manage how many tasks are in progress at once - dynamically adjust that number - shut down cleanly, including - recovering jobs that were queued but hadn't been dispatched This uses a combination of a DeferredQueue, a task.Cooperator, and the DeferredPool I posted on Monday. For now I named it ResizableDispatchQueue (not a great name, suggestions welcome). You can pick it up from http://pastebin.com/f7dc9320e I can think of lots of uses. Here's a simple example. You want to write a server with a web interface that allows people to enter their phone number so you can send them an SMS. You anticipate lots of people will use the service. But sending SMS messages is quite slow, and the company that you ship those jobs off to is concerned that you'll overrun their service (or maybe they have an API limit, etc). So you need to queue up jobs locally and send them off at a certain rate. You'd like to be able to adjust that rate up or down. You also want to be able to shut your service down cleanly (i.e., not in the middle of a task), and when you restart it you want to be able to re-queue the jobs that were queued last time but which hadn't gone out. For example, suppose your function that sends the SMS is called sendSMS and that it takes a (number, message) tuple arg. Then: dispatcher = ResizableDispatchQueue(sendSMS) # Tell it to send at most 5 things at once. dispatcher.start(5) # Same as dispatcher.width = 5 # Later... send off some SMS messages. dispatcher.put((2127399921, 'Hello...')) dispatcher.put((5052929919, 'Test...')) # Later, bump up to 10 simultaneous jobs. dispatcher.width = 10 # Oops, turns out we're sending too fast, turn it down a little. dispatcher.narrow(3) # Get a copy of the list of pending jobs. jobs = dispatcher.pending() # Arrange to increase the number of jobs in an hour's time. reactor.callLater(3600, dispatcher.setWidth, 20) # Time to shutdown. Wait for any tasks underway to complete, and save # the list of jobs not yet dispatched. def saveJobs(jobs): pickle.dump(jobs, ...) d = dispatcher.stop() d.addCallback(saveJobs) On restart you just unpickle the old job list and pass its items to dispatcher.put(). I have a small test suite that's a bit weird (it schedules various things and tests how long the overall job takes and what's still pending when stop is called). It could be much better, but it does at least illustrate that the code seems to work. Let me know if you want it. There's also the issue about what to do when the dispatch function hits an error. An option could be added to re-queue the job, but it's perhaps better to let the dispatch function do that along with whatever else it needs. As usual, I'd be happy to hear comments and suggestions. I'll probably adjust this so the DeferredQueue uses a priority queue. Terry

3 2

Re: [Twisted-Python] Deferred on PyPI
by exarkun＠twistedmatrix.com 10 Dec '09

10 Dec '09

On 12:12 am, glyph(a)twistedmatrix.com wrote: > >When Deferred is in the standard library, it will enable communication >between code written for different asynchronous backends, which is not >currently possible. Let's be careful how we talk about Deferred, particularly when involving the standard library. Communication is currently possible, but involves using whatever interface each different system uses. Deferred is, as always, about providing a uniform interface. Jean-Paul

1 0

[Twisted-Python] Twistd logging
by Landreville 09 Dec '09

09 Dec '09

Is there a way I can customize the format of log file? I want to add more information to each line (about the xmlrpc method being called), but I can't find where these lines get written out. I know I can print a message to the log in my _getFunction, but I would prefer to have one line per request. Thanks, Landreville

3 3

Re: [Twisted-Python] HTML shoudl not be baked into twisted.web HTTP implementation
by Terry Jones 09 Dec '09

09 Dec '09

>>>>> "James" == James Y Knight <foom(a)fuhm.net> writes: James> Essentially every web server on the internet responds with HTML to James> error conditions, so you should probably modify your client. But there are plenty of clients that can't or don't want to or shouldn't have to handle HTML. James> As I said, I'm all for customizability, but returning HTML bodies James> for error pages is unambiguously the correct default. I agree. This is, I think, a good example of where respecting the Accept header makes sense. If no Accept header is sent, it's the same as sending */*, in which case t.web could do as it currently does and send text/html. But if there is an Accept header you can pass it and the range of response types you have to a library like http://code.google.com/p/mimeparse/ and it will tell you what to send back. Although making an XML and a JSON response available just in case feels a bit arbitrary, it's no more arbitrary than an HTML response that was designed to look good to a human with a browser. They reply contents just need to be documented. We got this complaint with FluidDB (using t.web), and you also see it floated in public from time to time - I saw someone complaining on Twitter about it just the other day. Another way to mitigate this, at least in the context of errors is to put more information into returned headers. We added an X-FluidDB-Error-Class and X-FluidDB-Request-Id that don't require anyone to dig through a body in some perhaps unpredictable or unhandled format. In simple cases you can then just omit a payload and a Content-Type altogether. Terry

1 0

[Twisted-Python] serialport protocol as a service
by Aníbal Pacheco 09 Dec '09

09 Dec '09

Hello list, Is there any way to get a service like the twisted.internet.TCPServer but for serialport? I read the serialport code but couldn't figure out how to build such an analogy with the tcp server. The goal I wanna reach is to attach this "serial port service" to my service.MultiService object which already works great with a TCPServer and two TimerService thanks!

2 2

[Twisted-Python] reactor.callLater() or reactor.callFromThread() when sending msg just before quit
by Gabriel Rossetti 08 Dec '09

08 Dec '09

Hello everyone, I have to send a message just before my service quits, but sometimes the reactor quits before the msg is sent. I was wondering if I should user reactor.callLater like so : self.sendMessage(msg) reactor.callLater(1, self.stop) (which I don't really like), or use reactor.callFromThread like so : self.sendMessage(msg) reactor.callFromThread(self.stop) (self.stop calls reactor.stop at some point). I really need for that msg to be sent before the reactor shuts down. Thank you, Gabriel

3 2

[Twisted-Python] voiced/opped in twisted IRC
by Mike.lifeguard 07 Dec '09

07 Dec '09

-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1 Hello, I cannot see how to get whether a nick is voiced/opped currently in a given channel in the twisted IRC framework. I see methhods of changing modes, but not querying. Can this be easily done? Thanks, - -Mike -----BEGIN PGP SIGNATURE----- Version: GnuPG v1.4.9 (GNU/Linux) iEYEARECAAYFAksdNcAACgkQst0AR/DaKHtptwCdEMPBHa4wC57+nJorxRpddqaP PQQAniZbpQ1u/bmv9QF4NmBJXGkDBWvS =tCQ/ -----END PGP SIGNATURE-----

2 2

Re: [Twisted-Python] TimAllen changed [#3956 - Add arraysize option to runQuery in adbapi]
by Gerrat Rickert 07 Dec '09

07 Dec '09

1. About `t.e.adbapi.Transaction.cursor`: It seems that the only >read/write attribute on a cursor object is `arraysize`, and it seems >clunky to mess with `Transaction`'s public interface just for that. How >about leaving the instance variable as `_cursor` and adding >`getArraySize`/`setArraySize` methods? (a property would be even better, >but that requires a new-style class) > >I'm kind of ambivalent about the whole approach of this patch, really: >the only method that `runQuery()` ever calls on the cursor is >`fetchall()`, which DBAPI-2.0 describes with "Note that the cursor's >`arraysize` attribute can affect the performance of this operation." >Presumably in sensible DBAPI modules, `fetchall()` will read chunks as >large as possible rather than limiting itself to `arraysize`, but Gerrat >appears to have found a module that needlessly limits itself, so some >configuration is needed. > >However, do we really need to set a separate `arraysize` for every >query? Considering we always call `fetchall()`, presumably we'll always >want to use whatever `arraysize` makes `fetchall()` fastest. I doubt >there's a value that would work for every DBAPI module in every Twisted >installation all over the world, but it seems sensible that every query >in a given `ConnectionPool` would want to use the same arraysize. How >about adding a `cp_arraysize` keyword parameter to `ConnectionPool`, and >applying that setting in `_runInteraction()`? It's a pretty easy way to >configure `arraysize`, it has no backwards compatibility problems, and >it shouldn't be too hard for each Twisted user to find a value that >improves things overall in their environment. > >Of course, when you're pushing for performance there's always >exceptions, and some users might need to set `arraysize` differently for >different queries, and maybe use other calls than `fetchall()`. They can >continue using `runInteraction()` as they presumably already do. Sorry I've dropped the ball on this whole request that I initiated. I'm kind of swamped at work this time of year (I should have more time starting mid Jan.) I have a minute to weigh in though. As for the exact details on how this is implemented, I don't have any strong preferences. I like the suggestion to add the `cp_arraysize` parameter to `ConnectionPool` and apply it in `_runInteraction()`...wish I'd thought of it. I also agree that it's unfortunate that the performance of `fetchall()` is impacted by the `arraysize` attribute; but I think that if it's a deficiency, the issue is with the DBAPI-2.0 specification, not with this implementation. (It would be kind of pointless mentioning that `arraysize` could affect the performance of this method, if this method ignored `arraysize`.) Regards, Gerrat

1 0