Mailman 3 December 2009 - Twisted

[Twisted-Python] HTML shoudl not be baked into twisted.web HTTP implementation
by Jared Gisin Dec. 10, 2009

Dec. 10, 2009

I'm writing a HTTP server that exposes various resources as an API. Unless I'm missing something, twisted's HTTP protocol implementation is in twisted.web.http. The problem with this package is that it's inexplicably wrapped up in HTML. HTML has nothing to do with HTTP as a whole. Sure, HTML is often what HTTP requests return, but there's no reason why it should nor is there any RFC that says it should. HTTP request can return anything. The software I'm writing is a programmatic … [View More]

8 10

Re: [Twisted-Python] A resizable cooperator class for queuing and dispatching jobs
by Terry Jones Dec. 10, 2009

Dec. 10, 2009

>>>>> "Glyph" == Glyph Lefkowitz <glyph(a)twistedmatrix.com> writes: Glyph> On Dec 8, 2009, at 11:09 PM, Terry Jones wrote: >> I just wrote a fun class that lets you >> >> - submit jobs to be dispatched to a queue >> - manage how many tasks are in progress at once >> - dynamically adjust that number >> - shut down cleanly, including >> - recovering jobs that were queued but hadn't been dispatched Glyph> That does sound like … [View More]

2 1

[Twisted-Python] A resizable cooperator class for queuing and dispatching jobs
by Terry Jones Dec. 9, 2009

Dec. 9, 2009

I just wrote a fun class that lets you - submit jobs to be dispatched to a queue - manage how many tasks are in progress at once - dynamically adjust that number - shut down cleanly, including - recovering jobs that were queued but hadn't been dispatched This uses a combination of a DeferredQueue, a task.Cooperator, and the DeferredPool I posted on Monday. For now I named it ResizableDispatchQueue (not a great name, suggestions welcome). You can pick it up from http://pastebin.… [View More]com/f7dc9320e I can think of lots of uses. Here's a simple example. You want to write a server with a web interface that allows people to enter their phone number so you can send them an SMS. You anticipate lots of people will use the service. But sending SMS messages is quite slow, and the company that you ship those jobs off to is concerned that you'll overrun their service (or maybe they have an API limit, etc). So you need to queue up jobs locally and send them off at a certain rate. You'd like to be able to adjust that rate up or down. You also want to be able to shut your service down cleanly (i.e., not in the middle of a task), and when you restart it you want to be able to re-queue the jobs that were queued last time but which hadn't gone out. For example, suppose your function that sends the SMS is called sendSMS and that it takes a (number, message) tuple arg. Then: dispatcher = ResizableDispatchQueue(sendSMS) # Tell it to send at most 5 things at once. dispatcher.start(5) # Same as dispatcher.width = 5 # Later... send off some SMS messages. dispatcher.put((2127399921, 'Hello...')) dispatcher.put((5052929919, 'Test...')) # Later, bump up to 10 simultaneous jobs. dispatcher.width = 10 # Oops, turns out we're sending too fast, turn it down a little. dispatcher.narrow(3) # Get a copy of the list of pending jobs. jobs = dispatcher.pending() # Arrange to increase the number of jobs in an hour's time. reactor.callLater(3600, dispatcher.setWidth, 20) # Time to shutdown. Wait for any tasks underway to complete, and save # the list of jobs not yet dispatched. def saveJobs(jobs): pickle.dump(jobs, ...) d = dispatcher.stop() d.addCallback(saveJobs) On restart you just unpickle the old job list and pass its items to dispatcher.put(). I have a small test suite that's a bit weird (it schedules various things and tests how long the overall job takes and what's still pending when stop is called). It could be much better, but it does at least illustrate that the code seems to work. Let me know if you want it. There's also the issue about what to do when the dispatch function hits an error. An option could be added to re-queue the job, but it's perhaps better to let the dispatch function do that along with whatever else it needs. As usual, I'd be happy to hear comments and suggestions. I'll probably adjust this so the DeferredQueue uses a priority queue. Terry [View Less]

3 2

Re: [Twisted-Python] Deferred on PyPI
by exarkun＠twistedmatrix.com Dec. 9, 2009

Dec. 9, 2009

On 12:12 am, glyph(a)twistedmatrix.com wrote: > >When Deferred is in the standard library, it will enable communication >between code written for different asynchronous backends, which is not >currently possible. Let's be careful how we talk about Deferred, particularly when involving the standard library. Communication is currently possible, but involves using whatever interface each different system uses. Deferred is, as always, about providing a uniform interface. Jean-Paul

1 0

[Twisted-Python] Twistd logging
by Landreville Dec. 9, 2009

Dec. 9, 2009

Is there a way I can customize the format of log file? I want to add more information to each line (about the xmlrpc method being called), but I can't find where these lines get written out. I know I can print a message to the log in my _getFunction, but I would prefer to have one line per request. Thanks, Landreville

3 3

Re: [Twisted-Python] HTML shoudl not be baked into twisted.web HTTP implementation
by Terry Jones Dec. 9, 2009

Dec. 9, 2009

>>>>> "James" == James Y Knight <foom(a)fuhm.net> writes: James> Essentially every web server on the internet responds with HTML to James> error conditions, so you should probably modify your client. But there are plenty of clients that can't or don't want to or shouldn't have to handle HTML. James> As I said, I'm all for customizability, but returning HTML bodies James> for error pages is unambiguously the correct default. I agree. This is, I think, a good … [View More]

1 0

[Twisted-Python] serialport protocol as a service
by Aníbal Pacheco Dec. 8, 2009

Dec. 8, 2009

Hello list, Is there any way to get a service like the twisted.internet.TCPServer but for serialport? I read the serialport code but couldn't figure out how to build such an analogy with the tcp server. The goal I wanna reach is to attach this "serial port service" to my service.MultiService object which already works great with a TCPServer and two TimerService thanks!

2 2

[Twisted-Python] reactor.callLater() or reactor.callFromThread() when sending msg just before quit
by Gabriel Rossetti Dec. 8, 2009

Dec. 8, 2009

Hello everyone, I have to send a message just before my service quits, but sometimes the reactor quits before the msg is sent. I was wondering if I should user reactor.callLater like so : self.sendMessage(msg) reactor.callLater(1, self.stop) (which I don't really like), or use reactor.callFromThread like so : self.sendMessage(msg) reactor.callFromThread(self.stop) (self.stop calls reactor.stop at some point). I really need for that msg to be sent before the reactor shuts down. Thank you, Gabriel

3 2

[Twisted-Python] voiced/opped in twisted IRC
by Mike.lifeguard Dec. 7, 2009

Dec. 7, 2009

-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1 Hello, I cannot see how to get whether a nick is voiced/opped currently in a given channel in the twisted IRC framework. I see methhods of changing modes, but not querying. Can this be easily done? Thanks, - -Mike -----BEGIN PGP SIGNATURE----- Version: GnuPG v1.4.9 (GNU/Linux) iEYEARECAAYFAksdNcAACgkQst0AR/DaKHtptwCdEMPBHa4wC57+nJorxRpddqaP PQQAniZbpQ1u/bmv9QF4NmBJXGkDBWvS =tCQ/ -----END PGP SIGNATURE-----

2 2

Re: [Twisted-Python] TimAllen changed [#3956 - Add arraysize option to runQuery in adbapi]
by Gerrat Rickert Dec. 7, 2009

Dec. 7, 2009

1. About `t.e.adbapi.Transaction.cursor`: It seems that the only >read/write attribute on a cursor object is `arraysize`, and it seems >clunky to mess with `Transaction`'s public interface just for that. How >about leaving the instance variable as `_cursor` and adding >`getArraySize`/`setArraySize` methods? (a property would be even better, >but that requires a new-style class) > >I'm kind of ambivalent about the whole approach of this patch, really: >the only method … [View More]that `runQuery()` ever calls on the cursor is >`fetchall()`, which DBAPI-2.0 describes with "Note that the cursor's >`arraysize` attribute can affect the performance of this operation." >Presumably in sensible DBAPI modules, `fetchall()` will read chunks as >large as possible rather than limiting itself to `arraysize`, but Gerrat >appears to have found a module that needlessly limits itself, so some >configuration is needed. > >However, do we really need to set a separate `arraysize` for every >query? Considering we always call `fetchall()`, presumably we'll always >want to use whatever `arraysize` makes `fetchall()` fastest. I doubt >there's a value that would work for every DBAPI module in every Twisted >installation all over the world, but it seems sensible that every query >in a given `ConnectionPool` would want to use the same arraysize. How >about adding a `cp_arraysize` keyword parameter to `ConnectionPool`, and >applying that setting in `_runInteraction()`? It's a pretty easy way to >configure `arraysize`, it has no backwards compatibility problems, and >it shouldn't be too hard for each Twisted user to find a value that >improves things overall in their environment. > >Of course, when you're pushing for performance there's always >exceptions, and some users might need to set `arraysize` differently for >different queries, and maybe use other calls than `fetchall()`. They can >continue using `runInteraction()` as they presumably already do. Sorry I've dropped the ball on this whole request that I initiated. I'm kind of swamped at work this time of year (I should have more time starting mid Jan.) I have a minute to weigh in though. As for the exact details on how this is implemented, I don't have any strong preferences. I like the suggestion to add the `cp_arraysize` parameter to `ConnectionPool` and apply it in `_runInteraction()`...wish I'd thought of it. I also agree that it's unfortunate that the performance of `fetchall()` is impacted by the `arraysize` attribute; but I think that if it's a deficiency, the issue is with the DBAPI-2.0 specification, not with this implementation. (It would be kind of pointless mentioning that `arraysize` could affect the performance of this method, if this method ignored `arraysize`.) Regards, Gerrat [View Less]

1 0