Mailman 3 September 2005 - Twisted

Re: [Twisted-Python] Re: GUI responsiveness
by glyph＠divmod.com Sept. 16, 2005

Sept. 16, 2005

On Thu, 15 Sep 2005 10:51:11 -0700, Jasper <jasper(a)peak.org> wrote: >Nicola Larosa wrote: >>>Worse, I don't see a satisfactory way to deal >>>with the problem if it stems from some internal bit of Twisted taking >>>too big of a time slice. >>Two noticeable cases of that are (de)serialization in PB and in XML-RPC >>code. There is a thing that SEDA does to make these kinds of things more reasonable. Deferreds are sort of a degenerate case of … [View More]

2 1

[Twisted-Python] Re: GUI responsiveness
by Ed Suominen Sept. 16, 2005

Sept. 16, 2005

I've been following this discussion with some interest because I'm starting a new project that will be heavily centered around both MySQL-based persistency and a graphically demanding GUI. I took a look at wxPython but was somewhat turned off by the complication of integrating it with Twisted, which I'm using to do top-level dispatching of time-consuming stuff in threads and in separate processes, and to perform networking when that functionality inevitably winds up getting added to the … [View More]

6 11

[Twisted-Python] application shutdown and control-C
by Antony Kummel Sept. 15, 2005

Sept. 15, 2005

Hi, When running an application on Windows with twistd, and stopping it with Control-C, sometimes it shuts down properly, and sometimes it just stops abruptly, which also means it doesn't get written to *-shutdown.tap. That's no good! Has anyone experienced this before? Thanks, Antony Kummel __________________________________________________ Do You Yahoo!? Tired of spam? Yahoo! Mail has the best spam protection around http://mail.yahoo.com

3 2

Re: [Twisted-Python] Re: GUI responsiveness
by Brian Warner Sept. 15, 2005

Sept. 15, 2005

Nicola Larosa <nico(a)tekNico.net> writes: > > Is there going to be any facility, in newpb, for splitting up > (de)serialization in small chunks? Yes, for serialization of custom classes. The serialization process can be throttled by either the network side (producer/consumer style) or by the serializer side. Each "Slicer" object is responsible for turning a single object into either low-level tokens (numbers and strings) or other Slicable objects. The 'slice' method is … [View More]actually a generator, expected to yield a series of smaller objects. If it yields a Deferred, it will not be prodded again until that Deferred fires. The outbound side of the PB connection will be stalled until that object resumes serialization. (I'm considering an extension that would let you have multiple serialization contexts running in parallel over a single connection: if implemented, this deferred-serialization would not stall the connection.. on the other hand methods could be invoked out-of-order, which might be a problem). You could also register a Slicer adapter to handle existing classes (or conceivably for built-in types, like 'list', although I'm not sure that actually works right now), so you could have a ListSlicer which does something like: def slice(self): ITEMLIMIT = 10 for i in range(0, len(self.original), ITEMLIMIT): for j in range(i, min(len(self.original),i+ITEMLIMIT)): yield self.original[j] d = defer.Deferred() reactor.callLater(0, d.callback, None) yield d to limit how much gets serialized before giving up control for a turn. (of course, this control may be too coarse to achieve what you want, if some of the items are ints and some are big monster nested classes.. ideally you would pay attention to the elapsed time or CPU cycles or pending workload or something and just yield the Deferred when you need to). I don't currently have anything in place for the deserialization side. Unslicer objects have their receiveChild() method called repeatedly with low-level tokens as they arrive off the wire, until you run out. Each time read() returns a buffer of data, the Unslicer does as much work as it can before returning. This means you're naturally limited by the network speed. Unlike Slicers, the Unslicers can only be throttled by the network side. I suspect you would need to have a pretty fast pipe, saturated with inbound data, and a pretty slow (or overloaded) CPU, before you would see a problem with this. If someone thinks it is important, we could probably allow receiveChild() to return a Deferred that means "stop reading from the socket until this Deferred fires", which might be useful. It might also be useful to add some code to the end of dataReceived() (right after it has finished processing everything in the buffer) to do transport.stopReading() (and arrange for it to be started again later) if not enough other work had been done recently. I suspect this is a bit too much to squeeze into PB: if receive-side CPU time is really a problem, we need a more generalized way to handle this. In a previous life, where I implemented a Reactor in C instead of python, I built some priority-queue/round-robin/WFQ stuff into it, to reassure some fellow developers who were concerned about things like traffic on one socket swamping all the others, and who would have really preferred some full-blown real-time guarantees. I'm hesitant to drag this sort of thing into Twisted, but if there's enough of a demand for it (and someone can show actual problems with the existing approach), then maybe it'd be worth investigating. cheers, -Brian [View Less]

1 0

[Twisted-Python] jelly tracebacks
by Antony Kummel Sept. 15, 2005

Sept. 15, 2005

Hi, I just had a grueling experience debugging some unjellying problem. I wonder, why not make jelly tracebacks more useful by adding the details of the PB message that gave rise to them (i.e. which method of which object was called with which arguments). Any thoughts? Antony Kummel __________________________________________________ Do You Yahoo!? Tired of spam? Yahoo! Mail has the best spam protection around http://mail.yahoo.com

1 0

[Twisted-Python] unsafe tracebacks in PB
by Antony Kummel Sept. 15, 2005

Sept. 15, 2005

Hi, Just wondering, why is it possible to turn on unsafe tracebacks only at the server and not at the client? PB being symmetrical, this discrimination makes it arbitrarily difficuly to debug PB applications sometimes. Antony Kummel __________________________________________________ Do You Yahoo!? Tired of spam? Yahoo! Mail has the best spam protection around http://mail.yahoo.com

1 0

[Twisted-Python] threadedselectreactor and twistd
by Antony Kummel Sept. 15, 2005

Sept. 15, 2005

Hi, Just wondering how and if to integrate threadedselectreactor into twistd. I'm thinking to use it for a GUI application which is also a server (and a client), so I think using mktap/twistd makes sense, but there is the matter of threadedselectreactor not being supported. What I've been thinking is for threadedselectreactor to have another method, or it could be combined with interleave, that accepts a callable that makes the main loop run. It could also be a special reactor for, say, … [View More]

13 22

[Twisted-Python] ReconnectingClientFactory creates multiple Protocols?
by mikah＠ceruleansoftware.com Sept. 15, 2005

Sept. 15, 2005

Hello -- My network app consists of a ReconnectingClientFactory whose Protocol instances connect to a another server. The protocol class logs its activities to a file. Each transaction has a unique number that increments. As far as I can tell, it works perfectly, with one strange quirk. If I leave the app running for several days, I start seeing more than one protocol showing up in the logs. I can tell that it's not the same protocol since there's more than one sequence of transaction … [View More]

2 1

[Twisted-Python] process timing / idle handler
by Henning.Ramm＠mediapro-gmbh.de Sept. 14, 2005

Sept. 14, 2005

Ahoi! I often use reactor.callLater(0, something) to let twisted handle and intercept my call queue. Is this really preferable to calling 'something' directly? Or do I just fool myself? Is there something like reactor.OnIdle()? Or, more generally, how could I delay some actions (intermediate cleanup etc.) until the computer has free resources? (Some of my processes run on a very high loaded server.) Best regards, Henning Hraban Ramm Südkurier Medienhaus / MediaPro Support/Admin/Development Dept.

3 2

RE: [Twisted-Python] Re: GUI responsiveness
by Henning.Ramm＠mediapro-gmbh.de Sept. 14, 2005

Sept. 14, 2005

Please everyone stop complaining about how bad twisted and wx would integrate etc. while you didn't try threadedselectreactor first! regards, HR

1 0