Mailman 3 November 2006 - Twisted

[Twisted-Python] blockingCallFromThread, ticket 1042
by Markus Schiltknecht 30 Nov '06

30 Nov '06

Hi, A long time ago, I've been asking about a blocking callFromThread function and have been pointed to ticket 1042 at [1]. What's the state on that? Does that have a change of being integrated into twisted? Regards Markus [1]: http://twistedmatrix.com/trac/ticket/1042

1 0

Re: [Twisted-Python] trac's reliability
by glyph＠divmod.com 29 Nov '06

29 Nov '06

On 05:53 am, foom(a)fuhm.net wrote: >Current trac status: >more or less working. On behalf of everyone using Twisted's trac, thanks a lot for putting some work into this. :)

1 0

Re: [Twisted-Python] trac's reliability
by exarkun＠divmod.com 29 Nov '06

29 Nov '06

On 12:31 am, jml(a)mumak.net wrote: >Hello, > >The reliability of our trac instance is ludicrous. It is becoming >extremely difficult to do any work on Twisted, particularly during the >hours when America is asleep. > >I'm very grateful for Jp's tireless efforts in making trac work as >much as it has. I have some idea of how busy he is, and can imagine >how frustrating the task must be. However, we can't continue running >our issue tracker like this. > >First question is, what is causing the outages? People on #twisted >have commented that they haven't seen similar behaviour on their own >tracs. The outages are so frequent that this is becoming an FAQ. There are several problems: Segfaults in the svn bindings - in correspondence with the trac team, I have been told (almost in so many words) that bdb-backed svn repositories are unsupported and we should switch to fsfs. Segfaults in the SQLite bindings - likewise, in correspondence with the trac team, I have more or less been told that SQLite is not a supported database engine and that we should switch to PostgreSQL. Deadlocks in... who knows where. > >The second question is, how can I help trac to work better? Would it >help to throw more hardware at the problem? Should we switch to >another tracker? (blech) Are there open tickets on trac itself that I >could submit patches for? It might be possible to resolve the above mentioned problems. If we convert the repository to fsfs, we might find the segfaults from the svn bindings disappear (of course, we might not - I think we can all recognize the quality of this sort of bug stomping). In addition to the actual task of converting, involved in this would be some level of investigation into the level of stability of the fsfs backend available in the version of debian used on wolfwood, an effort to package or have packaged a newer version, or an upgrade of wolfwood (however, since even edgy lacks svn 1.4, this probably isn't a useful endeavour). This may also involve recompiling several packages on pyramid to add fsfs support or remove bdb support. If we convert the trac database to use PostgreSQL, the SQLite segfaults will hopefully go away. ;) This involves setting up a PostgreSQL server which we can use for trac (one is running on pyramid now, for the benefit of buildbot, beyond that, what state it is in is not clear to me). There is a tool available from edgewall which is supposed to be capable of moving data from a SQLite database to a PostgreSQL database. The various scripts and utilities which we have (eg, the weekly bug summary) may also need to be adjusted (I forget to what extent they are tied to SQLite). Then, as an ongoing task, someone will need to maintain the PostgreSQL server. As for the random deadlocks which occur... I see no realistic course of action which is likely to resolve these. Perhaps, if the above changes are enacted, we will be lucky they will just go away by themselves. That seems to be the attitude of the trac developers, anyway. > >Finally, assuming Twisted's trac isn't going to get much better any >time soon, I would greatly appreciate being given the permissions and >training to restore trac. I think it would also be a good idea to >share the responsibility with someone in a European timezone. Before I left last week I set up a cron job to take care of this. When I returned it was still doing its job, so the level of availability seen over the past week or so may be the highest we can expect until we do something else fundamental to fix the issue. However, the SSH key you gave me long ago is still in place and you should still have access to restart the server (just connecting should do it). Personally, I am in favor of switching away from trac, as I have been since shortly before we adopted it. ;) The only open question is when the replacement will be ready. Jean-Paul

9 11

[Twisted-Python] Why I have not gotten any mail in this list?
by Steve Han 29 Nov '06

29 Nov '06

Anyone help me?

1 0

[Twisted-Python] Perspective Broker
by le dahut 28 Nov '06

28 Nov '06

Hello, Using PB, I've setup a system where clients can control other clients in this way : client1 executes "remote_do_something" on server1 with "ip_client2" as argument. Then server1 connects to client2 and executes "remote_smthng" on it. code is like this : - client1: """ factory = pb.PBClientFactory() reactor.connectTCP(ip_server1, port, factory) d = factory.getRootObject() d.addCallback(lambda object: object.callRemote("do_something", ip_client2)) d.addCallback(client1_call_b) d.addErrback(client1_err_b) """ - server1: """ def remote_do_something(ip): factory = pb.PBClientFactory() reactor.connectTCP(ip_client2, port, factory) d = factory.getRootObject() d.addCallback(lambda object: object.callRemote("smthng")) d.addCallback(server1_call_b) d.addErrback(server1_err_b) """ - client2: """ def remote_smthng(): <code> return "Ok, everything's done" """ How can client1 get client2's answer ? Thanks

2 2

Re: [Twisted-Python] SQLAlchemy integration
by Ed Suominen 28 Nov '06

28 Nov '06

>>> I'm writing some modules for integrating SQLALchemy with Twisted. >>> The idea is to write a custom Pool, EngineStrategy and Engine for Twisted. >>> I would like to know if there are some other people working on this >>> (excluding sAsync) >> Is there really a good reason for having *two* different efforts to >> integrate Twisted with a single other library? Surely it must be >> possible to collaborate somehow. I agree with Glyph. I've put a ton of effort into sAsync, which would cost thousands of dollars at even a fraction of what I charge for "real" work. Why not make use of that? It is GPL'd, but I have one user implementing it in a commercial project now under a special license that he obtained by doing some coding of his own for me. (My thanks to the Twisted developers for making Twisted itself available under a no-strings-attached license, and thus providing me the opportunity for that sort of modest compensation.) > This works because Engine implements the Connectable interface. > There is an issue here: if a query/table is bound to this Engine, every > query will run in a separate thread; this is not what one want when > inside a transaction). Perhaps you should take a closer look at what sAsync does before dismissing it. sAsync has a taskqueue module that it uses to queue up transactions so that they are *all* run in a single, dedicated thread. > More complex is the integration with sqlalchemy session handling. > > As an example with Nevow: > - a thread retrieve an object from the database > - another thread modify the object > > It would be much simple if the two operations occur in the same thread > (this require to run renderHTTP or locateChild in a separate thread but > I ignore if this is safe). Again, the two operations *do* run in the same thread under sAsync, as do all other SQLAlchemy operations that are run via methods of AccessBroker. All you have to do is decorate the method with @transact and it will run via the queue and the dedicated thread. So, you may not want to re-invent the wheel on this one. By the way, I try to be responsive to user comments, ideas, and patches. Some have already improved sAsync quite a bit. Best regards, Ed Suominen Developer of sAsync: http://foss.eepatents.com/sAsync

3 2

Re: [Twisted-Python] Need Exception that will stop ther reactor within twistd
by glyph＠divmod.com 28 Nov '06

28 Nov '06

On 12:55 am, ncesar(a)lunix.com.ar wrote: >> This is bad form, however. Particular types of exception should not stop >> the reactor. Code that wants the reactor to stop, should call >> reactor.stop. Maybe you could make a method that both calls reactor.stop() >> *and* raises an exception. > >This is the way i'm doing it right now, but i'm starting to detect new >exceptions raised within standard library (among others) code... and instead >of catching all posibilities I wanted the application to quit (with the >corresponding logging done). > >I was wondering if twistd/the application class could have a "exit on >Exception" for this type of operations. In general this is a bad idea. There are basically 3 kinds of code you can write with Twisted: 1. Infrastructure code, which implements a protocol and provides APIs for parsing and dealing with events. This kind of code should never contain a reactor.stop at all. 2. Application code which exits in response to a particular user event. This kind of code should contain exactly 1 call to reactor.stop, in the handler for the user event that is the explicit stop. 3. Programs which do one operation, then stop themselves. This kind of code should contain exactly 1 call to reactor.stop, in the final callback/errback of the Deferred representing the operation that it is run to perform. It sounds like you are writing code of type 3. I'm guessing you're writing a program which does one thing, then shuts itself down. If that is the case, the "program" should be encapsulated by a Deferred, and there should be a final addBoth which waits for the program to finish when it exits. If you *don't* structure your program this way, then you'll discover later that sometimes, you want to run your particular operation TWICE in the lifetime of the process instead of once, and you will have to deal with dozens of calls to reactor.stop littered around your application in lots of places. Don't try to always stop the *process* when your program is complete: stop the *program* by keeping a Deferred around which represents its complete run. That way the *driver* for the program is responsible for stopping it, and when you want to aggregate multiple runs of it, you can easily use a tool like DeferredList, or do something different in the callbacks at the end.

2 1

Re: [Twisted-Python] SQLAlchemy integration
by Maciej Szumocki 27 Nov '06

27 Nov '06

> Date: Thu, 23 Nov 2006 07:39:58 -0000 > From: glyph(a)divmod.com > Subject: Re: [Twisted-Python] SQLAlchemy integration > To: Twisted general discussion <twisted-python(a)twistedmatrix.com> > Message-ID:> <20061123073958.22613.1053884361.divmod.xquotient.5(a)joule.divmod.com> > Content-Type: text/plain; charset="utf-8" > > On 22 Nov, 11:19 am, manlio_perillo(a)libero.it wrote: > > >I'm writing some modules for integrating SQLALchemy with Twisted. > >The idea is to write a custom Pool, EngineStrategy and Engine for Twisted. > > >I would like to know if there are some other people working on this > >(excluding sAsync) > > Is there really a good reason for having *two* different efforts to > integrate Twisted with a single other library? Surely it must be possible > to collaborate somehow. I also use both Twisted and SQLAlchemy in one project. Unfortunately, using sAsync is not an option for me due to licensing issue (sAsync is released under GPL). Maciej Szumocki

1 0

Re: [Twisted-Python] Using Conch as a ssh server on MS Windows
by glyph＠divmod.com 27 Nov '06

27 Nov '06

On 09:27 am, rasjidw(a)gmail.com wrote: >I'll probably stick with stdin/stdout, at least initially, as I have >done something along these lines before, just not within the twisted >framework. Previously I used threading and emulated a non-blocking >process through buffering the data read. I've been looking into >non-blocking pipe reading in win32, and it looks like there are a >coule of choices. Possibly the easiest I've seen so far uses some >calls from wxWidgets which adds another dependancy but seems to give >more control. If you're going to use stdin/stdout, just use the Twisted reactor's "spawnProcess" method, as the UNIX version of Conch does (although without the "pty" feature). A great deal of trouble has already been put into making the Windows version of Twisted behave as if the operating system actually supported asynchronous interprocess communication.

2 1

[Twisted-Python] trac's reliability
by Jonathan Lange 27 Nov '06

27 Nov '06

Hello, The reliability of our trac instance is ludicrous. It is becoming extremely difficult to do any work on Twisted, particularly during the hours when America is asleep. I'm very grateful for Jp's tireless efforts in making trac work as much as it has. I have some idea of how busy he is, and can imagine how frustrating the task must be. However, we can't continue running our issue tracker like this. First question is, what is causing the outages? People on #twisted have commented that they haven't seen similar behaviour on their own tracs. The outages are so frequent that this is becoming an FAQ. The second question is, how can I help trac to work better? Would it help to throw more hardware at the problem? Should we switch to another tracker? (blech) Are there open tickets on trac itself that I could submit patches for? Finally, assuming Twisted's trac isn't going to get much better any time soon, I would greatly appreciate being given the permissions and training to restore trac. I think it would also be a good idea to share the responsibility with someone in a European timezone. Thanks, jml

1 0

Twisted November 2006