[Twisted-Python] PEP 342 now in python CVS
Hi all, referring to the news about the implementation of PEP 342 in python CVS: http://dirtsimple.org/2005/08/pep-342-implementation-now-in-cvs.html I know that is a great news for asynchronous programming in python and for other aspects but I wonder what will be the future of Twisted, what will change (if something will change) in Twisted when PEP 342 will come? I think that with PEP 342 we could write a more readable code but does it mean rewrite all twisted? What is the opinion of Twisted developers?? thanks, Tazzo -- La presente comunicazione potrebbe contenere informazioni riservate e/o protette da segreto professionale ed e' indirizzata esclusivamente ai destinatari della medesima qui indicati. Se avete ricevuto per errore la presente comunicazione, siete invitati a segnalarcelo, rispondendo a questo stesso indirizzo di e-mail, e a cancellare il presente messaggio dal Vostro sistema. E' strettamente proibito e potrebbe essere fonte di violazione di legge qualsiasi uso, comunicazione, copia o diffusione dei contenuti di questa comunicazione da parte di chi la abbia ricevuta per errore o in violazione degli scopi della presente. Il messaggio e' stato analizzato alla ricerca di virus o contenuti pericolosi ed e' risultato NON infetto.
Hi, For what it's worth, I started writing a pure Python cooperative threads package, some weeks ago. It's called "softlets" and uses, of course, generators. There is a simple scheduler centered around the notion of "wait objects" - you yield a wait object (e.g. a Queue) and it comes back when the underlying condition is satisfied (e.g. the Queue has something to give to you). I'm writing it for fun but it does have some interesting characteristics: - no C extension required, no compiling, no annoying dependency (even not Twisted actually) - scalability: you can create thousands of threads with nearly no overhead, and run them all "at the same time" - a more comfortable way of writing certain operation sequences than with cascades of deferreds - robustness: in contrast to preemptive multithreading, cooperative MT means you define the synchronisation points explicitly, so you don't have any resource sharing problems in case you forget some synchonisation points (latency will be worse, but your program will stay correct) Although there is no web page currently, you can get the source here (*) and play with the examples in the examples directory (it's very alpha and lacks most fonctionality but it does work, and it won't hose your machine). Nothing is needed except Python 2.4; no setup required, just checkout the sources. (*) http://developer.berlios.de/svn/?group_id=4330 Future work would be to add actual useful functionality (timers, integration with Twisted deferreds, etc.). Feel free to subscribe to the mailing-list and talk a little ;) http://developer.berlios.de/mail/?group_id=4330 I'm open to feedback, and to people willing to contribute. Regards Antoine. Le mercredi 17 août 2005 à 17:02 +0200, tazzo a écrit :
That's Kam*a*elia, unless you really are talking about a porn star. :) (googling "kamelia python" was pretty interesting). On 8/18/05, Tommi Virtanen <tv@twistedmatrix.com> wrote:
-- Twisted | Christopher Armstrong: International Man of Twistery Radix | -- http://radix.twistedmatrix.com | Release Manager, Twisted Project \\\V/// | -- http://twistedmatrix.com |o O| | w----v----w-+
Le jeudi 18 août 2005 à 16:05 +1000, Christopher Armstrong a écrit :
That's Kam*a*elia, unless you really are talking about a porn star. :) (googling "kamelia python" was pretty interesting).
Thanks! I didn't know Kamaelia. According to the simple example given here (http://kamaelia.sourceforge.net/Docs/Axon.html), the programming paradigm is not quite the same: with Kamaelia you explicitly link producers and consumers together, whereas in my approach you just wait on the shared object that represents the communication channel (this object can be used by more than one reader and one writer, by the way). This means a cooperative thread is more self-contained and doesn't need external logic to be linked to other threads. If you want a point of comparison, my approach is similar to that of gtasklets (http://www.gnome.org/~gjc/gtasklet/gtasklets.html), except that I don't use GTK at all - everything is done in plain Python. regards Antoine.
"tazzo" == tazzo <tazzo@email.it> writes:
> I know that is a great news for asynchronous programming in > python I have to wonder if it really is. I think that even if this goes into python, I will continue to write my code in the same way, using separate callback functions. I'd do this mainly because I find it less confusing -- that way code that executes in different reactor iterations will be in different blocks. Getting confused about what the time-relationship between various pieces of code is one of the problems that multithreading increases -- coroutines do that as well, I think, though obviously not to the same degree.
On 8/18/05, tazzo <tazzo@email.it> wrote:
Have you seen Deferred Generators, in twisted.internet.defer (deferredGenerator and waitForDeferred are relevant)? I'm planning on either adding support for PEP342-style generators to them, or writing an alternate implementation specifically for PEP342. I'm happy about PEP342, because it means I'll no longer have to install a buggy C extension module (greenlet) to get decent cooperative threading. This integration is very lightweight and doesn't require any mangling of the guts of Twisted. I'll just note that I am mostly alone amongst the developers of Twisted who think that this kind of cooperative threading + Deferred integration is a good idea. Glyph, Jp, and Allen Short (as he posted) have all said it's a bad idea to have such magical code. On the other hand, it's definitely not as bad as pre-emptive threading, where you basically can't know where context switches will happen. -- Twisted | Christopher Armstrong: International Man of Twistery Radix | -- http://radix.twistedmatrix.com | Release Manager, Twisted Project \\\V/// | -- http://twistedmatrix.com |o O| | w----v----w-+
On Aug 17, 2005, at 1:20 PM, Christopher Armstrong wrote:
I think it's a good idea, so long as you know when to use it. It's particularly useful when doing things that are inherently serial when you don't need the power of arbitrary call graphs hooked up to your asynchronous goodness. -bob
On Thu, 18 Aug 2005 09:20:15 +1000, Christopher Armstrong <radeex@gmail.com> wrote:
Perhaps my opinion has glacially shifted over time, (I think I've always thought this way) but it's a bit more nuanced than that. I think that there is some merit to synchronous-looking code in some situations. I've written some networking code recently that had indentation 10-deep; this PEP might have really helped with that. I think it's a good thing in some situations and will probably reduce the length of some of the nastiest parts of my Q2Q implementation (AKA "Divmod Vertex") by at least half. However, it is limiting - it still doesn't expose the full power of a cooperative threading system. At the same time, it provides some of the same implicit-state facilities that create problems with those same cooperative thread systems. Depending on how it's regarded by Twisted developers, it could give us the bbest of both worlds or the worst of them. How it's regarded is important because that will affect how it's used. I believe that in particularly hairy code, it will probably help, and reduce bugs (code that already has hugely nested functions, that needs to be audited by 3 or 4 people before it works at all). However, in average code it will likely be a source of them, since it tends to reduce line count at the cost of explicit transfer of control. Since people like code that looks blocking a *whole lot*, and are willing to forego the benefits of event-driven programming most of the time (and by "most of the time" I mean the 90% of the world that doesn't even use Twisted, or any other event driven system), I am a bit concerned that this PEP will catch on like wildfire and Twisted programs will get a reputation for being really buggy and never handling edge cases or out-of-order input correctly. So please, use this and enahnce your Twisted programs with it, but please tread lightly. It's a powerful tool but it is still (perhaps even therefore) dangerous.
Christopher Armstrong wrote:
Yes I saw it and I use it very often, this is why I made the question. When I write code for a intensive and massive cooperation task, writing code in a synchronous-looking way help to understand what happen.
:-) -- La presente comunicazione potrebbe contenere informazioni riservate e/o protette da segreto professionale ed e' indirizzata esclusivamente ai destinatari della medesima qui indicati. Se avete ricevuto per errore la presente comunicazione, siete invitati a segnalarcelo, rispondendo a questo stesso indirizzo di e-mail, e a cancellare il presente messaggio dal Vostro sistema. E' strettamente proibito e potrebbe essere fonte di violazione di legge qualsiasi uso, comunicazione, copia o diffusione dei contenuti di questa comunicazione da parte di chi la abbia ricevuta per errore o in violazione degli scopi della presente. Il messaggio e' stato analizzato alla ricerca di virus o contenuti pericolosi ed e' risultato NON infetto.
Hi, For what it's worth, I started writing a pure Python cooperative threads package, some weeks ago. It's called "softlets" and uses, of course, generators. There is a simple scheduler centered around the notion of "wait objects" - you yield a wait object (e.g. a Queue) and it comes back when the underlying condition is satisfied (e.g. the Queue has something to give to you). I'm writing it for fun but it does have some interesting characteristics: - no C extension required, no compiling, no annoying dependency (even not Twisted actually) - scalability: you can create thousands of threads with nearly no overhead, and run them all "at the same time" - a more comfortable way of writing certain operation sequences than with cascades of deferreds - robustness: in contrast to preemptive multithreading, cooperative MT means you define the synchronisation points explicitly, so you don't have any resource sharing problems in case you forget some synchonisation points (latency will be worse, but your program will stay correct) Although there is no web page currently, you can get the source here (*) and play with the examples in the examples directory (it's very alpha and lacks most fonctionality but it does work, and it won't hose your machine). Nothing is needed except Python 2.4; no setup required, just checkout the sources. (*) http://developer.berlios.de/svn/?group_id=4330 Future work would be to add actual useful functionality (timers, integration with Twisted deferreds, etc.). Feel free to subscribe to the mailing-list and talk a little ;) http://developer.berlios.de/mail/?group_id=4330 I'm open to feedback, and to people willing to contribute. Regards Antoine. Le mercredi 17 août 2005 à 17:02 +0200, tazzo a écrit :
That's Kam*a*elia, unless you really are talking about a porn star. :) (googling "kamelia python" was pretty interesting). On 8/18/05, Tommi Virtanen <tv@twistedmatrix.com> wrote:
-- Twisted | Christopher Armstrong: International Man of Twistery Radix | -- http://radix.twistedmatrix.com | Release Manager, Twisted Project \\\V/// | -- http://twistedmatrix.com |o O| | w----v----w-+
Le jeudi 18 août 2005 à 16:05 +1000, Christopher Armstrong a écrit :
That's Kam*a*elia, unless you really are talking about a porn star. :) (googling "kamelia python" was pretty interesting).
Thanks! I didn't know Kamaelia. According to the simple example given here (http://kamaelia.sourceforge.net/Docs/Axon.html), the programming paradigm is not quite the same: with Kamaelia you explicitly link producers and consumers together, whereas in my approach you just wait on the shared object that represents the communication channel (this object can be used by more than one reader and one writer, by the way). This means a cooperative thread is more self-contained and doesn't need external logic to be linked to other threads. If you want a point of comparison, my approach is similar to that of gtasklets (http://www.gnome.org/~gjc/gtasklet/gtasklets.html), except that I don't use GTK at all - everything is done in plain Python. regards Antoine.
"tazzo" == tazzo <tazzo@email.it> writes:
> I know that is a great news for asynchronous programming in > python I have to wonder if it really is. I think that even if this goes into python, I will continue to write my code in the same way, using separate callback functions. I'd do this mainly because I find it less confusing -- that way code that executes in different reactor iterations will be in different blocks. Getting confused about what the time-relationship between various pieces of code is one of the problems that multithreading increases -- coroutines do that as well, I think, though obviously not to the same degree.
On 8/18/05, tazzo <tazzo@email.it> wrote:
Have you seen Deferred Generators, in twisted.internet.defer (deferredGenerator and waitForDeferred are relevant)? I'm planning on either adding support for PEP342-style generators to them, or writing an alternate implementation specifically for PEP342. I'm happy about PEP342, because it means I'll no longer have to install a buggy C extension module (greenlet) to get decent cooperative threading. This integration is very lightweight and doesn't require any mangling of the guts of Twisted. I'll just note that I am mostly alone amongst the developers of Twisted who think that this kind of cooperative threading + Deferred integration is a good idea. Glyph, Jp, and Allen Short (as he posted) have all said it's a bad idea to have such magical code. On the other hand, it's definitely not as bad as pre-emptive threading, where you basically can't know where context switches will happen. -- Twisted | Christopher Armstrong: International Man of Twistery Radix | -- http://radix.twistedmatrix.com | Release Manager, Twisted Project \\\V/// | -- http://twistedmatrix.com |o O| | w----v----w-+
On Aug 17, 2005, at 1:20 PM, Christopher Armstrong wrote:
I think it's a good idea, so long as you know when to use it. It's particularly useful when doing things that are inherently serial when you don't need the power of arbitrary call graphs hooked up to your asynchronous goodness. -bob
On Thu, 18 Aug 2005 09:20:15 +1000, Christopher Armstrong <radeex@gmail.com> wrote:
Perhaps my opinion has glacially shifted over time, (I think I've always thought this way) but it's a bit more nuanced than that. I think that there is some merit to synchronous-looking code in some situations. I've written some networking code recently that had indentation 10-deep; this PEP might have really helped with that. I think it's a good thing in some situations and will probably reduce the length of some of the nastiest parts of my Q2Q implementation (AKA "Divmod Vertex") by at least half. However, it is limiting - it still doesn't expose the full power of a cooperative threading system. At the same time, it provides some of the same implicit-state facilities that create problems with those same cooperative thread systems. Depending on how it's regarded by Twisted developers, it could give us the bbest of both worlds or the worst of them. How it's regarded is important because that will affect how it's used. I believe that in particularly hairy code, it will probably help, and reduce bugs (code that already has hugely nested functions, that needs to be audited by 3 or 4 people before it works at all). However, in average code it will likely be a source of them, since it tends to reduce line count at the cost of explicit transfer of control. Since people like code that looks blocking a *whole lot*, and are willing to forego the benefits of event-driven programming most of the time (and by "most of the time" I mean the 90% of the world that doesn't even use Twisted, or any other event driven system), I am a bit concerned that this PEP will catch on like wildfire and Twisted programs will get a reputation for being really buggy and never handling edge cases or out-of-order input correctly. So please, use this and enahnce your Twisted programs with it, but please tread lightly. It's a powerful tool but it is still (perhaps even therefore) dangerous.
Christopher Armstrong wrote:
Yes I saw it and I use it very often, this is why I made the question. When I write code for a intensive and massive cooperation task, writing code in a synchronous-looking way help to understand what happen.
:-) -- La presente comunicazione potrebbe contenere informazioni riservate e/o protette da segreto professionale ed e' indirizzata esclusivamente ai destinatari della medesima qui indicati. Se avete ricevuto per errore la presente comunicazione, siete invitati a segnalarcelo, rispondendo a questo stesso indirizzo di e-mail, e a cancellare il presente messaggio dal Vostro sistema. E' strettamente proibito e potrebbe essere fonte di violazione di legge qualsiasi uso, comunicazione, copia o diffusione dei contenuti di questa comunicazione da parte di chi la abbia ricevuta per errore o in violazione degli scopi della presente. Il messaggio e' stato analizzato alla ricerca di virus o contenuti pericolosi ed e' risultato NON infetto.
participants (7)
-
Allen Short -
Antoine Pitrou -
Bob Ippolito -
Christopher Armstrong -
glyph＠divmod.com -
tazzo -
Tommi Virtanen