[Python-Dev] Improved thread switching

[Jesse Noller]

FYI: I shot an email to stdlib-sig about the fact I am proposing the
inclusion of the pyProcessing module into the stdlib. Comments and
thoughts regarding that would be welcome. I've got a rough outline of
the PEP, but I need to spend more time with the code examples.

-jesse

On Wed, Mar 19, 2008 at 9:52 PM, Alex Martelli <aleaxit at gmail.com> wrote:
> Hmmm, sorry if I'm missing something obvious, but, if the occasional
>  background computations are sufficiently heavy -- why not fork, do
>  said computations in the child thread, and return the results via any
>  of the various available IPC approaches?  I've recently (at Pycon,
>  mostly) been playing devil's advocate (i.e., being PRO-threads, for
>  once) on the subject of utilizing multiple cores effectively -- but
>  the classic approach (using multiple _processes_ instead) actually
>  works quite well in many cases, and this application server would
>  appear to be one.  (the pyProcessing package appears to offer an easy
>  way to migrate threaded code to multiple-processes approaches,
>  although I've only played around with it, not [yet] used it for
>  production code).
>
>
>  Alex
>
>
>
>  On Wed, Mar 19, 2008 at 10:49 AM, Adam Olsen <rhamph at gmail.com> wrote:
>  > On Wed, Mar 19, 2008 at 11:25 AM, Stefan Ring <s.r at visotech.at> wrote:
>  >  > Adam Olsen <rhamph <at> gmail.com> writes:
>  >  >
>  >  >
>  >  > > So you want responsiveness when idle but throughput when busy?
>  >  >
>  >  >  Exactly ;)
>  >  >
>  >  >
>  >  >  > Are those calculations primarily python code, or does a C library do
>  >  >  > the grunt work?  If it's a C library you shouldn't be affected by
>  >  >  > safethread's increased overhead.
>  >  >  >
>  >  >
>  >  >  It's Python code all the way. Frankly, it's a huge mess, but it would be very
>  >  >  very hard to come up with a scalable solution that would allow to optimize
>  >  >  certain hotspots and redo them in C or C++. There isn't even anything left to
>  >  >  optimize in particular because all those low hanging fruit have already been
>  >  >  taken care of. So it's just ~30kloc Python code over which the total time spent
>  >  >  is quite uniformly distributed :(.
>  >
>  >  I see.  Well, at this point I think the most you can do is file a bug
>  >  so the problem doesn't get forgotten.  If nothing else, if my
>  >  safethread stuff goes in it'll very likely include a --with-gil
>  >  option, so I may put together a FIFO scheduler.
>  >
>  >
>  >  --
>  >  Adam Olsen, aka Rhamphoryncus
>  >
>  >
>  > _______________________________________________
>  >  Python-Dev mailing list
>  >  Python-Dev at python.org
>  >  http://mail.python.org/mailman/listinfo/python-dev
>  >  Unsubscribe: http://mail.python.org/mailman/options/python-dev/aleaxit%40gmail.com
>
>
> >
>  _______________________________________________
>  Python-Dev mailing list
>  Python-Dev at python.org
>  http://mail.python.org/mailman/listinfo/python-dev
>  Unsubscribe: http://mail.python.org/mailman/options/python-dev/jnoller%40gmail.com
>