[Python-ideas] More general "for" loop handling

[Todd]

On Thu, Apr 30, 2015 at 1:36 PM, Steven D'Aprano <steve at pearwood.info>
wrote:

> On Thu, Apr 30, 2015 at 11:48:21AM +0200, Todd wrote:
> > Looking at pep 492, it seems to me the handling of "for" loops has use
> > outside of just asyncio.  The primary use-case I can think of is
> > multiprocessing and multithreading.
> >
> > For example, you could create a multiprocessing pool, and let the pool
> > handle the items in a "for" loop, like so:
> >
> >     from multiprocessing import Pool
> >
> >     mypool = Pool(10, maxtasksperchild=2)
> >
> >     mypool for item in items:
> >         do_something_here
> >         do_something_else
> >         do_yet_another_thing
>
>
> A parallel version of map makes sense, because the semantics of map are
> well defined: given a function f and a sequence [a, b, c, ...] it
> creates a new sequence [f(a), f(b), f(c), ...]. The assumption is that f
> is a pure-function which is side-effect free (if it isn't, you're going
> to have a bad time). The specific order in which a, b, c etc. are
> processed doesn't matter. If it does matter, then map is the wrong way
> to process it.
>
>
multiprocessing.Pool.map guarantees ordering.  It is
multiprocessing.Pool.imap_unordered that doesn't.



> > Of course this sort of thing is possible with iterators and maps today,
> but
> > I think a lot of the same advantages that apply to asyncio also apply to
> > these sorts of cases.  So I think that, rather than having a special
> > keyword just for asyncio, I think it would be better to have a more
> > flexible approach.  Perhaps something like a "__for__" magic method that
> > lets a class implement "for" loop handling, along with the corresponding
> > changes in how the language processes the "for" loop.
>
> "async for" hasn't proven itself yet, and you are already looking to
> generalise it? Shouldn't it prove itself as not a mistake first?
>
>
Two reasons:

1. It may be hard to generalize it later without breaking backwards
compatibility.
2. Whether it can be generalized or not may have some bearing on whether it
gets accepted.
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-ideas/attachments/20150430/38b91324/attachment.html>