[Python-ideas] Making it easy to prepare for PEP479

[Chris Angelico]

On Tue, May 19, 2015 at 1:15 PM, Steven D'Aprano <steve at pearwood.info> wrote:
> On Tue, May 19, 2015 at 03:36:12AM +1000, Chris Angelico wrote:
>
>> My understanding of the OP's problem is this:
>>
>> # Utility file
>> def some_generator():
>>     yield stuff
>>
>> # Tests file
>> import utils
>> assert next(some_generator()) == stuff
>
> That doesn't test that some_generator never raises StopIteration
> directly. All it does is test that it yields correctly. See below for a
> meaningful test.

Right; this is an existing codebase which (presumably) already has
tests. These tests will continue to pass post-479, but they are
inadequate as proof that the transformation to "never raise
StopIteration" has been completed.

> It's not clear to me whether you're talking about Ram testing that PEP
> 479 is working as claimed ("to prove that the change has indeed
> happened"), or testing *his own generators* to check that he doesn't
> accidentally call "raise StopIteration" inside them, regardless of
> version.
>
> More likely, Ram is testing his own generators, not the interpreter. He
> wants to ensure that none of his generators raise StopIteration but
> always use return instead. Whatever test he writes, he has to run it on
> a generator which is passed in, not on a test generator he writes
> specifically for the test.

Correct.

> It's hard to test arbitrary generators for compliance with the rule
> "don't raise StopIterator directly", since you cannot distinguish a
> return from a raise from the outside unless PEP 479 is in effect.
> Ideally, the test should still fail even if PEP 479 is not implemented.
> Otherwise it's a useless test under 3.4 and older, and you might as well
> not even bother running it.

Indeed. You have summed up the problem.

> Before PEP 479 is in effect, I can't think of any practical way to
> distinguish the cases:
>
> (1) generator exits by raising (fail);
> (2) generator exits by returning (pass);
>
> since both cases end up raising StopIteration. Perhaps Ram is cleverer
> than me and can come up with a definite test, but I'd like to see it
> before commenting.
>
> The only solution I can come up with is to use the inspect module to
> fetch the generator's source code and scan it for "raise StopIteration".
> Parsing the AST will also work, or even the byte-code at a pinch, but
> the source code is easiest:
>
> assertFalse("raise StopIteration" in source)
>
> That will fail if the generator raises StopIteration directly regardless
> of version. It doesn't catch *all possible* violations, e.g.:

More significant example: It doesn't catch a codebase that has some
functions which are used in generators and others which are used in
class-based iterators.

> So, I believe that the whole __future__ directive is a red herring, and
> doesn't actually help Ram do what he wants, which is to write tests
> which will fail if his generators call raise StopIteration regardless of
> what version of Python he runs the test under.

Okay. So how do you ensure that Python 3.7 and Python 3.4 can both run
your code?

ChrisA