[Python-ideas] Making it easy to prepare for PEP479

[Chris Angelico]

On Tue, May 19, 2015 at 3:13 AM, Steven D'Aprano <steve at pearwood.info> wrote:
> If I interpret your words as you wrote them, the solution seems to risk
> becoming as convoluted and messy as the code I had to work with in real
> life. If I try to interpret your words more sensibly ("surely Terry
> cannot possibly mean what he said...?") the suggestion is *still*
> convoluted. If Ram is permitted multiple test files, then the simplest
> solution is to split off the code that relies on the future directive
> into its own file:
>
> try:
>     import xyz  # requires the future directive
> except ImportError:
>     xyz = None
>
> if xyz:
>     # tests with directive
> else:
>     # tests without
>
>
> Instead of going through your process of editing the source code,
> compiling, importing, re-editing, just have two source files.

My understanding of the OP's problem is this:

# Utility file
def some_generator():
    yield stuff

# Tests file
import utils
assert next(some_generator()) == stuff


Now, PEP 479 says that his code should never raise StopIteration in a
generator, or in anything called by a generator. He has no problem
with this, philosophically, but to prove that the change has indeed
happened, it would be good to run the test suite with generator_stop
active - equivalent to running Python 3.7 on the test suite. However,
simply adding a future directive to the tests file will have no effect
(obviously), and adding a future directive to the utility module
itself will break it on Python <3.5, even though it would work just
fine. So the options are:

1) Omit the directive, and trust that it's all working - no benefit
from PEP 479 until Python 3.7.
2) Include the directive, and require Python 3.5+ for no reason other
than this check.
3) Hack something so that the tests are run with the directive active,
but normal running doesn't use it.
4) Hack something so Python 3.5 and 3.6 use the directive, and others don't.

The first two are easy, but have nasty consequences. The third is what
I provided a hack to accomplish (exec the code with a line prepended),
and which Terry suggested the "adorn, import, unadorn" scheme, which
probably also counts as a hack. The fourth is the notion of try/except
around future directives, which I think won't fly.

Terry's proposal doesn't actually require that the .pyc bytecode file
differ from the source code; it will simply mean that the
in-memory-being-executed bytecode will differ from the source. In the
case of future directives like unicode_literals, yes, that would be a
nightmare to debug; but for generator_stop, I doubt it'll cause
problems.

The trouble here is that it's not so much "some code needs the future
directive, some doesn't" as "some use-cases want strict checking, but
we still want compatibility". Ideally, it should be possible to prove
that your test suite now passes in a post-PEP-479 world, without
breaking anything on 3.4 or 2.7. The only question is, how much
hackery are we prepared to accept in order to do this?

Maybe the simplest hackery of all is just to build a tweaked Python
that just always uses generator_stop. It's not hard to do - either
hard-code the bitflag into the default value for ff_features
(future.c:135), or remove part of the condition that actually does the
work (genobject.c:137) - and then you have a 3.7-like Python that
assumes generator_stop semantics. Run your tests with that, and don't
use the future directive at all.

ChrisA