[New-bugs-announce] [issue30440] document peephole optimizer effects

[Andrew Dalke]

New submission from Andrew Dalke:

The peephole optimizer is an overall benefit to Python but it has some side-effects that occasionally cause problems. These are well-known in the issue tracker, but there is no other documentation which will help a Python programmer figure out which constructs may be a problem.

1) it will compute large integer constants and save them in the .pyc file. The following results in a 19M .pyc file. 

def compute_largest_known_prime():
  return 2**74207281 - 1

As an example of someone affected by the compile-time evaluation, see https://stackoverflow.com/questions/34113609/why-does-python-preemptively-hang-when-trying-to-calculate-a-very-large-number/ . Note the many people who struggled to find definitive documentation.

2) it will create and discard large strings. Consider this variant of the code in test_zlib.py, where I have replaced the imported module variable "_1G" with its value:

    @bigmemtest(size=_4G + 4, memuse=1, dry_run=False)
    def test_big_buffer(self, size):
        data = b"nyan" * (2**30 + 1)  # the original uses "_1G"
        self.assertEqual(zlib.crc32(data), 1044521549)
        self.assertEqual(zlib.adler32(data), 2256789997)

The byte compiler will create the ~4GB string then discard it, even though the function will not be called on machines with insufficient RAM.

As an example of how I was affected by this, see #21074 .

3) The optimizer affects control flow such that the coverage.py gives false positives about unreached code.

As examples of how people are affected, see #2506 and https://bitbucket.org/ned/coveragepy/issues/198/continue-marked-as-not-covered . The last item on the coverage.py tracker asks "Is this limitation documented anywhere?"

I do not believe that the current peephole optimizer should be changed to support these use cases, only that there should be documentation about how the optimizer may negatively affect real-world code.

The domain expert on this topic is Raymond Hettinger. He does not feel safe in issues where I am involved. As I believe my continued presence on this issue will inhibit the documentation changes which I think are needed, I will remove my name from this issue and not be further involved.

----------
assignee: docs at python
components: Documentation
messages: 294248
nosy: dalke, docs at python
priority: normal
severity: normal
status: open
title: document peephole optimizer effects
versions: Python 3.7

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue30440>
_______________________________________