I see no issue backporting since I don't think we have any compatibility promises when it comes to Makefile commands. Plus if the perf changes to add PGO support could be backported then I don't see why this shouldn't be allowed. On Thu, 4 May 2017 at 10:15 Victor Stinner wrote:

Hi,

tl;dr Are you ok to backport my change replacing "make touch" with "make regen-all"? (commit a5c62a8e)

Since the creation of CPython, generated files were regenerated depending on file modification time. For development, that's a convenient feature. But in practice, it caused a long list of pain points. It caused me many issues in my experience:

* On Solaris, Python failed to regenerated the AST because only the system Python was Python 2.6 and the script required Python 2.7 or newer. The "make touch" workaround didn't help, also because of the old version of the system Python.

* On FreeBSD, generated files require "python" but only "python2.7" or "python3.6" programs are available. In The build system was enhanced to try pythonX.Y and then "python".

* "make touch" workaround requires Mercurial, but also a specific version of Mercurial: more than once, "make touch" failed because my Mercurial version was too old.

* Since CPython migrated to Git, "make touch" doesn't work anymore. Sorry, I didn't check why exactly, but I would prefer to not depend on Git *and* Mercurial.

For all these reasons, it was decided to modify the CPython (UNIX/BSD) build system to not regenerate generated files based on file modification time anymore, but require an explicit action: "make regen-all". I already pushed my change to the master branch:

https://github.com/python/cpython/commit/a5c62a8e9f0de6c4133825a5710984a3cd5...

--- commit a5c62a8e9f0de6c4133825a5710984a3cd5e102b Author: Victor Stinner Date: Wed May 3 18:21:48 2017 +0200

bpo-23404: make touch becomes make regen-all (#1405)

Don't rebuild generated files based on file modification time anymore, the action is now explicit. Replace "make touch" with "make regen-all".

Changes:

* Remove "make touch", Tools/hg/hgtouch.py and .hgtouch * Add a new "make regen-all" command to rebuild all generated files * Add subcommands to only generate specific files:

- regen-ast: Include/Python-ast.h and Python/Python-ast.c - regen-grammar: Include/graminit.h and Python/graminit.c - regen-importlib: Python/importlib_external.h and Python/importlib.h - regen-opcode: Include/opcode.h - regen-opcode-targets: Python/opcode_targets.h - regen-typeslots: Objects/typeslots.inc

* Rename PYTHON_FOR_GEN to PYTHON_FOR_REGEN * pgen is now only built by by "make regen-grammar" * Add $(srcdir)/ prefix to paths to source files to handle correctly compilation outside the source directory

Note: $(PYTHON_FOR_REGEN) is no more used nor needed by "make" default target building Python. ---

See the issue for the full rationale:

http://bugs.python.org/issue23404

My commit fixed the two remaining FreeBSD buildbots which were broken because of broken "make touch". All FreeBSD buildbots are repaired on master!

Ok, now the question is: are you ok to backport this change to Python 2.7, 3.5 and 3.6?

I started with a backport to 3.6:

https://github.com/python/cpython/pull/1461

See also "Test somehow that generated files are up to date: run make regen-all" issue: http://bugs.python.org/issue30259

Victor _______________________________________________ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/brett%40python.org

FWIW this will also make cross-compiling a lot easier, since you can't accidentally overwrite the cross-compiled pgen as easily. On Thu, May 4, 2017 at 12:13 PM, Victor Stinner wrote:

Hi,

tl;dr Are you ok to backport my change replacing "make touch" with "make regen-all"? (commit a5c62a8e)

Since the creation of CPython, generated files were regenerated depending on file modification time. For development, that's a convenient feature. But in practice, it caused a long list of pain points. It caused me many issues in my experience:

* On Solaris, Python failed to regenerated the AST because only the system Python was Python 2.6 and the script required Python 2.7 or newer. The "make touch" workaround didn't help, also because of the old version of the system Python.

* On FreeBSD, generated files require "python" but only "python2.7" or "python3.6" programs are available. In The build system was enhanced to try pythonX.Y and then "python".

* "make touch" workaround requires Mercurial, but also a specific version of Mercurial: more than once, "make touch" failed because my Mercurial version was too old.

* Since CPython migrated to Git, "make touch" doesn't work anymore. Sorry, I didn't check why exactly, but I would prefer to not depend on Git *and* Mercurial.

For all these reasons, it was decided to modify the CPython (UNIX/BSD) build system to not regenerate generated files based on file modification time anymore, but require an explicit action: "make regen-all". I already pushed my change to the master branch:

https://github.com/python/cpython/commit/a5c62a8e9f0de6c4133825a5710984a3cd5...

--- commit a5c62a8e9f0de6c4133825a5710984a3cd5e102b Author: Victor Stinner Date: Wed May 3 18:21:48 2017 +0200

bpo-23404: make touch becomes make regen-all (#1405)

Don't rebuild generated files based on file modification time anymore, the action is now explicit. Replace "make touch" with "make regen-all".

Changes:

* Remove "make touch", Tools/hg/hgtouch.py and .hgtouch * Add a new "make regen-all" command to rebuild all generated files * Add subcommands to only generate specific files:

- regen-ast: Include/Python-ast.h and Python/Python-ast.c - regen-grammar: Include/graminit.h and Python/graminit.c - regen-importlib: Python/importlib_external.h and Python/importlib.h - regen-opcode: Include/opcode.h - regen-opcode-targets: Python/opcode_targets.h - regen-typeslots: Objects/typeslots.inc

* Rename PYTHON_FOR_GEN to PYTHON_FOR_REGEN * pgen is now only built by by "make regen-grammar" * Add $(srcdir)/ prefix to paths to source files to handle correctly compilation outside the source directory

Note: $(PYTHON_FOR_REGEN) is no more used nor needed by "make" default target building Python. ---

See the issue for the full rationale:

http://bugs.python.org/issue23404

My commit fixed the two remaining FreeBSD buildbots which were broken because of broken "make touch". All FreeBSD buildbots are repaired on master!

Ok, now the question is: are you ok to backport this change to Python 2.7, 3.5 and 3.6?

I started with a backport to 3.6:

https://github.com/python/cpython/pull/1461

See also "Test somehow that generated files are up to date: run make regen-all" issue: http://bugs.python.org/issue30259

Victor _______________________________________________ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/rymg19%40gmail.com

-- Ryan (ライアン) Yoko Shimomura > ryo (supercell/EGOIST) > Hiroyuki Sawano >> everyone else http://refi64.com/

Le 5 mai 2017 6:31 AM, "Nick Coghlan" a écrit : The note just needs to say that folks that care about doing "complete" builds need to adjust their command sequence to be "./configure <options> && make regen-all && make install", rather than the previous pattern of "./configure <options> && make install". Hum, you only need to regenerate files if you modify Grammar, AST or opcodes. Do you mean that Linux distro make such downstream changes? Victor

2017-05-05 6:31 GMT+02:00 Nick Coghlan :

For the benefit of Linux distros attempting to ensure they're doing full "from source" builds, it would be good to note this in a "Notable changes in maintenance releases", akin to the existing ones for 3.4 and 2.7 (perhaps retitling the latter accordingly).

Sorry, I'm not aware of these notes. Where can I found them? Where should we document the build system change? You may comment the issue directly: http://bugs.python.org/issue23404 Victor

On Tue, May 9, 2017 at 5:20 AM, Victor Stinner wrote:

This change broke the Coverage job on Travis CI, because sysconfig.py uses get_config_var('AST_H_DIR') to build sysconfig.get_python_inc(): http://bugs.python.org/issue30273

sysconfig was modified in 2012. "Include" was replaced with get_config_var('AST_H_DIR') by: https://hg.python.org/cpython/rev/998c8a8f2aea => see http://bugs.python.org/issue15366

To fix the Coverage job, I reverted this change: https://github.com/python/cpython/commit/b109a1d3360fc4bb87b9887264e3634632d...

I'm unable to reproduce http://bugs.python.org/issue15366 bug: "venv assumes header files in sys._home + '/Include' ".

To reproduce, Python need to be compiled {builddir} != {srcdir} as noted in the README. I've submitted a PR that fixes this (previous?) regression and still permits the Makefile changes.

It seems like venv and virtualend have been updated in the meanwhile to add ${venv}/include to the include directories when building an extension.

Yes, http://bugs.python.org/issue16116 -- Jeremy Kloth