Re: [Python-Dev] "make touch" replaced with "make regen-all"

FWIW this will also make cross-compiling a lot easier, since you can't accidentally overwrite the cross-compiled pgen as easily. On Thu, May 4, 2017 at 12:13 PM, Victor Stinner wrote:

Hi,

tl;dr Are you ok to backport my change replacing "make touch" with "make regen-all"? (commit a5c62a8e)

Since the creation of CPython, generated files were regenerated depending on file modification time. For development, that's a convenient feature. But in practice, it caused a long list of pain points. It caused me many issues in my experience:

* On Solaris, Python failed to regenerated the AST because only the system Python was Python 2.6 and the script required Python 2.7 or newer. The "make touch" workaround didn't help, also because of the old version of the system Python.

* On FreeBSD, generated files require "python" but only "python2.7" or "python3.6" programs are available. In The build system was enhanced to try pythonX.Y and then "python".

* "make touch" workaround requires Mercurial, but also a specific version of Mercurial: more than once, "make touch" failed because my Mercurial version was too old.

* Since CPython migrated to Git, "make touch" doesn't work anymore. Sorry, I didn't check why exactly, but I would prefer to not depend on Git *and* Mercurial.

For all these reasons, it was decided to modify the CPython (UNIX/BSD) build system to not regenerate generated files based on file modification time anymore, but require an explicit action: "make regen-all". I already pushed my change to the master branch:

https://github.com/python/cpython/commit/a5c62a8e9f0de6c4133825a5710984a3cd5...

--- commit a5c62a8e9f0de6c4133825a5710984a3cd5e102b Author: Victor Stinner Date: Wed May 3 18:21:48 2017 +0200

bpo-23404: make touch becomes make regen-all (#1405)

Don't rebuild generated files based on file modification time anymore, the action is now explicit. Replace "make touch" with "make regen-all".

Changes:

* Remove "make touch", Tools/hg/hgtouch.py and .hgtouch * Add a new "make regen-all" command to rebuild all generated files * Add subcommands to only generate specific files:

- regen-ast: Include/Python-ast.h and Python/Python-ast.c - regen-grammar: Include/graminit.h and Python/graminit.c - regen-importlib: Python/importlib_external.h and Python/importlib.h - regen-opcode: Include/opcode.h - regen-opcode-targets: Python/opcode_targets.h - regen-typeslots: Objects/typeslots.inc

* Rename PYTHON_FOR_GEN to PYTHON_FOR_REGEN * pgen is now only built by by "make regen-grammar" * Add $(srcdir)/ prefix to paths to source files to handle correctly compilation outside the source directory

Note: $(PYTHON_FOR_REGEN) is no more used nor needed by "make" default target building Python. ---

See the issue for the full rationale:

http://bugs.python.org/issue23404

My commit fixed the two remaining FreeBSD buildbots which were broken because of broken "make touch". All FreeBSD buildbots are repaired on master!

Ok, now the question is: are you ok to backport this change to Python 2.7, 3.5 and 3.6?

I started with a backport to 3.6:

https://github.com/python/cpython/pull/1461

See also "Test somehow that generated files are up to date: run make regen-all" issue: http://bugs.python.org/issue30259

Victor _______________________________________________ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/rymg19%40gmail.com

-- Ryan (ライアン) Yoko Shimomura > ryo (supercell/EGOIST) > Hiroyuki Sawano >> everyone else http://refi64.com/