asdl.py and Python-ast.[hc]
Hi All, Earlier this morning I had a slight tackle with a couple of the 3.4 bots (sorry everyone!). I fixed some problems in asdl.py - http://hg.python.org/cpython/rev/21d46e3ae60c - and used the 'with' statement. Some bots don't have Python 2.6+ and couldn't bootstrap Python-ast.h/c Two questions: * Should I always check-in Python-ast.h and Python-ast.c when I touch asdl* ? The generated files are unchanged, it's only the timestamp that changed. * Can we, in theory, use new Pythons for asdl* code, because Python-ast.* are, in fact, checked in so they don't have to be rebuilt by the bots or users? Thanks in advance, Eli
On Thu, Sep 26, 2013 at 9:52 AM, Eli Bendersky <eliben@gmail.com> wrote:
Hi All,
Earlier this morning I had a slight tackle with a couple of the 3.4 bots (sorry everyone!). I fixed some problems in asdl.py - http://hg.python.org/cpython/rev/21d46e3ae60c - and used the 'with' statement. Some bots don't have Python 2.6+ and couldn't bootstrap Python-ast.h/c
Two questions:
* Should I always check-in Python-ast.h and Python-ast.c when I touch asdl* ? The generated files are unchanged, it's only the timestamp that changed.
If Python-ast.* are checked in then yes.
* Can we, in theory, use new Pythons for asdl* code, because Python-ast.* are, in fact, checked in so they don't have to be rebuilt by the bots or users?
I don't see why not. the touch extension for hg is there specifically for these files to prevent having to regenerate them.
On Thu, Sep 26, 2013 at 7:18 AM, Brett Cannon <brett@python.org> wrote:
On Thu, Sep 26, 2013 at 9:52 AM, Eli Bendersky <eliben@gmail.com> wrote:
Hi All,
Earlier this morning I had a slight tackle with a couple of the 3.4 bots (sorry everyone!). I fixed some problems in asdl.py - http://hg.python.org/cpython/rev/21d46e3ae60c - and used the 'with' statement. Some bots don't have Python 2.6+ and couldn't bootstrap Python-ast.h/c
Two questions:
* Should I always check-in Python-ast.h and Python-ast.c when I touch asdl* ? The generated files are unchanged, it's only the timestamp that changed.
If Python-ast.* are checked in then yes.
Hmm, how do I do that? hg doesn't accept them for commit since their contents are not changed.
* Can we, in theory, use new Pythons for asdl* code, because Python-ast.* are, in fact, checked in so they don't have to be rebuilt by the bots or users?
I don't see why not. the touch extension for hg is there specifically for these files to prevent having to regenerate them.
2013/9/26 Eli Bendersky <eliben@gmail.com>:
Hi All,
Earlier this morning I had a slight tackle with a couple of the 3.4 bots (sorry everyone!). I fixed some problems in asdl.py - http://hg.python.org/cpython/rev/21d46e3ae60c - and used the 'with' statement. Some bots don't have Python 2.6+ and couldn't bootstrap Python-ast.h/c
Two questions:
* Should I always check-in Python-ast.h and Python-ast.c when I touch asdl* ? The generated files are unchanged, it's only the timestamp that changed. * Can we, in theory, use new Pythons for asdl* code, because Python-ast.* are, in fact, checked in so they don't have to be rebuilt by the bots or users?
We should have the buildbots run "make touch", so they don't need to run asdl_c.py. -- Regards, Benjamin
On Thu, Sep 26, 2013 at 7:28 AM, Benjamin Peterson <benjamin@python.org>wrote:
2013/9/26 Eli Bendersky <eliben@gmail.com>:
Hi All,
Earlier this morning I had a slight tackle with a couple of the 3.4 bots (sorry everyone!). I fixed some problems in asdl.py - http://hg.python.org/cpython/rev/21d46e3ae60c - and used the 'with' statement. Some bots don't have Python 2.6+ and couldn't bootstrap Python-ast.h/c
Two questions:
* Should I always check-in Python-ast.h and Python-ast.c when I touch asdl* ? The generated files are unchanged, it's only the timestamp that changed. * Can we, in theory, use new Pythons for asdl* code, because Python-ast.* are, in fact, checked in so they don't have to be rebuilt by the bots or users?
We should have the buildbots run "make touch", so they don't need to run asdl_c.py.
How should we go about doing this? I don't think we have it documented in the devguide how to tweak the buildbot scripts? Eli
2013/9/26 Eli Bendersky <eliben@gmail.com>:
On Thu, Sep 26, 2013 at 7:28 AM, Benjamin Peterson <benjamin@python.org> wrote:
2013/9/26 Eli Bendersky <eliben@gmail.com>:
Hi All,
Earlier this morning I had a slight tackle with a couple of the 3.4 bots (sorry everyone!). I fixed some problems in asdl.py - http://hg.python.org/cpython/rev/21d46e3ae60c - and used the 'with' statement. Some bots don't have Python 2.6+ and couldn't bootstrap Python-ast.h/c
Two questions:
* Should I always check-in Python-ast.h and Python-ast.c when I touch asdl* ? The generated files are unchanged, it's only the timestamp that changed. * Can we, in theory, use new Pythons for asdl* code, because Python-ast.* are, in fact, checked in so they don't have to be rebuilt by the bots or users?
We should have the buildbots run "make touch", so they don't need to run asdl_c.py.
How should we go about doing this?
Complain to Antoine I suppose. :) -- Regards, Benjamin
2013/9/26 Eli Bendersky <eliben@gmail.com>:
Hi All,
Earlier this morning I had a slight tackle with a couple of the 3.4 bots (sorry everyone!). I fixed some problems in asdl.py - http://hg.python.org/cpython/rev/21d46e3ae60c - and used the 'with' statement. Some bots don't have Python 2.6+ and couldn't bootstrap Python-ast.h/c
Two questions:
* Should I always check-in Python-ast.h and Python-ast.c when I touch asdl* ? The generated files are unchanged, it's only the timestamp that changed. * Can we, in theory, use new Pythons for asdl* code, because Python-ast.* are, in fact, checked in so they don't have to be rebuilt by the bots or users?
We should have the buildbots run "make touch", so they don't need to run asdl_c.py.
How should we go about doing this?
Complain to Antoine I suppose. :)
[+ Antoine] Done :) While we're at it, it seems that .hgtouch is wrong: Include/ast.h: Parser/Python.asdl Parser/asdl.py Parser/asdl_c.py Python/Python-ast.c: Include/ast.h The file Include/ast.h is not, in fact, auto-generated. But Include/Python-ast.h *is*, and it does not appear in this file. Typo? [+Ezio] Eli
On Thu, 26 Sep 2013 11:46:07 -0700 Eli Bendersky <eliben@gmail.com> wrote:
2013/9/26 Eli Bendersky <eliben@gmail.com>:
Hi All,
Earlier this morning I had a slight tackle with a couple of the 3.4 bots (sorry everyone!). I fixed some problems in asdl.py - http://hg.python.org/cpython/rev/21d46e3ae60c - and used the 'with' statement. Some bots don't have Python 2.6+ and couldn't bootstrap Python-ast.h/c
Two questions:
* Should I always check-in Python-ast.h and Python-ast.c when I touch asdl* ? The generated files are unchanged, it's only the timestamp that changed. * Can we, in theory, use new Pythons for asdl* code, because Python-ast.* are, in fact, checked in so they don't have to be rebuilt by the bots or users?
We should have the buildbots run "make touch", so they don't need to run asdl_c.py.
How should we go about doing this?
Complain to Antoine I suppose. :)
[+ Antoine] Done :)
"make touch"? Wow, I'm learning new stuff everyday :-) (what point is there in adding things to Makefile if nobody knows about them?) cheers Antoine.
On Thu, Sep 26, 2013 at 1:45 PM, Antoine Pitrou <solipsis@pitrou.net> wrote:
On Thu, 26 Sep 2013 11:46:07 -0700 Eli Bendersky <eliben@gmail.com> wrote:
2013/9/26 Eli Bendersky <eliben@gmail.com>:
Hi All,
Earlier this morning I had a slight tackle with a couple of the 3.4 bots (sorry everyone!). I fixed some problems in asdl.py - http://hg.python.org/cpython/rev/21d46e3ae60c - and used the 'with' statement. Some bots don't have Python 2.6+ and couldn't bootstrap Python-ast.h/c
Two questions:
* Should I always check-in Python-ast.h and Python-ast.c when I touch asdl* ? The generated files are unchanged, it's only the timestamp that changed. * Can we, in theory, use new Pythons for asdl* code, because Python-ast.* are, in fact, checked in so they don't have to be rebuilt by the bots or users?
We should have the buildbots run "make touch", so they don't need to run asdl_c.py.
How should we go about doing this?
Complain to Antoine I suppose. :)
[+ Antoine] Done :)
"make touch"? Wow, I'm learning new stuff everyday :-)
(what point is there in adding things to Makefile if nobody knows about them?)
The devguide would be a good place to mention this. http://bugs.python.org/issue15964 tracked the task for a while, but I'm not sure what the status is. Eli
On Thu, 26 Sep 2013 14:43:48 -0400 Benjamin Peterson <benjamin@python.org> wrote:
2013/9/26 Eli Bendersky <eliben@gmail.com>:
On Thu, Sep 26, 2013 at 7:28 AM, Benjamin Peterson <benjamin@python.org> wrote:
2013/9/26 Eli Bendersky <eliben@gmail.com>:
Hi All,
Earlier this morning I had a slight tackle with a couple of the 3.4 bots (sorry everyone!). I fixed some problems in asdl.py - http://hg.python.org/cpython/rev/21d46e3ae60c - and used the 'with' statement. Some bots don't have Python 2.6+ and couldn't bootstrap Python-ast.h/c
Two questions:
* Should I always check-in Python-ast.h and Python-ast.c when I touch asdl* ? The generated files are unchanged, it's only the timestamp that changed. * Can we, in theory, use new Pythons for asdl* code, because Python-ast.* are, in fact, checked in so they don't have to be rebuilt by the bots or users?
We should have the buildbots run "make touch", so they don't need to run asdl_c.py.
How should we go about doing this?
Complain to Antoine I suppose. :)
Here you are: http://buildbot.python.org/all/builders/AMD64%20Snow%20Leop%203.x/builds/28/... Of course, when it's using "-jN" there may be a race condition :-) Regards Antoine.
On Thu, Sep 26, 2013 at 2:00 PM, Antoine Pitrou <solipsis@pitrou.net> wrote:
On Thu, 26 Sep 2013 14:43:48 -0400 Benjamin Peterson <benjamin@python.org> wrote:
2013/9/26 Eli Bendersky <eliben@gmail.com>:
On Thu, Sep 26, 2013 at 7:28 AM, Benjamin Peterson <
benjamin@python.org>
wrote:
2013/9/26 Eli Bendersky <eliben@gmail.com>:
Hi All,
Earlier this morning I had a slight tackle with a couple of the 3.4
bots
(sorry everyone!). I fixed some problems in asdl.py - http://hg.python.org/cpython/rev/21d46e3ae60c - and used the 'with' statement. Some bots don't have Python 2.6+ and couldn't bootstrap Python-ast.h/c
Two questions:
* Should I always check-in Python-ast.h and Python-ast.c when I touch asdl* ? The generated files are unchanged, it's only the timestamp that changed. * Can we, in theory, use new Pythons for asdl* code, because Python-ast.* are, in fact, checked in so they don't have to be rebuilt by the bots or users?
We should have the buildbots run "make touch", so they don't need to run asdl_c.py.
How should we go about doing this?
Complain to Antoine I suppose. :)
Here you are:
http://buildbot.python.org/all/builders/AMD64%20Snow%20Leop%203.x/builds/28/...
Of course, when it's using "-jN" there may be a race condition :-)
Right, which is probably even worse than before because now it's non-deterministic (first build will fail, subsequent will succeed). Is there a way to split it to: $ make touch $ make -jN all ?
On Thu, 26 Sep 2013 14:19:59 -0700 Eli Bendersky <eliben@gmail.com> wrote:
On Thu, Sep 26, 2013 at 2:00 PM, Antoine Pitrou <solipsis@pitrou.net> wrote:
On Thu, 26 Sep 2013 14:43:48 -0400 Benjamin Peterson <benjamin@python.org> wrote:
2013/9/26 Eli Bendersky <eliben@gmail.com>:
On Thu, Sep 26, 2013 at 7:28 AM, Benjamin Peterson <
benjamin@python.org>
wrote:
2013/9/26 Eli Bendersky <eliben@gmail.com>:
Hi All,
Earlier this morning I had a slight tackle with a couple of the 3.4
bots
(sorry everyone!). I fixed some problems in asdl.py - http://hg.python.org/cpython/rev/21d46e3ae60c - and used the 'with' statement. Some bots don't have Python 2.6+ and couldn't bootstrap Python-ast.h/c
Two questions:
* Should I always check-in Python-ast.h and Python-ast.c when I touch asdl* ? The generated files are unchanged, it's only the timestamp that changed. * Can we, in theory, use new Pythons for asdl* code, because Python-ast.* are, in fact, checked in so they don't have to be rebuilt by the bots or users?
We should have the buildbots run "make touch", so they don't need to run asdl_c.py.
How should we go about doing this?
Complain to Antoine I suppose. :)
Here you are:
http://buildbot.python.org/all/builders/AMD64%20Snow%20Leop%203.x/builds/28/...
Of course, when it's using "-jN" there may be a race condition :-)
Right, which is probably even worse than before because now it's non-deterministic (first build will fail, subsequent will succeed).
Only if "N" in "-jN" is big enough, I suspect.
Is there a way to split it to:
$ make touch $ make -jN all
Probably but... I'm really too lazy to do this myself. Buildbot isn't very pleasant. Perhaps someone else wants to do it? Regards Antoine.
Am 26.09.13 23:00, schrieb Antoine Pitrou:
Here you are: http://buildbot.python.org/all/builders/AMD64%20Snow%20Leop%203.x/builds/28/...
Of course, when it's using "-jN" there may be a race condition :-)
Could you make "make touch" a separate build step? Regards, Martin
Le Fri, 27 Sep 2013 16:34:41 +0200, "Martin v. Löwis" <martin@v.loewis.de> a écrit :
Am 26.09.13 23:00, schrieb Antoine Pitrou:
Here you are: http://buildbot.python.org/all/builders/AMD64%20Snow%20Leop%203.x/builds/28/...
Of course, when it's using "-jN" there may be a race condition :-)
Could you make "make touch" a separate build step?
I'm not sure it fits in buildbot's standard "GNUAutoconf" build factory: http://docs.buildbot.net/current/manual/cfg-buildfactories.html#index-1 Note that "compile" is a "shell command" so perhaps it is actually possible to pass "make touch && make -jN". Otherwise, another build factory will have to be used. I'll give an access to Eli so that he can try to figure it out :-) Regards Antoine.
Le Thu, 26 Sep 2013 10:28:46 -0400, Benjamin Peterson <benjamin@python.org> a écrit :
2013/9/26 Eli Bendersky <eliben@gmail.com>:
Hi All,
Earlier this morning I had a slight tackle with a couple of the 3.4 bots (sorry everyone!). I fixed some problems in asdl.py - http://hg.python.org/cpython/rev/21d46e3ae60c - and used the 'with' statement. Some bots don't have Python 2.6+ and couldn't bootstrap Python-ast.h/c
Two questions:
* Should I always check-in Python-ast.h and Python-ast.c when I touch asdl* ? The generated files are unchanged, it's only the timestamp that changed. * Can we, in theory, use new Pythons for asdl* code, because Python-ast.* are, in fact, checked in so they don't have to be rebuilt by the bots or users?
We should have the buildbots run "make touch", so they don't need to run asdl_c.py.
So, you're gonna like this: hg --config extensions.touch=Tools/hg/hgtouch.py touch -v Tools/hg/hgtouch.py:21: Warning: 'with' will become a reserved keyword in Python 2.6 *** failed to import extension touch from Tools/hg/hgtouch.py: invalid syntax (hgtouch.py, line 21) hg: unknown command 'touch' http://buildbot.python.org/all/buildslaves/bolen-ubuntu (that buildbot runs Ubuntu 8.04 LTS, for the record) Regards Antoine.
On Fri, Sep 27, 2013 at 5:09 AM, Antoine Pitrou <solipsis@pitrou.net> wrote:
Le Thu, 26 Sep 2013 10:28:46 -0400, Benjamin Peterson <benjamin@python.org> a écrit :
2013/9/26 Eli Bendersky <eliben@gmail.com>:
Hi All,
Earlier this morning I had a slight tackle with a couple of the 3.4 bots (sorry everyone!). I fixed some problems in asdl.py - http://hg.python.org/cpython/rev/21d46e3ae60c - and used the 'with' statement. Some bots don't have Python 2.6+ and couldn't bootstrap Python-ast.h/c
Two questions:
* Should I always check-in Python-ast.h and Python-ast.c when I touch asdl* ? The generated files are unchanged, it's only the timestamp that changed. * Can we, in theory, use new Pythons for asdl* code, because Python-ast.* are, in fact, checked in so they don't have to be rebuilt by the bots or users?
We should have the buildbots run "make touch", so they don't need to run asdl_c.py.
So, you're gonna like this:
hg --config extensions.touch=Tools/hg/hgtouch.py touch -v Tools/hg/hgtouch.py:21: Warning: 'with' will become a reserved keyword in Python 2.6 *** failed to import extension touch from Tools/hg/hgtouch.py: invalid syntax (hgtouch.py, line 21) hg: unknown command 'touch'
http://buildbot.python.org/all/buildslaves/bolen-ubuntu
(that buildbot runs Ubuntu 8.04 LTS, for the record)
We have to define a clear boundary between what needs to be compatible to old Pythons and what doesn't. As far as auto-generated files are concerned, once we set up the bots properly and fix the .hgtouch bugs, there's no reason for anyone to need an old Python for anything except the hgtouch extension itself. But the latter has to be back-ported to work on older Pythons (I suppose from 2.4, or is there reason to go earlier?) FWIW Antoine, I'll be happy to help with the buildbot setup, but you'll have to tell me how to get access to our buildbot scripts :-) Eli
Am 26.09.13 15:52, schrieb Eli Bendersky:
* Should I always check-in Python-ast.h and Python-ast.c when I touch asdl* ? The generated files are unchanged, it's only the timestamp that changed.
If they really didn't change, I don't think it matters much. I believe there is a fundamental problem in Mercurial which fails to get it "right" - with the defense that one shouldn't check in generated files in the first place. "hg touch" is supposed to work around this limitation. So when you only check in the generator, anybody updating should do "make touch" after the update, which should touch the generated files even though they didn't change.
* Can we, in theory, use new Pythons for asdl* code, because Python-ast.* are, in fact, checked in so they don't have to be rebuilt by the bots or users?
I'd say yes. That's the point of checking in generated files, so that users don't need to run the generator.
While we're at it, it seems that .hgtouch is wrong:
Include/ast.h: Parser/Python.asdl Parser/asdl.py Parser/asdl_c.py Python/Python-ast.c: Include/ast.h
The file Include/ast.h is not,
It may well be incorrect - feel free to fix it. Regards, Martin
On Fri, Sep 27, 2013 at 7:26 AM, "Martin v. Löwis" <martin@v.loewis.de>wrote:
Am 26.09.13 15:52, schrieb Eli Bendersky:
* Should I always check-in Python-ast.h and Python-ast.c when I touch asdl* ? The generated files are unchanged, it's only the timestamp that changed.
If they really didn't change, I don't think it matters much. I believe there is a fundamental problem in Mercurial which fails to get it "right" - with the defense that one shouldn't check in generated files in the first place.
"hg touch" is supposed to work around this limitation. So when you only check in the generator, anybody updating should do "make touch" after the update, which should touch the generated files even though they didn't change.
* Can we, in theory, use new Pythons for asdl* code, because Python-ast.* are, in fact, checked in so they don't have to be rebuilt by the bots or users?
I'd say yes. That's the point of checking in generated files, so that users don't need to run the generator.
Thank you for the explanation, Martin.
While we're at it, it seems that .hgtouch is wrong:
Include/ast.h: Parser/Python.asdl Parser/asdl.py Parser/asdl_c.py Python/Python-ast.c: Include/ast.h
The file Include/ast.h is not,
It may well be incorrect - feel free to fix it.
Thanks. There are a couple of things to address before we can try to actually run 'make touch' on all bots. I opened http://bugs.python.org/issue19106 to track this. Eli
participants (5)
-
"Martin v. Löwis" -
Antoine Pitrou -
Benjamin Peterson -
Brett Cannon -
Eli Bendersky