Assign(expr* targets, expr value) - why targetS?
http://hg.python.org/cpython/file/1ee45eb6aab9/Parser/Python.asdl In Assign(expr* targets, expr value), why the first argument is a list? -- anatoly t.
2013/11/10 anatoly techtonik <techtonik@gmail.com>:
http://hg.python.org/cpython/file/1ee45eb6aab9/Parser/Python.asdl
In Assign(expr* targets, expr value), why the first argument is a list?
x = y = 42 -- Regards, Benjamin
On Sun, Nov 10, 2013 at 8:34 AM, Benjamin Peterson <benjamin@python.org> wrote:
2013/11/10 anatoly techtonik <techtonik@gmail.com>:
http://hg.python.org/cpython/file/1ee45eb6aab9/Parser/Python.asdl
In Assign(expr* targets, expr value), why the first argument is a list?
x = y = 42
Thanks. Speaking of this ASDL. `expr* targets` means that multiple entities of `expr` under the name 'targets' can be passed to Assign statement. Assign uses them as left value. But `expr` definition contains things that can not be used as left side assignment targets: expr = BoolOp(boolop op, expr* values) | BinOp(expr left, operator op, expr right) ... | Str(string s) -- need to specify raw, unicode, etc? | Bytes(bytes s) | NameConstant(singleton value) | Ellipsis -- the following expression can appear in assignment context | Attribute(expr value, identifier attr, expr_context ctx) | Subscript(expr value, slice slice, expr_context ctx) | Starred(expr value, expr_context ctx) | Name(identifier id, expr_context ctx) | List(expr* elts, expr_context ctx) | Tuple(expr* elts, expr_context ctx) If I understand correctly, this is compiled into C struct definitions (Python-ast.c), and there is a code to traverse the structure, but where is code that validates that the structure is correct? Is it done on the first level - text file parsing, before ASDL is built? If so, then what is the role of this ADSL exactly that the first step is unable to solve? Is it possible to fix ADSL to move `expr` that are allowed in Assign into `expr` subset? What effect will it achieve? I mean - will ADSL compiler complain about wrong stuff on the left side, or it will still be a role of some other component. Which one?
anatoly techtonik wrote:
where is code that validates that the structure is correct? Is it done on the first level - text file parsing, before ASDL is built? If so, then what is the role of this ADSL exactly that the first step is unable to solve?
I think it's done by the code that traverses the AST and generates bytecode. It complains if it comes across something that it doesn't know how to generate assignment code for. The reason it's done this way is that Python's LL parser can't see far enough ahead to know that it's processing an assignment until it has parsed the whole LHS and gets to the '='. Up to that point, it could well be just something to be evaluated, so it has to accept a general expression. I suppose a validation step could be performed before putting it into the AST, but there's no point, because the problem will become obvious during code generation anyway. -- Greg
2013/11/12 anatoly techtonik <techtonik@gmail.com>:
On Sun, Nov 10, 2013 at 8:34 AM, Benjamin Peterson <benjamin@python.org> wrote:
2013/11/10 anatoly techtonik <techtonik@gmail.com>:
http://hg.python.org/cpython/file/1ee45eb6aab9/Parser/Python.asdl
In Assign(expr* targets, expr value), why the first argument is a list?
x = y = 42
Thanks.
Speaking of this ASDL. `expr* targets` means that multiple entities of `expr` under the name 'targets' can be passed to Assign statement. Assign uses them as left value. But `expr` definition contains things that can not be used as left side assignment targets:
expr = BoolOp(boolop op, expr* values) | BinOp(expr left, operator op, expr right) ... | Str(string s) -- need to specify raw, unicode, etc? | Bytes(bytes s) | NameConstant(singleton value) | Ellipsis
-- the following expression can appear in assignment context | Attribute(expr value, identifier attr, expr_context ctx) | Subscript(expr value, slice slice, expr_context ctx) | Starred(expr value, expr_context ctx) | Name(identifier id, expr_context ctx) | List(expr* elts, expr_context ctx) | Tuple(expr* elts, expr_context ctx)
If I understand correctly, this is compiled into C struct definitions (Python-ast.c), and there is a code to traverse the structure, but where is code that validates that the structure is correct? Is it done on the first level - text file parsing, before ASDL is built? If so, then what is the role of this ADSL exactly that the first step is unable to solve?
Only valid expression targets are allowed during AST construction. See set_expr_context in ast.c.
Is it possible to fix ADSL to move `expr` that are allowed in Assign into `expr` subset? What effect will it achieve? I mean - will ADSL compiler complain about wrong stuff on the left side, or it will still be a role of some other component. Which one?
I'm not sure what you mean by an `expr` subset. -- Regards, Benjamin
On Tue, Nov 12, 2013 at 5:08 PM, Benjamin Peterson <benjamin@python.org> wrote:
2013/11/12 anatoly techtonik <techtonik@gmail.com>:
On Sun, Nov 10, 2013 at 8:34 AM, Benjamin Peterson <benjamin@python.org> wrote:
2013/11/10 anatoly techtonik <techtonik@gmail.com>:
http://hg.python.org/cpython/file/1ee45eb6aab9/Parser/Python.asdl
In Assign(expr* targets, expr value), why the first argument is a list?
x = y = 42
Thanks.
Speaking of this ASDL. `expr* targets` means that multiple entities of `expr` under the name 'targets' can be passed to Assign statement. Assign uses them as left value. But `expr` definition contains things that can not be used as left side assignment targets:
expr = BoolOp(boolop op, expr* values) | BinOp(expr left, operator op, expr right) ... | Str(string s) -- need to specify raw, unicode, etc? | Bytes(bytes s) | NameConstant(singleton value) | Ellipsis
-- the following expression can appear in assignment context | Attribute(expr value, identifier attr, expr_context ctx) | Subscript(expr value, slice slice, expr_context ctx) | Starred(expr value, expr_context ctx) | Name(identifier id, expr_context ctx) | List(expr* elts, expr_context ctx) | Tuple(expr* elts, expr_context ctx)
If I understand correctly, this is compiled into C struct definitions (Python-ast.c), and there is a code to traverse the structure, but where is code that validates that the structure is correct? Is it done on the first level - text file parsing, before ASDL is built? If so, then what is the role of this ADSL exactly that the first step is unable to solve?
Only valid expression targets are allowed during AST construction. See set_expr_context in ast.c.
Oh my. Now there is also CST in addition to AST. This stuff - http://docs.python.org/devguide/ - badly needs diagrams about data transformation toolchain from Python source code to machine execution instructions. I'd like some pretty stuff, but raw blogdiag hack will do the job http://blockdiag.com/en/blockdiag/index.html There is no set_expr_context in my copy of CPython code, which seems to be some alpha of Python 3.4
Is it possible to fix ADSL to move `expr` that are allowed in Assign into `expr` subset? What effect will it achieve? I mean - will ADSL compiler complain about wrong stuff on the left side, or it will still be a role of some other component. Which one?
I'm not sure what you mean by an `expr` subset.
Transform this: expr = BoolOp(boolop op, expr* values) | BinOp(expr left, operator op, expr right) ... | Str(string s) -- need to specify raw, unicode, etc? | Bytes(bytes s) | NameConstant(singleton value) | Ellipsis -- the following expression can appear in assignment context | Attribute(expr value, identifier attr, expr_context ctx) | Subscript(expr value, slice slice, expr_context ctx) | Starred(expr value, expr_context ctx) | Name(identifier id, expr_context ctx) | List(expr* elts, expr_context ctx) | Tuple(expr* elts, expr_context ctx) to this: expr = BoolOp(boolop op, expr* values) | BinOp(expr left, operator op, expr right) ... | Str(string s) -- need to specify raw, unicode, etc? | Bytes(bytes s) | NameConstant(singleton value) | Ellipsis -- the following expression can appear in assignment context | expr_asgn expr_asgn = Attribute(expr value, identifier attr, expr_context ctx) | Subscript(expr value, slice slice, expr_context ctx) | Starred(expr value, expr_context ctx) | Name(identifier id, expr_context ctx) | List(expr* elts, expr_context ctx) | Tuple(expr* elts, expr_context ctx)
On Fri, Nov 15, 2013 at 12:54 PM, anatoly techtonik <techtonik@gmail.com> wrote:
On Tue, Nov 12, 2013 at 5:08 PM, Benjamin Peterson <benjamin@python.org> wrote:
2013/11/12 anatoly techtonik <techtonik@gmail.com>:
On Sun, Nov 10, 2013 at 8:34 AM, Benjamin Peterson <benjamin@python.org> wrote:
2013/11/10 anatoly techtonik <techtonik@gmail.com>:
http://hg.python.org/cpython/file/1ee45eb6aab9/Parser/Python.asdl
In Assign(expr* targets, expr value), why the first argument is a list?
x = y = 42
Thanks.
Speaking of this ASDL. `expr* targets` means that multiple entities of `expr` under the name 'targets' can be passed to Assign statement. Assign uses them as left value. But `expr` definition contains things that can not be used as left side assignment targets:
expr = BoolOp(boolop op, expr* values) | BinOp(expr left, operator op, expr right) ... | Str(string s) -- need to specify raw, unicode, etc? | Bytes(bytes s) | NameConstant(singleton value) | Ellipsis
-- the following expression can appear in assignment context | Attribute(expr value, identifier attr, expr_context ctx) | Subscript(expr value, slice slice, expr_context ctx) | Starred(expr value, expr_context ctx) | Name(identifier id, expr_context ctx) | List(expr* elts, expr_context ctx) | Tuple(expr* elts, expr_context ctx)
If I understand correctly, this is compiled into C struct definitions (Python-ast.c), and there is a code to traverse the structure, but where is code that validates that the structure is correct? Is it done on the first level - text file parsing, before ASDL is built? If so, then what is the role of this ADSL exactly that the first step is unable to solve?
Only valid expression targets are allowed during AST construction. See set_expr_context in ast.c.
Oh my. Now there is also CST in addition to AST. This stuff - http://docs.python.org/devguide/ - badly needs diagrams about data transformation toolchain from Python source code to machine execution instructions. I'd like some pretty stuff, but raw blogdiag hack will do the job http://blockdiag.com/en/blockdiag/index.html
There is no set_expr_context in my copy of CPython code, which seems to be some alpha of Python 3.4
Is it possible to fix ADSL to move `expr` that are allowed in Assign into `expr` subset? What effect will it achieve? I mean - will ADSL compiler complain about wrong stuff on the left side, or it will still be a role of some other component. Which one?
I'm not sure what you mean by an `expr` subset.
Transform this:
expr = BoolOp(boolop op, expr* values) | BinOp(expr left, operator op, expr right) ... | Str(string s) -- need to specify raw, unicode, etc? | Bytes(bytes s) | NameConstant(singleton value) | Ellipsis
-- the following expression can appear in assignment context | Attribute(expr value, identifier attr, expr_context ctx) | Subscript(expr value, slice slice, expr_context ctx) | Starred(expr value, expr_context ctx) | Name(identifier id, expr_context ctx) | List(expr* elts, expr_context ctx) | Tuple(expr* elts, expr_context ctx)
to this:
expr = BoolOp(boolop op, expr* values) | BinOp(expr left, operator op, expr right) ... | Str(string s) -- need to specify raw, unicode, etc? | Bytes(bytes s) | NameConstant(singleton value) | Ellipsis
-- the following expression can appear in assignment context | expr_asgn
expr_asgn = Attribute(expr value, identifier attr, expr_context ctx) | Subscript(expr value, slice slice, expr_context ctx) | Starred(expr value, expr_context ctx) | Name(identifier id, expr_context ctx) | List(expr* elts, expr_context ctx) | Tuple(expr* elts, expr_context ctx)
And also this: | Assign(expr* targets, expr value) to this: | Assign(expr_asgn* targets, expr value)
2013/11/15 anatoly techtonik <techtonik@gmail.com>:
On Tue, Nov 12, 2013 at 5:08 PM, Benjamin Peterson <benjamin@python.org> wrote:
2013/11/12 anatoly techtonik <techtonik@gmail.com>:
On Sun, Nov 10, 2013 at 8:34 AM, Benjamin Peterson <benjamin@python.org> wrote:
2013/11/10 anatoly techtonik <techtonik@gmail.com>:
http://hg.python.org/cpython/file/1ee45eb6aab9/Parser/Python.asdl
In Assign(expr* targets, expr value), why the first argument is a list?
x = y = 42
Thanks.
Speaking of this ASDL. `expr* targets` means that multiple entities of `expr` under the name 'targets' can be passed to Assign statement. Assign uses them as left value. But `expr` definition contains things that can not be used as left side assignment targets:
expr = BoolOp(boolop op, expr* values) | BinOp(expr left, operator op, expr right) ... | Str(string s) -- need to specify raw, unicode, etc? | Bytes(bytes s) | NameConstant(singleton value) | Ellipsis
-- the following expression can appear in assignment context | Attribute(expr value, identifier attr, expr_context ctx) | Subscript(expr value, slice slice, expr_context ctx) | Starred(expr value, expr_context ctx) | Name(identifier id, expr_context ctx) | List(expr* elts, expr_context ctx) | Tuple(expr* elts, expr_context ctx)
If I understand correctly, this is compiled into C struct definitions (Python-ast.c), and there is a code to traverse the structure, but where is code that validates that the structure is correct? Is it done on the first level - text file parsing, before ASDL is built? If so, then what is the role of this ADSL exactly that the first step is unable to solve?
Only valid expression targets are allowed during AST construction. See set_expr_context in ast.c.
Oh my. Now there is also CST in addition to AST. This stuff - http://docs.python.org/devguide/ - badly needs diagrams about data transformation toolchain from Python source code to machine execution instructions. I'd like some pretty stuff, but raw blogdiag hack will do the job http://blockdiag.com/en/blockdiag/index.html
There is no set_expr_context in my copy of CPython code, which seems to be some alpha of Python 3.4
It's actually called set_context.
Is it possible to fix ADSL to move `expr` that are allowed in Assign into `expr` subset? What effect will it achieve? I mean - will ADSL compiler complain about wrong stuff on the left side, or it will still be a role of some other component. Which one?
I'm not sure what you mean by an `expr` subset.
Transform this:
expr = BoolOp(boolop op, expr* values) | BinOp(expr left, operator op, expr right) ... | Str(string s) -- need to specify raw, unicode, etc? | Bytes(bytes s) | NameConstant(singleton value) | Ellipsis
-- the following expression can appear in assignment context | Attribute(expr value, identifier attr, expr_context ctx) | Subscript(expr value, slice slice, expr_context ctx) | Starred(expr value, expr_context ctx) | Name(identifier id, expr_context ctx) | List(expr* elts, expr_context ctx) | Tuple(expr* elts, expr_context ctx)
to this:
expr = BoolOp(boolop op, expr* values) | BinOp(expr left, operator op, expr right) ... | Str(string s) -- need to specify raw, unicode, etc? | Bytes(bytes s) | NameConstant(singleton value) | Ellipsis
-- the following expression can appear in assignment context | expr_asgn
expr_asgn = Attribute(expr value, identifier attr, expr_context ctx) | Subscript(expr value, slice slice, expr_context ctx) | Starred(expr value, expr_context ctx) | Name(identifier id, expr_context ctx) | List(expr* elts, expr_context ctx) | Tuple(expr* elts, expr_context ctx)
I doubt ASDL will let you do that. -- Regards, Benjamin
On Fri, Nov 15, 2013 at 5:43 PM, Benjamin Peterson <benjamin@python.org> wrote:
2013/11/15 anatoly techtonik <techtonik@gmail.com>:
On Tue, Nov 12, 2013 at 5:08 PM, Benjamin Peterson <benjamin@python.org> wrote:
2013/11/12 anatoly techtonik <techtonik@gmail.com>:
On Sun, Nov 10, 2013 at 8:34 AM, Benjamin Peterson <benjamin@python.org> wrote:
2013/11/10 anatoly techtonik <techtonik@gmail.com>:
http://hg.python.org/cpython/file/1ee45eb6aab9/Parser/Python.asdl
In Assign(expr* targets, expr value), why the first argument is a list?
x = y = 42
Thanks.
Speaking of this ASDL. `expr* targets` means that multiple entities of `expr` under the name 'targets' can be passed to Assign statement. Assign uses them as left value. But `expr` definition contains things that can not be used as left side assignment targets:
expr = BoolOp(boolop op, expr* values) | BinOp(expr left, operator op, expr right) ... | Str(string s) -- need to specify raw, unicode, etc? | Bytes(bytes s) | NameConstant(singleton value) | Ellipsis
-- the following expression can appear in assignment context | Attribute(expr value, identifier attr, expr_context ctx) | Subscript(expr value, slice slice, expr_context ctx) | Starred(expr value, expr_context ctx) | Name(identifier id, expr_context ctx) | List(expr* elts, expr_context ctx) | Tuple(expr* elts, expr_context ctx)
If I understand correctly, this is compiled into C struct definitions (Python-ast.c), and there is a code to traverse the structure, but where is code that validates that the structure is correct? Is it done on the first level - text file parsing, before ASDL is built? If so, then what is the role of this ADSL exactly that the first step is unable to solve?
Only valid expression targets are allowed during AST construction. See set_expr_context in ast.c.
Oh my. Now there is also CST in addition to AST. This stuff - http://docs.python.org/devguide/ - badly needs diagrams about data transformation toolchain from Python source code to machine execution instructions. I'd like some pretty stuff, but raw blogdiag hack will do the job http://blockdiag.com/en/blockdiag/index.html
There is no set_expr_context in my copy of CPython code, which seems to be some alpha of Python 3.4
It's actually called set_context.
Ok. So what is the process? SOURCE --> TOKEN STREAM --> SENTENCE STREAM --> CST --> --> AST --> BYTECODE Is that right?
Is it possible to fix ADSL to move `expr` that are allowed in Assign into `expr` subset? What effect will it achieve? I mean - will ADSL compiler complain about wrong stuff on the left side, or it will still be a role of some other component. Which one?
I'm not sure what you mean by an `expr` subset.
Transform this:
expr = BoolOp(boolop op, expr* values) | BinOp(expr left, operator op, expr right) ... | Str(string s) -- need to specify raw, unicode, etc? | Bytes(bytes s) | NameConstant(singleton value) | Ellipsis
-- the following expression can appear in assignment context | Attribute(expr value, identifier attr, expr_context ctx) | Subscript(expr value, slice slice, expr_context ctx) | Starred(expr value, expr_context ctx) | Name(identifier id, expr_context ctx) | List(expr* elts, expr_context ctx) | Tuple(expr* elts, expr_context ctx)
to this:
expr = BoolOp(boolop op, expr* values) | BinOp(expr left, operator op, expr right) ... | Str(string s) -- need to specify raw, unicode, etc? | Bytes(bytes s) | NameConstant(singleton value) | Ellipsis
-- the following expression can appear in assignment context | expr_asgn
expr_asgn = Attribute(expr value, identifier attr, expr_context ctx) | Subscript(expr value, slice slice, expr_context ctx) | Starred(expr value, expr_context ctx) | Name(identifier id, expr_context ctx) | List(expr* elts, expr_context ctx) | Tuple(expr* elts, expr_context ctx)
I doubt ASDL will let you do that.
asdl.py is plain broken - wrong number of arguments passed to output function asdl_c.py worked ok with fixed ASDL and generated - diff attached. I don't know what to check further - on Windows without Visual Studio. -- anatoly t.
2013/11/22 anatoly techtonik <techtonik@gmail.com>:
On Fri, Nov 15, 2013 at 5:43 PM, Benjamin Peterson <benjamin@python.org> wrote:
2013/11/15 anatoly techtonik <techtonik@gmail.com>:
On Tue, Nov 12, 2013 at 5:08 PM, Benjamin Peterson <benjamin@python.org> wrote:
2013/11/12 anatoly techtonik <techtonik@gmail.com>:
On Sun, Nov 10, 2013 at 8:34 AM, Benjamin Peterson <benjamin@python.org> wrote:
2013/11/10 anatoly techtonik <techtonik@gmail.com>: > http://hg.python.org/cpython/file/1ee45eb6aab9/Parser/Python.asdl > > In Assign(expr* targets, expr value), why the first argument is a list?
x = y = 42
Thanks.
Speaking of this ASDL. `expr* targets` means that multiple entities of `expr` under the name 'targets' can be passed to Assign statement. Assign uses them as left value. But `expr` definition contains things that can not be used as left side assignment targets:
expr = BoolOp(boolop op, expr* values) | BinOp(expr left, operator op, expr right) ... | Str(string s) -- need to specify raw, unicode, etc? | Bytes(bytes s) | NameConstant(singleton value) | Ellipsis
-- the following expression can appear in assignment context | Attribute(expr value, identifier attr, expr_context ctx) | Subscript(expr value, slice slice, expr_context ctx) | Starred(expr value, expr_context ctx) | Name(identifier id, expr_context ctx) | List(expr* elts, expr_context ctx) | Tuple(expr* elts, expr_context ctx)
If I understand correctly, this is compiled into C struct definitions (Python-ast.c), and there is a code to traverse the structure, but where is code that validates that the structure is correct? Is it done on the first level - text file parsing, before ASDL is built? If so, then what is the role of this ADSL exactly that the first step is unable to solve?
Only valid expression targets are allowed during AST construction. See set_expr_context in ast.c.
Oh my. Now there is also CST in addition to AST. This stuff - http://docs.python.org/devguide/ - badly needs diagrams about data transformation toolchain from Python source code to machine execution instructions. I'd like some pretty stuff, but raw blogdiag hack will do the job http://blockdiag.com/en/blockdiag/index.html
There is no set_expr_context in my copy of CPython code, which seems to be some alpha of Python 3.4
It's actually called set_context.
Ok. So what is the process?
SOURCE --> TOKEN STREAM --> SENTENCE STREAM --> CST --> --> AST --> BYTECODE
Is that right?
I don't know what sentence stream is, but otherwise looks right. -- Regards, Benjamin
On Fri, Nov 22, 2013 at 7:29 AM, Benjamin Peterson <benjamin@python.org>wrote:
On Fri, Nov 15, 2013 at 5:43 PM, Benjamin Peterson <benjamin@python.org> wrote:
2013/11/15 anatoly techtonik <techtonik@gmail.com>:
On Tue, Nov 12, 2013 at 5:08 PM, Benjamin Peterson < benjamin@python.org> wrote:
2013/11/12 anatoly techtonik <techtonik@gmail.com>:
On Sun, Nov 10, 2013 at 8:34 AM, Benjamin Peterson < benjamin@python.org> wrote: > 2013/11/10 anatoly techtonik <techtonik@gmail.com>: >> http://hg.python.org/cpython/file/1ee45eb6aab9/Parser/Python.asdl >> >> In Assign(expr* targets, expr value), why the first argument is a
2013/11/22 anatoly techtonik <techtonik@gmail.com>: list?
> > x = y = 42
Thanks.
Speaking of this ASDL. `expr* targets` means that multiple entities of `expr` under the name 'targets' can be passed to Assign statement. Assign uses them as left value. But `expr` definition contains things that can not be used as left side assignment targets:
expr = BoolOp(boolop op, expr* values) | BinOp(expr left, operator op, expr right) ... | Str(string s) -- need to specify raw, unicode, etc? | Bytes(bytes s) | NameConstant(singleton value) | Ellipsis
-- the following expression can appear in assignment context | Attribute(expr value, identifier attr, expr_context ctx) | Subscript(expr value, slice slice, expr_context ctx) | Starred(expr value, expr_context ctx) | Name(identifier id, expr_context ctx) | List(expr* elts, expr_context ctx) | Tuple(expr* elts, expr_context ctx)
If I understand correctly, this is compiled into C struct definitions (Python-ast.c), and there is a code to traverse the structure, but where is code that validates that the structure is correct? Is it done on the first level - text file parsing, before ASDL is built? If so, then what is the role of this ADSL exactly that the first step is unable to solve?
Only valid expression targets are allowed during AST construction. See set_expr_context in ast.c.
Oh my. Now there is also CST in addition to AST. This stuff - http://docs.python.org/devguide/ - badly needs diagrams about data transformation toolchain from Python source code to machine execution instructions. I'd like some pretty stuff, but raw blogdiag hack will do the job http://blockdiag.com/en/blockdiag/index.html
There is no set_expr_context in my copy of CPython code, which seems to be some alpha of Python 3.4
It's actually called set_context.
Ok. So what is the process?
SOURCE --> TOKEN STREAM --> SENTENCE STREAM --> CST --> --> AST --> BYTECODE
Is that right?
I don't know what sentence stream is, but otherwise looks right.
If you want more of an overview: http://pyvideo.org/video/2331/from-source-to-code-how-cpythons-compiler-work...
On Fri, Nov 22, 2013 at 3:29 PM, Benjamin Peterson <benjamin@python.org> wrote:
2013/11/22 anatoly techtonik <techtonik@gmail.com>:
On Fri, Nov 15, 2013 at 5:43 PM, Benjamin Peterson <benjamin@python.org> wrote:
2013/11/15 anatoly techtonik <techtonik@gmail.com>:
On Tue, Nov 12, 2013 at 5:08 PM, Benjamin Peterson <benjamin@python.org> wrote:
2013/11/12 anatoly techtonik <techtonik@gmail.com>:
On Sun, Nov 10, 2013 at 8:34 AM, Benjamin Peterson <benjamin@python.org> wrote: > 2013/11/10 anatoly techtonik <techtonik@gmail.com>: >> http://hg.python.org/cpython/file/1ee45eb6aab9/Parser/Python.asdl >> >> In Assign(expr* targets, expr value), why the first argument is a list? > > x = y = 42
Thanks.
Speaking of this ASDL. `expr* targets` means that multiple entities of `expr` under the name 'targets' can be passed to Assign statement. Assign uses them as left value. But `expr` definition contains things that can not be used as left side assignment targets:
expr = BoolOp(boolop op, expr* values) | BinOp(expr left, operator op, expr right) ... | Str(string s) -- need to specify raw, unicode, etc? | Bytes(bytes s) | NameConstant(singleton value) | Ellipsis
-- the following expression can appear in assignment context | Attribute(expr value, identifier attr, expr_context ctx) | Subscript(expr value, slice slice, expr_context ctx) | Starred(expr value, expr_context ctx) | Name(identifier id, expr_context ctx) | List(expr* elts, expr_context ctx) | Tuple(expr* elts, expr_context ctx)
If I understand correctly, this is compiled into C struct definitions (Python-ast.c), and there is a code to traverse the structure, but where is code that validates that the structure is correct? Is it done on the first level - text file parsing, before ASDL is built? If so, then what is the role of this ADSL exactly that the first step is unable to solve?
Only valid expression targets are allowed during AST construction. See set_expr_context in ast.c.
Oh my. Now there is also CST in addition to AST. This stuff - http://docs.python.org/devguide/ - badly needs diagrams about data transformation toolchain from Python source code to machine execution instructions. I'd like some pretty stuff, but raw blogdiag hack will do the job http://blockdiag.com/en/blockdiag/index.html
There is no set_expr_context in my copy of CPython code, which seems to be some alpha of Python 3.4
It's actually called set_context.
Ok. So what is the process?
SOURCE --> TOKEN STREAM --> SENTENCE STREAM --> CST --> --> AST --> BYTECODE
Is that right?
I don't know what sentence stream is, but otherwise looks right.
I mean that when you have TOKENS, you need to validate that their order is valid and throw errors to console if it is not. Like every sentence should have certain word order to make sense, you won't be able to construct CST from tokens that are placed in wrong order. Right? Or is CST itself a tree of tokens, which position is validated when the tree is transformed to AST? FWIW, I've modified my astdump project to make it easy to explore Python AST and experiment with it. This dataset shows how to create coverage for AST transformations (visitors) https://bitbucket.org/techtonik/astdump/src/tip/dataset/?at=default
On Fri, Nov 22, 2013 at 3:21 AM, anatoly techtonik <techtonik@gmail.com>wrote:
2013/11/15 anatoly techtonik <techtonik@gmail.com>:
On Tue, Nov 12, 2013 at 5:08 PM, Benjamin Peterson <benjamin@python.org> wrote:
2013/11/12 anatoly techtonik <techtonik@gmail.com>:
On Sun, Nov 10, 2013 at 8:34 AM, Benjamin Peterson < benjamin@python.org> wrote:
2013/11/10 anatoly techtonik <techtonik@gmail.com>: > http://hg.python.org/cpython/file/1ee45eb6aab9/Parser/Python.asdl > > In Assign(expr* targets, expr value), why the first argument is a
On Fri, Nov 15, 2013 at 5:43 PM, Benjamin Peterson <benjamin@python.org> wrote: list?
x = y = 42
Thanks.
Speaking of this ASDL. `expr* targets` means that multiple entities of `expr` under the name 'targets' can be passed to Assign statement. Assign uses them as left value. But `expr` definition contains things that can not be used as left side assignment targets:
expr = BoolOp(boolop op, expr* values) | BinOp(expr left, operator op, expr right) ... | Str(string s) -- need to specify raw, unicode, etc? | Bytes(bytes s) | NameConstant(singleton value) | Ellipsis
-- the following expression can appear in assignment context | Attribute(expr value, identifier attr, expr_context ctx) | Subscript(expr value, slice slice, expr_context ctx) | Starred(expr value, expr_context ctx) | Name(identifier id, expr_context ctx) | List(expr* elts, expr_context ctx) | Tuple(expr* elts, expr_context ctx)
If I understand correctly, this is compiled into C struct definitions (Python-ast.c), and there is a code to traverse the structure, but where is code that validates that the structure is correct? Is it done on the first level - text file parsing, before ASDL is built? If so, then what is the role of this ADSL exactly that the first step is unable to solve?
Only valid expression targets are allowed during AST construction. See set_expr_context in ast.c.
Oh my. Now there is also CST in addition to AST. This stuff - http://docs.python.org/devguide/ - badly needs diagrams about data transformation toolchain from Python source code to machine execution instructions. I'd like some pretty stuff, but raw blogdiag hack will do the job http://blockdiag.com/en/blockdiag/index.html
There is no set_expr_context in my copy of CPython code, which seems to be some alpha of Python 3.4
It's actually called set_context.
Ok. So what is the process?
SOURCE --> TOKEN STREAM --> SENTENCE STREAM --> CST --> --> AST --> BYTECODE
Is that right?
Is it possible to fix ADSL to move `expr` that are allowed in Assign into `expr` subset? What effect will it achieve? I mean - will ADSL compiler complain about wrong stuff on the left side, or it will still be a role of some other component. Which one?
I'm not sure what you mean by an `expr` subset.
Transform this:
expr = BoolOp(boolop op, expr* values) | BinOp(expr left, operator op, expr right) ... | Str(string s) -- need to specify raw, unicode, etc? | Bytes(bytes s) | NameConstant(singleton value) | Ellipsis
-- the following expression can appear in assignment context | Attribute(expr value, identifier attr, expr_context ctx) | Subscript(expr value, slice slice, expr_context ctx) | Starred(expr value, expr_context ctx) | Name(identifier id, expr_context ctx) | List(expr* elts, expr_context ctx) | Tuple(expr* elts, expr_context ctx)
to this:
expr = BoolOp(boolop op, expr* values) | BinOp(expr left, operator op, expr right) ... | Str(string s) -- need to specify raw, unicode, etc? | Bytes(bytes s) | NameConstant(singleton value) | Ellipsis
-- the following expression can appear in assignment context | expr_asgn
expr_asgn = Attribute(expr value, identifier attr, expr_context ctx) | Subscript(expr value, slice slice, expr_context ctx) | Starred(expr value, expr_context ctx) | Name(identifier id, expr_context ctx) | List(expr* elts, expr_context ctx) | Tuple(expr* elts, expr_context ctx)
I doubt ASDL will let you do that.
asdl.py is plain broken - wrong number of arguments passed to output function asdl_c.py worked ok with fixed ASDL and generated - diff attached. I don't know what to check further - on Windows without Visual Studio. --
asdl.py is about to be replaced - please see http://bugs.python.org/issue19655 - I'm waiting for after the 3.4 branch to move forward with that. Let's discuss any needed fixes in that issue. Eli
participants (5)
-
anatoly techtonik -
Benjamin Peterson -
Brett Cannon -
Eli Bendersky -
Greg Ewing