On Sun, Nov 10, 2013 at 8:34 AM, Benjamin Peterson wrote:

2013/11/10 anatoly techtonik :

...

http://hg.python.org/cpython/file/1ee45eb6aab9/Parser/Python.asdl

In Assign(expr* targets, expr value), why the first argument is a list?

x = y = 42

Thanks. Speaking of this ASDL. `expr* targets` means that multiple entities of `expr` under the name 'targets' can be passed to Assign statement. Assign uses them as left value. But `expr` definition contains things that can not be used as left side assignment targets: expr = BoolOp(boolop op, expr* values) | BinOp(expr left, operator op, expr right) ... | Str(string s) -- need to specify raw, unicode, etc? | Bytes(bytes s) | NameConstant(singleton value) | Ellipsis -- the following expression can appear in assignment context | Attribute(expr value, identifier attr, expr_context ctx) | Subscript(expr value, slice slice, expr_context ctx) | Starred(expr value, expr_context ctx) | Name(identifier id, expr_context ctx) | List(expr* elts, expr_context ctx) | Tuple(expr* elts, expr_context ctx) If I understand correctly, this is compiled into C struct definitions (Python-ast.c), and there is a code to traverse the structure, but where is code that validates that the structure is correct? Is it done on the first level - text file parsing, before ASDL is built? If so, then what is the role of this ADSL exactly that the first step is unable to solve? Is it possible to fix ADSL to move `expr` that are allowed in Assign into `expr` subset? What effect will it achieve? I mean - will ADSL compiler complain about wrong stuff on the left side, or it will still be a role of some other component. Which one?

2013/11/12 anatoly techtonik :

On Sun, Nov 10, 2013 at 8:34 AM, Benjamin Peterson wrote:

...

2013/11/10 anatoly techtonik :

...

http://hg.python.org/cpython/file/1ee45eb6aab9/Parser/Python.asdl

In Assign(expr* targets, expr value), why the first argument is a list?

x = y = 42

Thanks.

Speaking of this ASDL. `expr* targets` means that multiple entities of `expr` under the name 'targets' can be passed to Assign statement. Assign uses them as left value. But `expr` definition contains things that can not be used as left side assignment targets:

expr = BoolOp(boolop op, expr* values) | BinOp(expr left, operator op, expr right) ... | Str(string s) -- need to specify raw, unicode, etc? | Bytes(bytes s) | NameConstant(singleton value) | Ellipsis

-- the following expression can appear in assignment context | Attribute(expr value, identifier attr, expr_context ctx) | Subscript(expr value, slice slice, expr_context ctx) | Starred(expr value, expr_context ctx) | Name(identifier id, expr_context ctx) | List(expr* elts, expr_context ctx) | Tuple(expr* elts, expr_context ctx)

If I understand correctly, this is compiled into C struct definitions (Python-ast.c), and there is a code to traverse the structure, but where is code that validates that the structure is correct? Is it done on the first level - text file parsing, before ASDL is built? If so, then what is the role of this ADSL exactly that the first step is unable to solve?

Only valid expression targets are allowed during AST construction. See set_expr_context in ast.c.

Is it possible to fix ADSL to move `expr` that are allowed in Assign into `expr` subset? What effect will it achieve? I mean - will ADSL compiler complain about wrong stuff on the left side, or it will still be a role of some other component. Which one?

I'm not sure what you mean by an `expr` subset. -- Regards, Benjamin

On Fri, Nov 15, 2013 at 12:54 PM, anatoly techtonik wrote:

On Tue, Nov 12, 2013 at 5:08 PM, Benjamin Peterson wrote:

...

2013/11/12 anatoly techtonik :

...

On Sun, Nov 10, 2013 at 8:34 AM, Benjamin Peterson wrote:

...

2013/11/10 anatoly techtonik :

...

http://hg.python.org/cpython/file/1ee45eb6aab9/Parser/Python.asdl

In Assign(expr* targets, expr value), why the first argument is a list?

x = y = 42

Thanks.

Speaking of this ASDL. `expr* targets` means that multiple entities of `expr` under the name 'targets' can be passed to Assign statement. Assign uses them as left value. But `expr` definition contains things that can not be used as left side assignment targets:

expr = BoolOp(boolop op, expr* values) | BinOp(expr left, operator op, expr right) ... | Str(string s) -- need to specify raw, unicode, etc? | Bytes(bytes s) | NameConstant(singleton value) | Ellipsis

-- the following expression can appear in assignment context | Attribute(expr value, identifier attr, expr_context ctx) | Subscript(expr value, slice slice, expr_context ctx) | Starred(expr value, expr_context ctx) | Name(identifier id, expr_context ctx) | List(expr* elts, expr_context ctx) | Tuple(expr* elts, expr_context ctx)

If I understand correctly, this is compiled into C struct definitions (Python-ast.c), and there is a code to traverse the structure, but where is code that validates that the structure is correct? Is it done on the first level - text file parsing, before ASDL is built? If so, then what is the role of this ADSL exactly that the first step is unable to solve?

Only valid expression targets are allowed during AST construction. See set_expr_context in ast.c.

Oh my. Now there is also CST in addition to AST. This stuff - http://docs.python.org/devguide/ - badly needs diagrams about data transformation toolchain from Python source code to machine execution instructions. I'd like some pretty stuff, but raw blogdiag hack will do the job http://blockdiag.com/en/blockdiag/index.html

There is no set_expr_context in my copy of CPython code, which seems to be some alpha of Python 3.4

...

...

Is it possible to fix ADSL to move `expr` that are allowed in Assign into `expr` subset? What effect will it achieve? I mean - will ADSL compiler complain about wrong stuff on the left side, or it will still be a role of some other component. Which one?

I'm not sure what you mean by an `expr` subset.

Transform this:

expr = BoolOp(boolop op, expr* values) | BinOp(expr left, operator op, expr right) ... | Str(string s) -- need to specify raw, unicode, etc? | Bytes(bytes s) | NameConstant(singleton value) | Ellipsis

-- the following expression can appear in assignment context | Attribute(expr value, identifier attr, expr_context ctx) | Subscript(expr value, slice slice, expr_context ctx) | Starred(expr value, expr_context ctx) | Name(identifier id, expr_context ctx) | List(expr* elts, expr_context ctx) | Tuple(expr* elts, expr_context ctx)

to this:

expr = BoolOp(boolop op, expr* values) | BinOp(expr left, operator op, expr right) ... | Str(string s) -- need to specify raw, unicode, etc? | Bytes(bytes s) | NameConstant(singleton value) | Ellipsis

-- the following expression can appear in assignment context | expr_asgn

expr_asgn = Attribute(expr value, identifier attr, expr_context ctx) | Subscript(expr value, slice slice, expr_context ctx) | Starred(expr value, expr_context ctx) | Name(identifier id, expr_context ctx) | List(expr* elts, expr_context ctx) | Tuple(expr* elts, expr_context ctx)

And also this: | Assign(expr* targets, expr value) to this: | Assign(expr_asgn* targets, expr value)

On Fri, Nov 15, 2013 at 5:43 PM, Benjamin Peterson wrote:

2013/11/15 anatoly techtonik :

...

On Tue, Nov 12, 2013 at 5:08 PM, Benjamin Peterson wrote:

...

2013/11/12 anatoly techtonik :

...

On Sun, Nov 10, 2013 at 8:34 AM, Benjamin Peterson wrote:

...

2013/11/10 anatoly techtonik :

...

http://hg.python.org/cpython/file/1ee45eb6aab9/Parser/Python.asdl

In Assign(expr* targets, expr value), why the first argument is a list?

x = y = 42

Thanks.

Speaking of this ASDL. `expr* targets` means that multiple entities of `expr` under the name 'targets' can be passed to Assign statement. Assign uses them as left value. But `expr` definition contains things that can not be used as left side assignment targets:

expr = BoolOp(boolop op, expr* values) | BinOp(expr left, operator op, expr right) ... | Str(string s) -- need to specify raw, unicode, etc? | Bytes(bytes s) | NameConstant(singleton value) | Ellipsis

-- the following expression can appear in assignment context | Attribute(expr value, identifier attr, expr_context ctx) | Subscript(expr value, slice slice, expr_context ctx) | Starred(expr value, expr_context ctx) | Name(identifier id, expr_context ctx) | List(expr* elts, expr_context ctx) | Tuple(expr* elts, expr_context ctx)

If I understand correctly, this is compiled into C struct definitions (Python-ast.c), and there is a code to traverse the structure, but where is code that validates that the structure is correct? Is it done on the first level - text file parsing, before ASDL is built? If so, then what is the role of this ADSL exactly that the first step is unable to solve?

Only valid expression targets are allowed during AST construction. See set_expr_context in ast.c.

Oh my. Now there is also CST in addition to AST. This stuff - http://docs.python.org/devguide/ - badly needs diagrams about data transformation toolchain from Python source code to machine execution instructions. I'd like some pretty stuff, but raw blogdiag hack will do the job http://blockdiag.com/en/blockdiag/index.html

There is no set_expr_context in my copy of CPython code, which seems to be some alpha of Python 3.4

It's actually called set_context.

Ok. So what is the process? SOURCE --> TOKEN STREAM --> SENTENCE STREAM --> CST --> --> AST --> BYTECODE Is that right?

...

...

...

Is it possible to fix ADSL to move `expr` that are allowed in Assign into `expr` subset? What effect will it achieve? I mean - will ADSL compiler complain about wrong stuff on the left side, or it will still be a role of some other component. Which one?

I'm not sure what you mean by an `expr` subset.

Transform this:

expr = BoolOp(boolop op, expr* values) | BinOp(expr left, operator op, expr right) ... | Str(string s) -- need to specify raw, unicode, etc? | Bytes(bytes s) | NameConstant(singleton value) | Ellipsis

-- the following expression can appear in assignment context | Attribute(expr value, identifier attr, expr_context ctx) | Subscript(expr value, slice slice, expr_context ctx) | Starred(expr value, expr_context ctx) | Name(identifier id, expr_context ctx) | List(expr* elts, expr_context ctx) | Tuple(expr* elts, expr_context ctx)

to this:

expr = BoolOp(boolop op, expr* values) | BinOp(expr left, operator op, expr right) ... | Str(string s) -- need to specify raw, unicode, etc? | Bytes(bytes s) | NameConstant(singleton value) | Ellipsis

-- the following expression can appear in assignment context | expr_asgn

expr_asgn = Attribute(expr value, identifier attr, expr_context ctx) | Subscript(expr value, slice slice, expr_context ctx) | Starred(expr value, expr_context ctx) | Name(identifier id, expr_context ctx) | List(expr* elts, expr_context ctx) | Tuple(expr* elts, expr_context ctx)

I doubt ASDL will let you do that.

asdl.py is plain broken - wrong number of arguments passed to output function asdl_c.py worked ok with fixed ASDL and generated - diff attached. I don't know what to check further - on Windows without Visual Studio. -- anatoly t.

On Fri, Nov 22, 2013 at 7:29 AM, Benjamin Peterson wrote:

...

On Fri, Nov 15, 2013 at 5:43 PM, Benjamin Peterson wrote:

...

2013/11/15 anatoly techtonik :

...

On Tue, Nov 12, 2013 at 5:08 PM, Benjamin Peterson < benjamin@python.org> wrote:

...

2013/11/12 anatoly techtonik :

...

On Sun, Nov 10, 2013 at 8:34 AM, Benjamin Peterson < benjamin@python.org> wrote: > 2013/11/10 anatoly techtonik : >> http://hg.python.org/cpython/file/1ee45eb6aab9/Parser/Python.asdl >> >> In Assign(expr* targets, expr value), why the first argument is a

2013/11/22 anatoly techtonik : list?

...

...

...

...

...

> > x = y = 42

Thanks.

Speaking of this ASDL. `expr* targets` means that multiple entities of `expr` under the name 'targets' can be passed to Assign statement. Assign uses them as left value. But `expr` definition contains things that can not be used as left side assignment targets:

expr = BoolOp(boolop op, expr* values) | BinOp(expr left, operator op, expr right) ... | Str(string s) -- need to specify raw, unicode, etc? | Bytes(bytes s) | NameConstant(singleton value) | Ellipsis

-- the following expression can appear in assignment context | Attribute(expr value, identifier attr, expr_context ctx) | Subscript(expr value, slice slice, expr_context ctx) | Starred(expr value, expr_context ctx) | Name(identifier id, expr_context ctx) | List(expr* elts, expr_context ctx) | Tuple(expr* elts, expr_context ctx)

If I understand correctly, this is compiled into C struct definitions (Python-ast.c), and there is a code to traverse the structure, but where is code that validates that the structure is correct? Is it done on the first level - text file parsing, before ASDL is built? If so, then what is the role of this ADSL exactly that the first step is unable to solve?

Only valid expression targets are allowed during AST construction. See set_expr_context in ast.c.

Oh my. Now there is also CST in addition to AST. This stuff - http://docs.python.org/devguide/ - badly needs diagrams about data transformation toolchain from Python source code to machine execution instructions. I'd like some pretty stuff, but raw blogdiag hack will do the job http://blockdiag.com/en/blockdiag/index.html

There is no set_expr_context in my copy of CPython code, which seems to be some alpha of Python 3.4

It's actually called set_context.

Ok. So what is the process?

SOURCE --> TOKEN STREAM --> SENTENCE STREAM --> CST --> --> AST --> BYTECODE

Is that right?

I don't know what sentence stream is, but otherwise looks right.

If you want more of an overview: http://pyvideo.org/video/2331/from-source-to-code-how-cpythons-compiler-work...

On Fri, Nov 22, 2013 at 3:21 AM, anatoly techtonik wrote:

...

2013/11/15 anatoly techtonik :

...

On Tue, Nov 12, 2013 at 5:08 PM, Benjamin Peterson wrote:

...

2013/11/12 anatoly techtonik :

...

On Sun, Nov 10, 2013 at 8:34 AM, Benjamin Peterson < benjamin@python.org> wrote:

...

2013/11/10 anatoly techtonik : > http://hg.python.org/cpython/file/1ee45eb6aab9/Parser/Python.asdl > > In Assign(expr* targets, expr value), why the first argument is a

On Fri, Nov 15, 2013 at 5:43 PM, Benjamin Peterson wrote: list?

...

...

...

...

...

x = y = 42

Thanks.

Speaking of this ASDL. `expr* targets` means that multiple entities of `expr` under the name 'targets' can be passed to Assign statement. Assign uses them as left value. But `expr` definition contains things that can not be used as left side assignment targets:

expr = BoolOp(boolop op, expr* values) | BinOp(expr left, operator op, expr right) ... | Str(string s) -- need to specify raw, unicode, etc? | Bytes(bytes s) | NameConstant(singleton value) | Ellipsis

-- the following expression can appear in assignment context | Attribute(expr value, identifier attr, expr_context ctx) | Subscript(expr value, slice slice, expr_context ctx) | Starred(expr value, expr_context ctx) | Name(identifier id, expr_context ctx) | List(expr* elts, expr_context ctx) | Tuple(expr* elts, expr_context ctx)

If I understand correctly, this is compiled into C struct definitions (Python-ast.c), and there is a code to traverse the structure, but where is code that validates that the structure is correct? Is it done on the first level - text file parsing, before ASDL is built? If so, then what is the role of this ADSL exactly that the first step is unable to solve?

Only valid expression targets are allowed during AST construction. See set_expr_context in ast.c.

Oh my. Now there is also CST in addition to AST. This stuff - http://docs.python.org/devguide/ - badly needs diagrams about data transformation toolchain from Python source code to machine execution instructions. I'd like some pretty stuff, but raw blogdiag hack will do the job http://blockdiag.com/en/blockdiag/index.html

There is no set_expr_context in my copy of CPython code, which seems to be some alpha of Python 3.4

It's actually called set_context.

Ok. So what is the process?

SOURCE --> TOKEN STREAM --> SENTENCE STREAM --> CST --> --> AST --> BYTECODE

Is that right?

...

...

...

...

Is it possible to fix ADSL to move `expr` that are allowed in Assign into `expr` subset? What effect will it achieve? I mean - will ADSL compiler complain about wrong stuff on the left side, or it will still be a role of some other component. Which one?

I'm not sure what you mean by an `expr` subset.

Transform this:

expr = BoolOp(boolop op, expr* values) | BinOp(expr left, operator op, expr right) ... | Str(string s) -- need to specify raw, unicode, etc? | Bytes(bytes s) | NameConstant(singleton value) | Ellipsis

-- the following expression can appear in assignment context | Attribute(expr value, identifier attr, expr_context ctx) | Subscript(expr value, slice slice, expr_context ctx) | Starred(expr value, expr_context ctx) | Name(identifier id, expr_context ctx) | List(expr* elts, expr_context ctx) | Tuple(expr* elts, expr_context ctx)

to this:

expr = BoolOp(boolop op, expr* values) | BinOp(expr left, operator op, expr right) ... | Str(string s) -- need to specify raw, unicode, etc? | Bytes(bytes s) | NameConstant(singleton value) | Ellipsis

-- the following expression can appear in assignment context | expr_asgn

expr_asgn = Attribute(expr value, identifier attr, expr_context ctx) | Subscript(expr value, slice slice, expr_context ctx) | Starred(expr value, expr_context ctx) | Name(identifier id, expr_context ctx) | List(expr* elts, expr_context ctx) | Tuple(expr* elts, expr_context ctx)

I doubt ASDL will let you do that.

asdl.py is plain broken - wrong number of arguments passed to output function asdl_c.py worked ok with fixed ASDL and generated - diff attached. I don't know what to check further - on Windows without Visual Studio. --

asdl.py is about to be replaced - please see http://bugs.python.org/issue19655 - I'm waiting for after the 3.4 branch to move forward with that. Let's discuss any needed fixes in that issue. Eli