On 1/27/2021 7:01 PM, Guido van Rossum wrote:

It's likely that the additions are going to break someone's code;

Since at least 2007, when Georg moved the 3i reST tree into the 3k branch, the instances have been described as "Class/Function Descriptor Objects", returned by readmodule(_ex), that "provide the following data members". There are no methods (maybe __repr__ should be added for debugging), so these are pure data classes, identical in purpose to pure dataclasses. The only intended reason to call these would be to create a tree for testing. We do this indirectly with a couple of private test-only pyclbr functions. Being private, *those* functions can have the new parameter added anywhere, regardless of the Class/Function signatures. test_pyclbr uses a constructed tree as the expected output from readline_ex(test_code). IDLE's test_browser uses a construction as test input to the browser widget. If someone does similar testing, it could break with the proposed changes.

the question seems to be do we care, since the constructors are undocumented?

If any library functions return dataclasses, are the signatures of the (possibly generated) __init__ functions documented? I care more about breaking tests than possibly non-existent 'off-label' uses, but Python won't know.

But shouldn't we just document the signatures,

The existing or proposed signatures?

so that in the future we're not going to make the same mistake?

I am not sure what you mean by the 'mistake' to not be repeated. I consider it be had been either not declaring the signature private or at least not making optional parameters keyword only.

I also wonder if some judicious use of keyword-only args might help users who currently call the old constructors catch the breakage where it's easily diagnosed (i.e. at the call site) instead of once the object is instantiated (e.g. "why is end_lineno/parent the wrong type")?

With 'end_lineno' added after existing required parameters, just before the existing optional parameter 'parent', an existing Class call ending with '..., posint, parent=SomeClass)' would fail with TypeError: Class.__init__ missing 1 required positional argument: 'end_lineno' If parent had previously been made 'keyword only', a call ending instead with '..., num, SomeClass' would be a failure now. But since not,

Perhaps even add some type checks to catch obvious mistakes (could be as simple as `assert isinstance(end_lineno, int)").

Do you consider this enough to make the proposed positioning of 'end_lineno' acceptible? The tradeoff is some immediate annoyance versus forever annoyance of a misplaced required 'optional' parameter. Since I have in mind a possible IDLE use for end_lineno, quite different from that of the patch author, I consider either way of adding it superior to never adding it.

On Wed, Jan 27, 2021 at 2:36 PM Terry Reedy <tjreedy@udel.edu <mailto:tjreedy@udel.edu>> wrote:

pyclbr is the stdlib module browser (once just class browser, hence the name).  The doc https://docs.python.org/3/library/pyclbr.html#module-pyclbr <https://docs.python.org/3/library/pyclbr.html#module-pyclbr>, which I revised in 2017, documents readline and readline_ex as the public call interface.  The functions return a hierarchical structure that includes Function and Class instances.  (Both subclass pyclbr._Object.)  The doc I revised already omitted signatures for these classes and just listed the attributes of instances.  Just before that listing, I added "Users are not expected to create instances of these classes."

https://bugs.python.org/issue38307 <https://bugs.python.org/issue38307> and https://github.com/python/cpython/pull/24348 <https://github.com/python/cpython/pull/24348> propose to add an end_lineno attribute. Since pyclbr is now, since last November, based on the ast tree and since the relevant ast nodes have an end_line attribute, the proposal amounts to copying that attribute, along with others, instead of ignoring it.

The patch proposes to add 'end_lineno' after existing start 'lineno' in the _Object, Function, and Class signatures.  I prefer doing this, rather than adding the new parameter at the end of the list, because a) being the the logical place would make the code easier to read, and b) new names as the end of the signature, follow optional arguments, would have to be optional, whereas

My question is whether the omission of signatures and my added sentence sufficiently defines the signatures of these classes (in particular, the argument order) as private to approve the patch as is?

-- Terry Jan Reedy _______________________________________________ Python-Dev mailing list -- python-dev@python.org <mailto:python-dev@python.org> To unsubscribe send an email to python-dev-leave@python.org <mailto:python-dev-leave@python.org> https://mail.python.org/mailman3/lists/python-dev.python.org/ <https://mail.python.org/mailman3/lists/python-dev.python.org/> Message archived at https://mail.python.org/archives/list/python-dev@python.org/message/SQWNNGY7... <https://mail.python.org/archives/list/python-dev@python.org/message/SQWNNGY7...> Code of Conduct: http://python.org/psf/codeofconduct/ <http://python.org/psf/codeofconduct/>

-- --Guido van Rossum (python.org/~guido <http://python.org/~guido>) /Pronouns: he/him //(why is my pronoun here?)/ <http://feministing.com/2015/02/03/how-using-they-as-a-singular-pronoun-can-c...>

-- Terry Jan Reedy

Guido, thank you for the helpful discussion. I now think that we should just add 'end_lineno=None' at the end of the Function/Class __init__ signatures. pyclbr makes one call to each to contruct the tree it returns, and the test functions make another call to each. If those are the only two calls to each, it hardly matters what they look like. If there are non-stdlib calls, it is not worth breaking them. And others might well localize their Function/Class calls within wrappers similar to _nest_function() and _nest_class(). A more important pyclbr issue, I think, is that readline and readline_ex return a 'half node', a dict of children, instead of a Module node. It is a nuisance, such as when constructing IDLE's module browser tree. It is the same as if ast_parse were to return the body list of ast.Module instead of ast.Module itself. A readmodule() function could return a proper tree with a root Module node, with attributes file, name, lineno (1), end_lineno, and children. On 1/28/2021 11:32 PM, Guido van Rossum wrote:

On Thu, Jan 28, 2021 at 8:08 PM Terry Reedy <tjreedy@udel.edu <mailto:tjreedy@udel.edu>> wrote:

On 1/27/2021 7:01 PM, Guido van Rossum wrote: > It's likely that the additions are going to break someone's code;

Since at least 2007, when Georg moved the 3i reST tree into the 3k branch, the instances have been described as "Class/Function Descriptor Objects", returned by readmodule(_ex), that "provide the following data members".  There are no methods (maybe __repr__ should be added for debugging), so these are pure data classes, identical in purpose to pure dataclasses.

Lots of people just read the source code though. In general we've often been careful with changing undocumented APIs if we believe they are commonly used in ways not endorsed by the official documentation.

The only intended reason to call these would be to create a tree for testing.

Really? Couldn't someone have an application where they insert new objects (like is documented for the ast module)?

We do this indirectly with a couple of private test-only pyclbr functions.  Being private, *those* functions can have the new parameter added anywhere, regardless of the Class/Function signatures.

test_pyclbr uses a constructed tree as the expected output from readline_ex(test_code).  IDLE's test_browser uses a construction as test input to the browser widget. If someone does similar testing, it could break with the proposed changes.

So the question is whether we care. I don't know how to assess that.

> the > question seems to be do we care, since the constructors are undocumented?

If any library functions return dataclasses, are the signatures of the (possibly generated) __init__ functions documented?

I'd say so, because we document how `@dataclass` constructs the `__init__()` method.

I care more about breaking tests than possibly non-existent 'off-label' uses, but Python won't know.

> But shouldn't we just document the signatures,

The existing or proposed signatures?

I meant the proposed signatures.

> so that in the future > we're not going to make the same mistake?

I am not sure what you mean by the 'mistake' to not be repeated.  I consider it be had been either not declaring the signature private or at least not making optional parameters keyword only.

Yeah, by 'mistake' I meant leaving it ambiguous whether the constructor signature is part of the API. And yes, we could also have designed the constructor signatures more carefully.

> I also wonder if some judicious use of keyword-only args might help > users who currently call the old constructors catch the breakage where > it's easily diagnosed (i.e. at the call site) instead of once the object > is instantiated (e.g. "why is end_lineno/parent the wrong type")?

With 'end_lineno' added after existing required parameters, just before the existing optional parameter 'parent', an existing Class call ending with '..., posint, parent=SomeClass)' would fail with

TypeError: Class.__init__ missing 1 required positional argument: 'end_lineno'

Unless you give end_lineno a default (I don't think we should do this unless we find significant rogue usages).

If parent had previously been made 'keyword only', a call ending instead with '..., num, SomeClass' would be a failure now.  But since not,

Right. One of the mistakes.

> Perhaps even add some type checks to catch obvious mistakes (could be as > simple as `assert isinstance(end_lineno, int)").

Do you consider this enough to make the proposed positioning of 'end_lineno' acceptible?  The tradeoff is some immediate annoyance versus forever annoyance of a misplaced required 'optional' parameter.

Yes, I think that would be sufficient -- we'd break rogue usages (if there are any) but at least they'd break pretty cleanly, in a way that's easy to debug.

Since I have in mind a possible IDLE use for end_lineno, quite different from that of the patch author, I consider either way of adding it superior to never adding it.

Agreed.

-- --Guido van Rossum (python.org/~guido <http://python.org/~guido>) /Pronouns: he/him //(why is my pronoun here?)/ <http://feministing.com/2015/02/03/how-using-they-as-a-singular-pronoun-can-c...>

-- Terry Jan Reedy

On 1/29/2021 8:57 PM, Guido van Rossum wrote:

On Fri, Jan 29, 2021 at 5:39 PM Terry Reedy <tjreedy@udel.edu <mailto:tjreedy@udel.edu>> wrote:

Guido, thank you for the helpful discussion.  I now think that we should just add 'end_lineno=None' at the end of the Function/Class __init__ signatures.  pyclbr makes one call to each to contruct the tree it returns, and the test functions make another call to each.

If those are the only two calls to each, it hardly matters what they look like.  If there are non-stdlib calls, it is not worth breaking them. And others might well localize their Function/Class calls within wrappers similar to _nest_function() and _nest_class().

Okay, I wasn't quite ready to recommend this, but I agree that that's the most compatible solution. (Put a `*` before the optional arg so it must be specified as a keyword.)

I presume you mean immediately before 'end_lineno', as putting it before all optional params would break positional passing.

A more important pyclbr issue, I think, is that readline and readline_ex return a 'half node', a dict of children, instead of a Module node.  It is a nuisance, such as when constructing IDLE's module browser tree.  It is the same as if ast_parse were to return the body list of ast.Module instead of ast.Module itself.  A readmodule() function could return a proper tree with a root Module node, with attributes file, name, lineno (1), end_lineno, and children.

Sounds good. Thanks for caring about this very old module! (I am not 100% sure of its origins but I think it may have been written in Python's earliest years by one of my office mates, Sjoerd Mullender.)

Adding a new API would be an opportunity to 'do it right'. The test of it being an improvement is if it allows simplification of the module browser code. -- Terry Jan Reedy